This chapter will focus on the origins of Puppet, why it was created, and how it is used in DevOps engineering. It will look at Puppet’s approach to configuration management and how its declarative approach differs from more regular procedural languages. Puppet has many features that are common in other languages such as variables, conditional statements, and functions. But in this chapter, we will cover the key terms, structure, and ideas of the language that make it different and how the underlying platform runs. We will give a clear, high-level overview of its approach and how it relates to customer needs and infrastructure environments. Finally, as there are a lot of preconceptions regarding Puppet, this chapter will finish by addressing some of the most common ones, including where they come from, and unwrap them.

This should ensure a fundamental understanding of Puppet and its approach before we build up a deeper, technical understanding of the language in upcoming chapters. It will also ensure this book is not just about technology but how genuine value can be delivered to customers using the service that Puppet provides.

In this chapter, we are going to cover the following main topics:

• Puppet’s history and relationship to DevOps
• Puppet as a declarative and idempotent language
• Key terms in the Puppet language
• Puppet as a platform
• Common misconceptions

Puppet was started by creator and founder Luke Kanies, who was working as a sysadmin and consultant. He was unable to find the tooling he wanted to use and that his customers could rely on, so he created Puppet as a Ruby-based open source configuration management language in 2005. The success of this open source project resulted in the release of a commercial offering, Puppet Enterprise, in February 2011. But as the demands increased and Puppet needed to reform and expand as both a company and an open source project, Luke stood down, stating that the challenges of growing Puppet to enterprise-scale were far from what I love to do most, and far from my core skills. We need to scale, and we need to execute.

The new leadership that followed took a direction that saw the company develop its professional services, and focus more effort on developer tooling and education while expanding its product range both organically and via acquisitions, striking a difficult balance between the open source community and its enterprise customer demands. Puppet was acquired by Perforce Software on May 17, 2022, following the Chef (2020) and Ansible (2015) acquisitions, as the last of the standalone configuration management start-ups. Luke summed up the change that has taken place in the industry: DevOps teams are different now. Companies are looking for a complete solution, rather than wanting to integrate individual best-of-breed vendors.

This history has seen Puppet move from a tool that left it to the developer to decided how best to use it to solve problems to, today, a tool with patterns and solutions that users can just consume to standardize their automation and deployment. This has allowed users to focus on their solutions and not the underlying technology.

DevOps itself has become a frustrating term in the IT industry; the definition given by formal sources differs hugely from how companies actually use it, and references to it can be used as a cynical buzzword or sales gimmick. The focus of this book is on DevOps engineering, as used particularly by large companies and has been well researched and discussed in studies such as the Puppet-run State of DevOps Report. DevOps engineering is normally delivered as part of projects such as digital transformations, cloud-first migrations, and various other modernization projects. What is typically seen in these projects is a desire to automate self-service deployment, compliance, and remove toil. This approach follows the DevOps goal of breaking down the silos between developers and ops teams by allowing better communication and establishing shared goals. What is noticeable is that the system administrator role in which Luke worked originally has effectively been replaced by roles such as DevOps engineers.

Puppet will be used as part of a DevOps toolchain, and Figure 1.1 shows an example set of tools and their relative functions. It is typical for Puppet to start its role at the end of a provisioning pipeline, as infrastructure is stood up in a platform and needs to be configured and enforced:

Figure 1.1 – A DevOps toolset

This book will focus not just on a technological understanding but also on how to use the maturity of the Puppet language, tooling, and platform with opinionated patterns. These approaches have been developed through years of customer engagements for Puppet and the communities’ own implementations to allow users to reduce their effort in finding the right approach, focus on their solutions, and deliver immediate benefit and return to their customers.

The first important thing to understand is how Puppet differs from normal scripting or coding languages. Puppet is declarative, meaning you describe the state you want the system to be in. For example, you could describe that your system should have a user called username with UID 1234, a configuration file should not exist, and a kernel setting should be at a particular value. In comparison to most languages where you have to describe the process to get to the state, Puppet’s approach brings us closer to how customers request services. They don’t want to know how it’s done, just that it will meet their requirements. These resource definitions can be saved in your version control system. Often, this approach is described as being part of Infrastructure as Code.

Puppet is idempotent, meaning that it will only make the changes required to get into the declared state. Meanwhile, most procedural languages will run steps every time and, typically, require various checks such as if statements to be added to make checks to avoid duplication. This is particularly powerful as what is called enforcement can be run with the Puppet language, ensuring the state you declared has been reached, and is capable of detecting whether a change happened because of you updating the state you wished the machine to be in or whether it was a change that happened on the machine itself moving away from the desired state. This can greatly assist with audits and avoid any configuration drifts in an estate and ensure change is managed and deliberate.

Puppet is OS-independent; the language is focused on the state, not the underlying implementation of how particular OSes install a package or add a user. This gives us a universal language that is independent of any underlying implementations, allowing for less duplication of code, avoiding the need to use layers of case/if statements to detect differences, and allowing multiple language implementations such as PowerShell for Windows and Bash for Unix-based systems. Additionally, it makes it easier to recover after failures in applying code. If in a procedural language, a step fails, it might not be safe to run the script in full again depending on how well the check steps have been coded. In contrast, Puppet code is able to resume only performing the steps it needs to reach the correct state.

A simple example of Puppet code to create a user would look like this:

user { 'david'
  uid => '123'
}

In contrast, a shell script might have a section like this:

if ! getent passwd david; then
  useradd -u 123 david
elif ! $(uid david) == 123; then
  usermod -u 123 david
fi

In the preceding shell example, we have to check whether a user exists, and if not, create one. If it does exist, then does it have the right UID? If not, we change it. This script only covers OSes that can use useradd and usermod. To achieve compatibility with multiple OSes, we would need a test to detect the OS type and produce a section of code like this for every OS or group of OSes and their required commands. Often, it would be more practical to write in multiple languages and scripts to cover a broader base of OS flavors, that is, if we wanted to cover both Unix and Windows, for example.

This compares to the Puppet declaration, which will work on multiple OSes without change as Puppet will detect the required commands and perform all the necessary state checks as part of that.

This example is all just for a single resource with a single attribute. You can quickly see how the shell script example will not scale as it becomes increasingly complex with almost endless checks and options.

Looking at the Puppet language in more detail, the most fundamental item in Puppet is a resource. Each resource describes some part of the system and the desired state you wish it to be in. Each resource has a type, which is a definition for the Puppet language of how this particular resource can be configured, which attributes can be set, and what providers can be used. The attributes are what describe the state. So, for a user, this might be a home directory or, for a file, the permissions. Providers are what make the Puppet OS independent since they do the underlying commands be they for creating a user or installing a package.

So, let’s take an example of a company that typically submits build request forms to an environments team to request the configuration for a server:

Table 1.1 – An example build request form

In Table 1.1, the request form, we see groupings of users, groups, and directories, which are all, essentially, types. Each item under them is a resource, and the configuration settings are the attributes.

This request could translate to something like the following:

user { 'exampleapp':
  uid => '1234'.
  gid => '123'
}
group { 'exampleapp':
  Gid => '123'
}
file { '/opt/exampleapp/':
  owner => 'exampleapp',
  group => 'exampleapp',
  mode  => 755
}
file { '/etc/exampleapp/':
  owner => 'exampleapp',
  group => 'exampleapp',
  mode  => 750
}

The preceding example shows how Puppet translates more directly to user requests and can remain readable without even understanding any of the Puppet language.

What isn’t visible, in this example, is the providers. Puppet has defaults, such as in the preceding example, where the user resource assumes a RedHat host will use the usermod provider. Instead, if I wished to use LDAP commands for user creation, I would set my provider attribute to LDAP.

The next important thing to note is that due to the nature of writing Puppet in a stateful way, we are not writing an ordered process that executes line by line but only declaring the state of resources that could be implemented in any order. Therefore, if we have any dependencies, we need to use the relationship parameter; this describes a before/after relationship, which is exactly as it sounds, or a subscribe/refresh, whereby, for example, updating a configuration file could cause a service to restart. In the previous example, Puppet automatically creates certain dependencies such as ensuring the group is created before the user, so we don’t have to add a relationship parameter. Often, these relationships are seen as one of the most difficult parts of Puppet to adapt to, as many coders are used to writing a process to follow and mistakes can be made. This can cause a cycle of dependencies, whereby a chain of these dependencies cycles round, and there is no way to create a starting resource that isn’t dependent on another.

Evidently, the resources we declare need a structure, and the first step is for this code to be in a file. Puppet calls these manifest files, which have an extension of .pp. Classes are blocks of Puppet code that give us a way to specifically call sections of code to be run on hosts. Normally, as a good practice, we only have one class in a manifest file. Puppet then uses modules as a way to group these manifests and classes. This grouping is based on the principle that a module should do a single thing well and represent a technical implementation, such as a module configuring the IIS application or configuring postfix as a mail relay. Modules are simply a directory structure storing the manifests, classes, and other Puppet items (which we will cover, in detail, in Chapter 8) and are not a keyword in the language itself. So, ideally, modules should be shareable and reusable for different users and organizations with many taken straight from the Puppet Forge, which is Puppet’s catalog of modules with both commercial and open source offerings.

An example of one common style and practice for modules is to have a manifest file with a single class for the following:

• install.pp (grouping resources related to installing software)
• config.pp (grouping resources related to configuring software)
• service.pp (grouping resources related to running services)
• init.pp (a way of initializing the module and accepting parameters)

At a higher level, we then have roles and profiles, which are used to create the structure of your organization. While modules should be sharable and repeatable installations of technical implementations, such as Oracle or IIS, roles and profiles will only have context within your organization. Roles and profiles are classes used to group modules and selected parameters into logical technical stacks and customer solutions. It is common to make a roles module and a profiles module while keeping together the classes used.

What can be confusing, at this point, is that you can end up with an Oracle role, an Oracle profile, and an Oracle module. So, while the Oracle module configures and installs Oracle with various parameters available to it to customize the installation, the Oracle profile is about how your organization uses this module and what other modules it might add to this technology stack. You might specify that you always use Oracle with a cluster service and, therefore, your Oracle profile contains both an Oracle module and a cluster module. Alternatively, it might pass parameters to the Oracle module within your profile, which set default kernel settings for your organization’s configuration.

You can think of a role as being what the customer actually wants when they submit a build request; they need a particular type of server, be it an Oracle or an IIS server. They don’t care about the underlying implementations – only that it meets their requirements. While the Oracle role will certainly need the Oracle profile, it will expect it to meet the OS security standard and to have any agents or other supporting tools your organization defines. Therefore, a common profile for many organizations is a base OS security standard that ensures every server is compliant and that is part of almost every role.

Figure 1.2 shows an example of what has just been described as an Oracle role class in the roles module, which includes an Oracle profile class and an OS security profile class, both from the profile module. Then, the Oracle profile includes an Oracle module, while the os_security profile includes the DNS module:

Figure 1.2 – The structure of roles, profiles, and modules

In Chapter 8, we will go into more technical detail, but the key takeaway from this overview is to understand that modules provide sharable and reusable single-use technical installations. In contrast, the roles and profiles pattern provides the context for your organization. Roles are for customers ordering server offerings; they don’t need to understand the technical implementation, only that it meets their business requirement. The profiles in your organization’s technology stack are managed by technical designers and architects, who combine and specify modules according to your organization’s standards and configurations. These roles are responsible for defining how different components are integrated to create the desired technology stack. So while an Oracle module by itself can configure and install Oracle, it is the profile that defines the exact configurations that should be passed to that Oracle module and the other modules it may be dependent on such as having a NetBackup client installed.

With what we have covered in modules, roles, and profiles, going back to Table 1.1, instead, we can have a customer submitting the build request form but not having to specify everything they need; they could simply order an exampleapp role server.

What we have seen so far is fine when servers meet all the specifications and are standard, but exceptions are commonplace. Hiera is Puppet's data system, and it can be used to pass parameters to the roles and profiles model to handle exceptions. Hiera, as its name suggests, is hierarchical. It defines an ordered lists of data sources to access to find the most relevant setting. These data sources will typically be ordered from the default value for all nodes to a more specific group such as a particular role and specific values for an individual node.

For example, if email servers were disabled by the default OS security profile but were required for exampleapp, we could have the following YAML file:

exampleapp.yaml

profile::os_security:email_enabled: true

Similarly, if server1 needed a different UID, we could have the following YAML file:

server1.yaml

profile::exampleapp:uid: '1235'

One of the most important points of creating these patterns is to avoid hardcoded values in your modules. By using Hiera, you give yourself a dynamic way to change the values in the future without modifying the code. This could evolve to access the data via a self-service portal – automating away from builds ordered via spreadsheets, emails, and discussions, which would have to be configured by the build teams instead of portals such as VMware vRealize Automation or ServiceNow:

Figure 1.3 – An example portal

In Figure 1.3, an example portal shows how customers can be presented with simplified products. The focus of the Puppet language should be to deliver consistent products to customers and allow customers, architects, and technical staff to focus on what they care about and not have to delve into the technical requirements or coding sections themselves.

So far, this chapter has focused on the Puppet language, but now we will look at the Puppet platform and how it applies the desired state to client servers. Puppet can be run with just an installed agent and all the files locally, which is common for testing, but this overview will focus on the client-server setup. In Chapters 10, 13, and 14, we will go into much more detail about resilience, scalability, and more advanced running options. However, for now, we will focus on how a Puppet client talks to a server to request and apply its desired state.

Every client under Puppet control will install a Puppet agent. Figure 1.4 shows the steps of a Puppet agent run, which this section will outline:

Figure 1.4 – The Puppet agent run life cycle

The first step is for the agent to identify itself to the primary server with SSL keys or to create new SSL keys for the primary server to sign. This will secure communication between the server and client.

The next action is for the client to use a Ruby library called Facter. This is a system profiler to gather what is known as facts about the system. This can be things such as the OS version or RAM size. These facts can be used in code or by Hiera to make choices about what state a host should be in, such as Windows Server 2022 having a particular registry setting.

Then, the server identifies what classes should be applied to a server. Typically, this is done by what is called an external node classifier (ENC) script, which is based on the facts and user definitions. Normally, this will apply a role class to a server, which, as we discussed in the previous section, builds up a definition of profiles and module classes.

Then, the primary server compiles a catalog and a YAML file of the resources to be applied to the node (ensuring the CPU-intensive work happens on the server and not the client).

This catalog is then sent to the client who uses the catalog as a blueprint of what the state should look like and makes any necessary changes to enforce the state on the client.

Finally, a report is sent back to the primary server confirming what resources were applied and whether these resources had to be changed due to a change in Puppet code or whether they were changed outside of Puppet control (which might be an audit or security breach).

In Figure 1.5, we see an example extract from a Puppet report showing the name of the resource, the type of change made, and the value it needed to change. Additionally, the report includes a record of unchanged resources highlighting what is part of Puppet's enforcement:

Figure 1.5 – The Puppet console server report

By default, this cycle takes place every 30 minutes. In the previous sections, the focus was on how the language can automate the building of servers. Here, we can see that, via the platform, we can ensure all our deployed servers are enforced with the state we set out to achieve; whether that be a security standard profile or whether we decided to update the settings in a particular implementation such as adding extra features to IIS. This avoids server drift, where servers on the estate are difficult to keep up to date or are vulnerable to changes made manually in error or that maliciously breach standards. Figure 1.6 shows the dashboard view of Puppet Enterprise, giving a clear view of an estate of servers and the status of the last run. This highlights whether the servers are in compliance with our state or had to make changes in their previous run:

Figure 1.6 – The Puppet console status dashboard

What we have reviewed so far would presume a common code base, and when any code changes are made, all clients would have a new state enforced within the next 30 minutes as agents contact the primary server. This would clearly be problematic, as bugs will affect all servers within a brief period. This is why Puppet has environments. An environment is a collection of versioned modules. This is achieved by storing the modules in revision control, such as git, where the version can be declared as a commit, a tag, or a branch, which we can list in a file called a Puppetfile.

An example module declaration would look like this:

mod 'apache',
  :git => 'https://github.com/exampleorg/exampleapp'
  :tag => '1.2'

By maintaining this Puppetfile in git, in what is known as a control repo, it is possible to represent multiple environments by having different branches with different versions of the Puppet file.

A common practice is to match environments against how your organization classifies server usage. Normally, this means a minimum of a development environment and a production environment. So, changes can be tested against servers in development, and then the successfully tested ones can be deployed to production. This can be taken further using canary environments to test small subsets of the server. This approach can all be customized to the change and risk setup of different organizations.

All the facts and reports we mentioned, as part of the agent cycle, are stored in PuppetDB, a frontend application using PostgreSQL as a backend database, which is designed to manage Puppet data such as reports and facts. This is used with the Puppet Query Language (PQL), which allows us to search the information we have gathered. This can allow for searching of facts giving CMDB style data and for combinations where we can check whether a certain resource for a role had changed, which could indicate a change breach had taken place.

So, in this section, we have seen that the Puppet platform gives a way to progressively deploy new code based on environments. It stores facts about the clients along with the reports generated on each run, giving a powerful view of CMDB along with audit and compliance information in the reports as we confirm what state the servers are in. This can all be searched using PQL. This can lead to huge savings in operational toil in terms of audit and compliance report generation and helps avoid building technical debt as standards and configurations change.

Isn’t Puppet dead?

The focus of bleeding-edge technology has moved on to serverless and other Software-as-a-Service (SaaS)/containerized offerings, while at an Infrastructure-as-a-Service (IaaS) level, development in Puppet has reached a much greater level of maturity. 10 years ago, you might have bought this book assuming it was relevant regardless of whether you were going to work with Puppet. Today, you have a Puppet solution to implement or understand.

I need to know Ruby to use Puppet.

Some basic knowledge of Ruby would be an advantage for certain areas of Puppet code. A focus on the good use of the Puppet language to get early returns is what this book will focus on, and the reality is that the majority of Puppet professionals don’t spend much time with Ruby trying to create customizations. Even specialists working for Puppet itself find that it can be some time before they need to write something custom in Ruby.

Puppet won’t work with our change management.

A big fear is the idea of Puppet making changes outside the scope of governance and change management. This often reflects assumptions and a lack of communication with change management teams. Puppet will enforce the state you have described; therefore, changes will only happen if the state described in the code has changed or if it has been changed outside of Puppet’s control. As previously mentioned, as long as it is agreed that Puppet is the way to define particular resources, any change to the state should be seen as outside of governance and, therefore, put back into place. Later chapters will discuss how to release code and environments to ensure that Puppet remains properly access-controlled and, therefore, within governance.

I can’t make manual changes or exceptions.

This could certainly happen if users try to work around Puppet. To avoid this, it’s important to define what Puppet is responsible for, what other tooling or manual processes are responsible for, and how exceptions should be requested and approved in your system. As will be discussed in Chapters 8 and 9, by using parameters in modules and Hiera for exceptions, a controlled method can be used for exceptions, which also keeps a record in code.

I need Puppet Enterprise to use add-ons and integrations.

There is a huge amount of confusion, particularly from industry analysts, who make comparisons about what users get with Puppet Enterprise and how open source might be limited. This book will go into more depth in Chapter 14, but the fundamental difference for Puppet Enterprise is you are paying for support, services, and pre-canned modules, infrastructure, and solutions. If you have the skills, developers, and time, all of these features can be replicated in open source. Ultimately, Enterprise runs on the open source components.

Everyone will need to learn Puppet.

A major focus of this book will be the importance of structuring code to allow for self-service processes. This avoids users who might wish to have small exceptions or integrations having to learn everything as a Puppet developer and only having to understand your offerings.

It will clash with other systems.

The key part is to understand what Puppet will be responsible for and what other systems will be responsible for, and to document this well. Many environments will run multiple configuration management, orchestration, and software management tools. The important thing is to use them to their strengths with clear boundaries.

In this chapter, you learned how Puppet was created by Luke Kanies as a stateful language to ease the automation of the configuration management of servers. We learned how using this stateful approach provides a language more natural for describing user requirements for configuration management and reduces the complexity involved in more traditional procedural approaches.

We looked at an overview of the core language terms and components and how they are structured via roles, profiles, and modules. This structure offers a natural way to create customer offerings, technical stacks, and reusable technical modules.

We looked at how the states described in the language are then applied to hosts via Puppet runs, and from these runs, we examined how valuable audit and compliance information can be gathered and stored in PuppetDB. We discussed how code can be managed in environments to allow the gradual release of state changes in a managed way in logical groups of servers that suits your organization’s risk appetite and development structure.

The chapter discussed some misconceptions around Puppet along with the main themes of relevance, complexity, and flexibility. Puppet’s maturity and focus on IaaS make it less fashionable, but using patterns and modules developed by Puppet and the community allows you to use Puppet to its strengths and deliver automation and self-service configuration and compliance to customers. Ensuring clear boundaries and responsibilities so that Puppet can integrate with, and work alongside, other tooling and teams avoids clashes and allows others to interact with Puppet and gain the benefits.

In the next chapter, we will review the major changes that have taken place in Puppet since version 5 and in the latest version, 7. Recommendations of the tooling to use to create an effective development environment will be made, and the creation of lab environments will be outlined and demonstrated. Additional reference sites will be outlined to allow readers to continue their research and stay up to date with developments in Puppet. This will ensure that as we start on the technical details in the following chapters, you will have the capability to test and experiment in your own environment and follow up in more detail on your points of interest.