Puppet6 and Ubuntu18.04 - Control Repo with r10k and Hiera

This is a compilations of notes that leads to the setup of a control repo managed with r10k and the use of Hiera.

This article follows:

[article-card:notes-on-installing-puppet6-on-ubuntu-18-04]

These notes were heavily inspired by various articles and documentation pages, notably:

FreeBSD: Getting started with puppet by Smortex

Why Ubuntu 18.04 and not some generic article? Because I tested on an Ubuntu 18.04 and don’t know what would change for other distros (paths, packages, …)

Creating a control repo managed with Git: r10k

Links:

Documentation: Managing and deploying Puppet code
r10k: repository

The goal these notes targets is simple: we want to use a version control system to manage our Puppet manifest.

“r10k” is a powerful tool to manage your Puppet manifests. It bases its comportment on the branches of a git repository; each branch corresponding to an environment. This makes testing new things really easy.

Setup

Links:

r10k: Quickstart

Install r10k on Puppet server:

# Well, installing gems with sudo is discouraged, but here, we need a system wide install
sudo gem install r10k

The control-repo

Links:

Sample control repo

Clone the control repo template anywhere on your host:

git clone https://github.com/puppetlabs/control-repo
cd control-repo

Configure r10k configuration in /etc/puppetlabs/r10k/r10k.yamlto specify the different paths used:

# The location to use for storing cached Git repos
:cachedir: '/var/puppet/r10k/cache'

# A list of git repositories to create
:sources:
  # This will clone the git repository and instantiate an environment per
  # branch in /usr/local/etc/puppet/environments
  :code:
    remote: '/path/to/the/control-repo/.git'
    basedir: '/etc/puppetlabs/code/environments'

Copy your current manifest:

cp /etc/puppetlabs/code/environments/production/manifests/site.pp manifests/site.pp

Add the modules you previously used and the modules they need in the Puppetfile, present in the repository:

mod 'zleslie-ssh', '1.2.0'

# These modules should be included too, as zleslie-ssh needs them.
mod 'puppetlabs-stdlib', '6.0.0'
mod 'puppetlabs-concat', '6.0.0'

My first thought was “Erk! we have to manually specify the modules dependencies”, and then I was like “Ah, ok, maybe that way I won’t use obscure things to manage my infrastructure”.

Note for a fresh repo:

Maybe you don’t want to keep the template’s history, so you can remove the .git directory and re-create a new repo.

rm -rf .git
git init

# There's no "master" environment in our control repo
git checkout -b production

In any case, commit your work.

Deploy

To deploy the changes in the control repo, use r10k from the control repo:

sudo r10k deploy production --verbose --puppetfile

The changes will be applied in the next 30 mins.

Automate “r10k deploy”

In order to automate the deployment of the changes done in the branches, we will configure a post-commit hook:

#!/usr/bin/env bash
branch=$(git rev-parse --abbrev-ref HEAD)

echo "Deploying environments ${branch} with r10k…"
sudo r10k deploy environment ${branch} --verbose --puppetfile

echo "Caching types for environments ${branch}…"
sudo /opt/puppetlabs/bin/puppet generate types --environment ${branch}

Note: If you use an online VCS for this repo, you may want to create a post-merge hook to, so it deploys on pull.

Each time a commit is done in a branch, the corresponding Puppet environment will be updated.

Each time a branch is pulled with changes, the corresponding Puppet environment will be updated.

You can make a change in a manifest and commit, to see if everything is still fine.

The role/profile organisation

Links:

Documentation: Designing system configs: roles and profiles

When we’re speaking of file organisation, everyone has it’s idea on what is good, and what is not. I propose to follow the Puppet’s recommendations: the Role/Profile model:

Profiles are manifests for a given resource (packages, ssh configuration, etc…).
Roles are composed of different profiles.
A node has one role only.

If we look at the repo template we cloned earlier, we can see this structure:

control-repo
|-- LICENSE
|-- Puppetfile        <-- Where you put modules
|-- README.md
|-- data              <-- Hiera stuff, for later
|   `-- [...]
|-- environment.conf
|-- hiera.yaml
|-- manifests
|   `-- site.pp       <-- All your manifest, for now
|-- scripts
|   `-- [...]
`-- site-modules      <-- Where you should split your site.pp
    |-- profile
    |   `-- manifests
    |       |-- base.pp
    |       `-- example.pp
    `-- role
        `-- manifests
            |-- database_server.pp
            |-- example.pp
            `-- webserver.pp

You should edit a few files from the role and profile manifests to understand what’s going on.

Split `site.pp`

A profile manifest looks like:

We’re going to split site.pp to have something more roleprofilesque:

Remove all files in profile/manifests

Create a base_packages.pp file with all the packages common to all your nodes:

# Notice the class name, it should match the file name.
class profile::base_packages {
  package { 'cloud-init':
    ensure => purged
  }
  # ...
}

Create a motd.pp file with the MOTD configuration
Create a openssh.pp file with the openssh configuration
Create some other <something>.pp files for the rest of the common stuff, leaving only the node-related entries in the site.pp

Now, let’s create a base profile. Puppet classes can inherit from other classes. It’s discouraged to do so, except for the roles, so we’re going to create a base_node role, and a puppet_server role, extending the base role.

Remove all files in roles/manifests

Create a base_node.pp, which will include all the common stuff:

class role::base_node {
  include profile::base_packages
  include profile::motd
  include profile::openssh
  #...
}

Create a puppet_server.pp, which will extend the base node and add the server-specific stuff:

class role::puppet_server inherits role::base_node {
  # include profile::<server-specific profiles, if any>
}

Move the code related to the server from site.pp to the role.
Repeat this for the “puppet-node” host, with a puppet_node role and the puppet-node’s specific configuration.

Let’s update site.pp to use the different roles:

# Puppet server
node puppet {
  include role::puppet_server
}

# Some testing node
node puppet-node {
  include role::puppet_node
}

Cleaner, huh ?

Scratching the Hiera’s surface

Links:

Documentation: Hiera

Hiera is a powerful tool that allows to define some values, per-node.

Because maybe we wanted to use different SSH keys or different usernames for the node and server, we should use Hiera to do so.

Hiera data is stored in data, in the control-repo

control-repo/data/
|-- common.yaml            <-- Values for each nodes
`-- nodes
    `-- example-node.yaml  <-- Values for a specific node

As this is only an intro, we’re not doing anything fancy here, but we’ll specify an user and a public key for all the nodes. This will remove the hardcoded username and ssh key from our manifests.

First, we’ll have to change the user profile to accept variables:

class profile::user (
  String[1] $username,
  String[1] $ssh_key,
) {
  user { $username:
    ensure         => present,
    groups         => [
      'adm',
      'sudo',
    ],
    shell          => '/bin/bash',
    home           => "/home/${username}",
    managehome     => true,
    password       => '*',
    purge_ssh_keys => true,
  }

  $ssh_elements = $ssh_key.split(' ')

  ssh_authorized_key { $ssh_elements[2]:
    ensure => present,
    user   => $username,
    type   => $ssh_elements[0],
    key    => $ssh_elements[1],
  }
}

And then, we tell Hiera which values to assign to the variables:

# data/common.yaml
---
profile::user::username: johndoe
profile::user::ssh_key: ssh-rsa SOME_LONG_KEY john_doe@somewhere

Commit and apply.

Conclusion

We now have a versioned control repo with a role/profile organisation. The deployment of the control repo is almost automated via git hooks and r10k.

What’s next?

Create linters and checkers to test the manifests before they are deployed.
Have a Git server on the puppet server to deploy on push.
Write your own modules.

Puppet6 and Ubuntu18.04 - Control Repo with r10k and Hiera

Creating a control repo managed with Git: r10k

Setup

The control-repo

Deploy

Automate “r10k deploy”

The role/profile organisation

Split site.pp

Scratching the Hiera’s surface

Conclusion

What’s next?

Split `site.pp`