Fleet Management

WARNING

This page is work in progress. Related issue: #41 (fleet monitoring), #43 (configuration management)

Organisations that wish to switch many computers from Windows to Linux need usually tooling to manage the Linux fleet efficiently. The larger the fleet is, the more relevant automation becomes. In this context, fleet management means:

• provisioning of new computers (bare-metal or virtual) with EU OS
• enrollment of the new computers into the fleet
• version monitoring
• force updates
• execute commands from remote
• support for special-casing individual or groups of computers (sub fleets)

Unlike servers, desktop computer are not always powered on and connected to the network ("offline-first" paradigm). This is why fleet management for EU OS should rely on computers pulling tasks and pushing information pro-actively instead of servers attempting connections to computers to pull and push.

foreman-katello installation

Base OS: Alma Linux 9.4

Minimum Requirements for Foreman:

• 4 CPU
• 24 GB RAM
• 200 GB HD (or more, depends on how many and what repositories are mirrored)

Repositories: /etc/yum.repos.d/foreman.repo

[foreman]
name=Foreman 3.14
baseurl=https://yum.theforeman.org/releases/3.14/el9/$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman

[foreman-source]
name=Foreman 3.14 - source
baseurl=https://yum.theforeman.org/releases/3.14/el9/source
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman

(maybe: dnf install foreman-release)

/etc/yum.repos.d/foreman-plugins.repo:

[foreman-plugins]
name=Foreman plugins 3.14
baseurl=https://yum.theforeman.org/plugins/3.14/el9/$basearch
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman

[foreman-plugins-source]
name=Foreman plugins 3.14 - source
baseurl=https://yum.theforeman.org/plugins/3.14/el9/source
enabled=0
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman

/etc/yum.repos.d/katello.repo

# Place this file in your /etc/yum.repos.d/ directory

[katello]
name=Katello 4.16
baseurl=https://yum.theforeman.org/katello/4.16/katello/el9/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=1
gpgcheck=1

# Candlepin RPMs as supported by Katello - This is a coordinated
# copy of Candlepin's packages in order to ensure compatibility

[candlepin]
name=Candlepin: an open source entitlement management system.
baseurl=https://yum.theforeman.org/candlepin/4.4/el9/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-candlepin
enabled=1
gpgcheck=1

[pulpcore]
name=pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
baseurl=https://yum.theforeman.org/pulpcore/3.63/el9/$basearch/
gpgkey=https://yum.theforeman.org/pulpcore/3.63/GPG-RPM-KEY-pulpcore
enabled=1
gpgcheck=1

# source repositories

[katello-source]
name=Katello 4.16 Source
baseurl=https://yum.theforeman.org/katello/4.16/katello/el9/source/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=0
gpgcheck=1

[candlepin-source]
name=Katello Candlepin source
baseurl=https://yum.theforeman.org/candlepin/4.16/el9/source/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-candlepin
enabled=0
gpgcheck=1

[pulpcore-source]
name=pulpcore source
baseurl=https://yum.theforeman.org/pulpcore/3.63/el9/source/
gpgkey=https://yum.theforeman.org/pulpcore/3.63/GPG-RPM-KEY-pulpcore
enabled=0
gpgcheck=1

or just install the RPMs:

dnf install https://yum.theforeman.org/releases/3.14/el9/x86_64/foreman-release.rpm
dnf install https://yum.theforeman.org/katello/4.16/katello/el9/x86_64/katello-repos-latest.rpm

additionally: dnf install https://yum.puppet.com/puppet8-release-el-9.noarch.rpm

Updating Package cache: dnf makecache

installation foreman-katello

# as root
dnf install foreman-installer-katello

Now install + configure foreman + katello:

# as root
foreman-installer \
  --scenario katello \
  --tuning development \
  --skip-checks-i-know-better \
  --foreman-initial-organization=Training \
  --foreman-initial-location=Hypervisor \
  --foreman-initial-admin-username admin \
  --foreman-initial-admin-password=s3cr3t \
  --foreman-proxy-content-enable-ostree=true \
  --enable-foreman-plugin-webhooks \
  --enable-foreman-proxy-plugin-shellhooks \
  --enable-foreman-plugin-templates \
  --enable-puppet \
  --puppet-server=true \
  --enable-foreman-plugin-puppet \
  --foreman-proxy-puppet=true \
  --foreman-proxy-puppetca=true \
  --enable-foreman-cli-puppet \
  --enable-foreman-plugin-ansible \
  --enable-foreman-proxy-plugin-ansible \
  --enable-foreman-cli-ansible \
  --foreman-proxy-plugin-remote-execution-script-mode=pull-mqtt \
  --enable-foreman-plugin-salt \
  --enable-foreman-proxy-plugin-salt

For this PoC we do not need DNS, TFTP nor DHCP, so do not activate it.

Now it needs some time. foreman-katello will be installed, configured and started.

Check output for errors. If you encounter some errors, read them, understand them and fix them. You can omit errors regarding sssd_kcm.

foreman is installed and setup is completed.

Now we need to configure some stuff.

Create Repository

Initially foreman only knows Alma Linux and you only can create hosts of OS type "Alma Linux". You have to create a new content.

In WebUI go to "content > products" and click on "create product".

Fill out the form. The name automatically fills the "label" field. Any other fields are optional. Fill them if needed or known. Klick Save at the bottom of the form.

Now add the repository you want to clone. Click on "New Repository". The Name field will fill the labal field, too. choose the type at the bottom and fill the fields Upstream-URL and any needed field, too. For an OSTree Repository choose "ostree".

preparing puppet master/foreman

NOTE

Note that Puppet may not be necessary for basic fleet monitoring and execution of remote commands.

hammer puppet-environment create \
 	--organizations Training \
  --locations Hypervisor \
 	--name management

hammer proxy import-classes \
 	--puppet-environment production \
 	--location Hypervisor \
 	--organization Training \
 	--name fleet.testing.eu-os.eu

hammer proxy import-classes \
  --puppet-environemt management \
  --location Hypervisor \
  --organization Training \
  --name fleet.testing.eu-os.eu

Check if directories "management" and "production" exists in /etc/puppetlabs/code/environments If not, create them.

mkdir /etc/puppetlabs/code/environments/{production,management}

Add new environment "management" in katello:

hammer host update \
  --name fleet.testing.eu-os-eu \
  --puppet-environment management

Register Clients

on client side

Install puppet yum repo: dnf install https://yum.puppet.com/puppet8-release-el-9.noarch.rpm && dnf makecache

puppet agent has to be installed first by dnf install puppet-agent

in file /etc/puppet/puppet.conf:

[main]
server = fleet.testing.eu-os.eu

Use the FQDN of your foreman/puppet server

registering client in puppet master/foreman

command puppet ssl bootstrap waitforcert 120 creates a SSL certificate. The certificate needs to be signed by puppet master.

on puppet master/foreman

puppetserver ca sign --certname <hostname>

adding custom facts

via subscription-manager

WARNING

With more recent bootc and rhsm versions, there is a better way to do it. So this section should be replaced.

Add a file to /etc/rhsm/facts/ what contains your custom facts in JSON formated.

Example /etc/rhsm/facts/bootc.facts:

{
    "bootc.booted.image": "registry.gitlab.com/eu-os/eu-os/euos-hackdays",
    "bootc.booted.version": "20250603_1117",
    "bootc.booted.digest": "sha256:89f696ad018ae079f10532b26b5266eddc3ec21a7603f89b3755e6a4c399d9c8",
    "bootc.staged.image": null,
    "bootc.staged.version": null,
    "bootc.staged.digest": null,
    "bootc.rollback.image": "registry.gitlab.com/eu-os/eu-os/euos-hackdays",
    "bootc.rollback.version": "20250603_0944",
    "bootc.rollback.digest": "sha256:89e7a86541e163ca2c014e792ca620aa761d9466d5af4449156511ba4e2831ed",
    "bootc.available.image": null,
    "bootc.available.version": null,
    "bootc.available.digest": null
}

subscription manager reads the file and sends informations into foreman.

puppet

puppet has the facter what provides facts to puppet master. you can add custom facts, too.

Just write the data in /etc/facter/facts.d/ and facter should read it.

Puppet facter accepts JSON and YAML formated files. It also accepts txt files with key=value

Example /etc/facter/facts.d/bootc.json:

{
 "bootc.booted.image": "registry.gitlab.com/eu-os/eu-os/euos-hackdays",
 "bootc.booted.version": "20250603_1117",
 "bootc.booted.digest": "sha256:89f696ad018ae079f10532b26b5266eddc3ec21a7603f89b3755e6a4c399d9c8",
 "bootc.staged.image": null,
 "bootc.staged.version": null,
 "bootc.staged.digest": null,
 "bootc.rollback.image": "registry.gitlab.com/eu-os/eu-os/euos-hackdays",
 "bootc.rollback.version": "20250603_0944",
 "bootc.rollback.digest": "sha256:89e7a86541e163ca2c014e792ca620aa761d9466d5af4449156511ba4e2831ed",
 "bootc.available.image": null,
 "bootc.available.version": null,
 "bootc.available.digest": null
}

Without Puppet

Quote from Foreman community member:

The bootc integration with Foreman/Katello relies on a minimum version of bootc that has the capability to create the bootc facts file for rhsm. Also, to be clear, you do need the Katello plugin installed with Foreman.

With this version of bootc, there should be no setup necessary besides registration.

The version of bootc you need is 1.1.5-2 or greater. This should be available at least in CentOS Stream 10, along with (I think) RHEL 9.6 and 10.

To verify the bootc rhsm functionality, check for the rhsm facts command under bootc internals:

$ bootc internals
Hidden, internal only options

Usage: bootc internals <COMMAND>

Commands:
  systemd-generator
...
  publish-rhsm-facts        Publish subscription-manager facts to
/etc/rhsm/facts/bootc.facts
...

You do not need to run this command - systemd runs it for you.

Note that the facts don't upload automatically after reboot (and hence bootc switch). rhsmcertd should trigger a fact reupload after 4 hours by default. Or, you can set up a systemd oneshot directive to run subscription-manager facts --update.

Connections from the bootc host to the Foreman all happen via the subscription-manager registration. Through subscription-manager / rhsm, the host knows which API endpoints to hit on the Foreman to do things like upload the package profile. Connections from Foreman to the bootc host happen via the Foreman Remote Execution plugin, which is installed with Katello by default. If using global registration via Foreman, ssh keys are set up by default on the host and Foreman will automatically be able to run remote jobs. Note that SSH isn't the only available remote execution method. There is also Pull Mode, which the docs have details about.