Fleet Management

WARNING

This page is work in progress. Related issue: #41 (fleet monitoring), #43 (configuration management)

Organisations that wish to switch many computers from Windows to Linux need usually tooling to manage the Linux fleet efficiently. The larger the fleet is, the more relevant automation becomes. In this context, fleet management means:

• provisioning of new computers (bare-metal or virtual) with EU OS
• enrollment of the new computers into the fleet
• version monitoring
• force updates
• execute commands from remote
• support for special-casing individual or groups of computers (sub fleets)
• privileged terminal sessions on fleet computers

Unlike servers, desktop computer are not always powered on and connected to the network ("offline-first" paradigm). This is why fleet management for EU OS should rely on computers pulling tasks and pushing information pro-actively instead of servers attempting connections to computers to pull and push.

EU OS proposes to employ Foreman (or the downstream versions Orcharhino/Redhat Satellite) for fleet management:

Foreman is a complete lifecycle management tool for physical and virtual servers. We give system administrators the power to easily automate repetitive tasks, quickly deploy applications, and proactively manage servers, on-premise or in the cloud.

Source: https://www.theforeman.org/ (companies in the EU providing professional services)

With the katello plugin, Foreman also supports:

• retrieving bootc image status of fleet computers
• OCI container registry with proxies to support fleet updates at scale
• flatpak registry to support flatpak app upates at scale

Note that Foreman supports not only fleet computers running Fedora, Redhat and other derivates, but also Ubuntu, openSUSE, Debian, and a few others.

Setup of Foreman with Katello

Foreman offers RPM repos than contain the necessary packages to run Foreman. With this installation method, the choice of the Foreman host OS is limited to those supported by the repos. For this Proof-of-Concept, AlmaLinux 9.6 is chosen, but RHEL should work as well.

Optimal requirements for Foreman for this PoC:

• 4 CPUs
• 24 GB RAM
• 200 GB HD (or more, depends on how many and what repositories are mirrored)

Minimal requirements for Foreman for this PoC (Robert's test server):

• 4 CPUs
• 8 GB RAM (+20 GB swapfile)
• 250 GB HD (40 GB effectively used)

INFO

Foreman supports many different plugins to interact with fleet computers, such as:

• puppet
• salt
• ansible

Not all of those are compatible with fleet computers that are not always on (e.g. ansible does not as of 2022). For this reason, this PoC relies on remote execution of simple (shell) scripts that already support the pull-mode important for fleet computers that are often disconnected or powered off.

# add extra packages needed for e.g. certbot
sudo dnf install -y epel-release
sudo dnf config-manager --set-enabled epel

# add 20 GiB swap
sudo touch /swapfile
sudo chattr +C /swapfile 
sudo lsattr /swapfile
sudo fallocate -l 20G /swapfile
sudo lsattr /swapfile
sudo chmod 600 /swapfile
sudo lsattr /swapfile  # make sure you see the letter C in the results: ---------------C------ /swapfile
sudo mkswap /swapfile # output example: Setting up swapspace version 1, size = 200 GiB (214748360704 bytes)
sudo swapon /swapfile

echo "/swapfile swap swap defaults 0 0" | sudo tee -a /etc/fstab

# setup repos for Foreman with Katello
sudo dnf clean all
sudo dnf install -y https://yum.theforeman.org/releases/3.15/el9/x86_64/foreman-release.rpm
sudo dnf install -y https://yum.theforeman.org/katello/4.17/katello/el9/x86_64/katello-repos-latest.rpm
sudo dnf install -y https://yum.puppet.com/puppet8-release-el-9.noarch.rpm

sudo dnf repolist enabled # verify repos
sudo dnf upgrade

# install the foreman installer
sudo dnf install -y foreman-installer-katello

sudo foreman-installer \
  --scenario katello \
  --tuning development \
  --foreman-initial-admin-username admin \
  --foreman-initial-admin-password admin-password-change-it-here \
  --foreman-initial-organization Public_Organisation \
  --foreman-initial-location Office_Location \
  --enable-foreman-proxy-plugin-remote-execution-script \
  --foreman-proxy-plugin-remote-execution-script-mode pull-mqtt

# open firewall
sudo firewall-cmd --zone=public --add-service=https --permanent
sudo firewall-cmd --zone=public --add-service=http --permanent
sudo firewall-cmd --zone=public --add-service=mqtt --permanent
sudo firewall-cmd --reload
# verify
sudo firewall-cmd --zone=public --list-services # or
sudo firewall-cmd --zone=public --list-ports
sudo firewall-cmd --list-all

The following foreman-installer options may be interesting as well:

sudo foreman-installer \
  # ... see above
  --foreman-server-ssl-cert /etc/letsencrypt/live/$HOSTNAME/cert.pem \
  --foreman-server-ssl-chain /etc/letsencrypt/live/$HOSTNAME/chain.pem \
  --foreman-server-ssl-key /etc/letsencrypt/live/$HOSTNAME/privkey.pem \
  --foreman-proxy-foreman-ssl-ca /etc/ssl/certs/ca-bundle.crt \
  --foreman-proxy-content-enable-ostree true \
  --enable-foreman-plugin-webhooks \
  --enable-foreman-plugin-templates \
  --enable-foreman-plugin-puppet \
  --foreman-proxy-puppet=true \
  --foreman-proxy-puppetca=true \
  --enable-foreman-cli-puppet \
  --skip-checks-i-know-better
  
  
# testing the configuration
hammer ping
hammer --ssl-ca-file /etc/pki/ca-trust/extracted/pem/directory-hash/ISRG_Root_X1.pem ping
sudo foreman-maintain health check

Note that Foreman will generate in the process a self-signed certificate if not given a custom SSL. This certificate is not only relevant for the HTTPS connection of the admin interface and API used by the fleet computers, but also on a lower level. So during tests, the use of LetsEncrypt turned out to be a not so good idea, as it would also delegate trust on the fleet management and enrollment of new devices to LetsEncrypt. So a better method is for a PoC to use an auto-generated self-signed certificate and add the corresponding public root certificate to the trust store of the OCI image deloyed to the fleet computers.

Learn more about using LetsEncrypt even if it is a bad idea

sudo yum install -y certbot # requires epel repo
HOSTNAME=$(hostname -f)

# avoid acme challenge being proxied to foreman
echo "ProxyPass /.well-known/acme-challenge !" | sudo tee -a /etc/httpd/conf.d/05-foreman.d/01-enable-certbot-http-challenge.conf
sudo systemctl restart httpd
sudo certbot certonly --agree-tos --email your@email.local -d $HOSTNAME --webroot-path /var/lib/foreman/public

# recall foreman-installer
sudo foreman-installer \
  # ... same options as above + those here below
  --foreman-server-ssl-cert /etc/letsencrypt/live/$HOSTNAME/cert.pem \
  --foreman-server-ssl-chain /etc/letsencrypt/live/$HOSTNAME/chain.pem \
  --foreman-server-ssl-key /etc/letsencrypt/live/$HOSTNAME/privkey.pem \
  --foreman-proxy-foreman-ssl-ca /etc/ssl/certs/ca-bundle.crt

Warning: you need to change :ssl_ca_file with sudo vi /etc/foreman/settings.yaml

# :ssl_ca_file: /etc/foreman/proxy_ca.pem
:ssl_ca_file: /etc/ssl/certs/ca-bundle.crt

hammer ping
hammer --ssl-ca-file /etc/pki/ca-trust/extracted/pem/directory-hash/ISRG_Root_X1.pem ping
sudo foreman-maintain health check

hammer conf: /etc/hammer/cli.modules.d/foreman.yml

Note that also the PEM certificate published under https://$HOSTNAME/pub/katello-rhsm-consumer is wrong and needs manual correction.

References:

• add swap file
• https://docs.theforeman.org/3.15/Quickstart/index-katello.html
• https://docs.theforeman.org/3.15/Installing_Server/index-katello.html
• https://docs.theforeman.org/nightly/Managing_Content/index-katello.html#con_managing-ostree-content_content-management

Enroll Fleet Computers

The fleet computers should have subscription-manager (sometimes abbreviated rhsm for Redhat subscription manager) and bootc with rhsm support.

Both Fedora and Almalinux have those packages in the standard repos. However, Fedora is not compiling rhsm support for bootc. This custom build offers bootc with rhsm for Fedora 42: https://copr.fedorainfracloud.org/coprs/rriemann/eu-os/package/bootc/

With the following bluebuild recipe module, bootc is switched in the image for the custom version above:

# […]
modules:
  # […]
  - type: dnf
    no-cache: true
    repos:
      copr:
        - rriemann/eu-os
    replace:
      - from-repo: copr:copr.fedorainfracloud.org:rriemann:eu-os
        packages:
          - new: bootc
            old: bootc

To enroll a test computer to the fleet, open a shell and follow those steps:

$FOREMAN=my-foreman-host
curl -ks https://$FOREMAN/pub/katello-rhsm-consumer | sudo bash

# maybe reboot

sudo subscription-manager register --username admin --password admin-password

The OCI image recipe should enable the respective systemd service to export bootc metadata upon boot to /etc/rhsm/facts/bootc.facts

# […]
modules:
  # […]
  - type: systemd
    system:
      enabled:
        # upon each reboot (restart?) `bootc internals publish-rhsm-facts` exports data to /etc/rhsm/facts/bootc.facts for the subcription manager
        - bootc-publish-rhsm-facts.service

A more sophisticated method entails to build the OCI image with the rpms at https://FOREMAN/pub/ to import the certificates and to create an activation key from the Foreman GUI (Content -> LifeCycle -> Activation Keys) for enrollement.

After the enrollement, the fleet machine should be visible in Foreman GUI. The details in the Foreman GUI of the given machine should also specifify the current OCI image and its manifest SHA256.

Manage Configuration and Execute Commands

WARNING

This page is work in progress. Related issue: #43

• foreman comes with some remote execution scripts for bootc preconfigured (bootc-update or so)
• TODO

References:

• https://docs.fedoraproject.org/en-US/bootc/dynamic-reconfiguration/
• https://docs.fedoraproject.org/en-US/bootc/building-containers/#_general_configuration_guidance

Cockpit

• https://cockpit-project.org/
• https://theforeman.org/plugins/foreman_remote_execution/
• TODO