EU OS

Appearance

Fleet Management ​

WARNING

This page is work in progress. Related issue: #41 (fleet monitoring), #43 (configuration management)

Organisations that wish to switch many computers from Windows to Linux need usually tooling to manage the Linux fleet efficiently. The larger the fleet is, the more relevant automation becomes. In this context, fleet management means:

  • provisioning of new computers (bare-metal or virtual) with EU OS
  • enrollment of the new computers into the fleet
  • version monitoring
  • force updates
  • execute commands from remote
  • support for special-casing individual or groups of computers (sub fleets)
  • privileged terminal sessions on fleet computers

Unlike servers, desktop computer are not always powered on and connected to the network ("offline-first" paradigm). This is why fleet management for EU OS should rely on computers pulling tasks and pushing information pro-actively instead of servers attempting connections to computers to pull and push.

EU OS proposes to employ Foreman (or the downstream versions Orcharhino/Redhat Satellite) for fleet management:

Foreman is a complete lifecycle management tool for physical and virtual servers. We give system administrators the power to easily automate repetitive tasks, quickly deploy applications, and proactively manage servers, on-premise or in the cloud.

Screenshot of Foreman Dashboard (Source: https://www.theforeman.org/)

Source: https://www.theforeman.org/ (companies in the EU providing professional services)

With the katello plugin, Foreman also supports:

  • retrieving bootc image status of fleet computers
  • OCI container registry with proxies to support fleet updates at scale
  • flatpak registry to support flatpak app upates at scale

Note that Foreman supports not only fleet computers running Fedora, Redhat and other derivates, but also Ubuntu, openSUSE, Debian, and a few others.

Setup of Foreman with Katello ​

This section assumes that the server has been prepared already with an empty AlmaLinux VM.

INFO

Foreman supports many different plugins to interact with fleet computers, such as:

Not all of those are compatible with fleet computers that are not always on (e.g. ansible does not as of 2022). For this reason, this PoC relies on remote execution of simple (shell) scripts that already support the pull-mode important for fleet computers that are often disconnected or powered off.

The following setup does the enrollment of the Foreman VM to the EU OS realm of the FreeIPA VM. For auto detection of the realm to work, the DNSMasq DNS Server managed by the virtualisation service must forward DNS requests for the domain eu-os.internal to FreeIPA. To do so, login to the EU OS Server (not the VM) and add one line with a forwarder tag to the XML definition of the network. You can do so with virsh net-edit default.

xml
<network>
  <name>default</name>
  <uuid>686535f8-0212-405a-8a01-23f8b5037e7f</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:00:00:10'/>
  <dns>
    <forwarder domain='eu-os.internal' addr='192.168.122.112'/>
    <host ip='192.168.122.111'>
      <hostname>fleet.eu-os.internal</hostname>
    </host>
    <host ip='192.168.122.112'>
      <hostname>users.eu-os.internal</hostname>
    </host>
  </dns>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <host mac='52:54:00:00:00:11' name='fleet' ip='192.168.122.111'/>
      <host mac='52:54:00:00:00:12' name='users' ip='192.168.122.112'/>
    </dhcp>
  </ip>
</network>

To prepare automatic provisioning of hosts, Foreman requires two templates Kickstart PXEGrub2 EU OS.erb and Kickstart EU OS.erb that should be in the $HOME folder to be picked up by hammer.

sh
sudo hostnamectl hostname fleet.eu-os.internal

# convenience tools
sudo dnf install -y nmap

# optionally setup swap using vdb
sudo cfdisk /dev/vdb # interactive! select swap!
sudo mkswap /dev/vdb1
sudo swapon /dev/vdb1
echo "/dev/vdb1 swap swap defaults 0 0" | sudo tee -a /etc/fstab

# disable eth1 connection (physical port)
sudo nmcli connection modify 'Wired connection 1' connection.autoconnect false
sudo nmcli connection down 'Wired connection 1'

FOREMAN_IP=192.0.2.1

sudo nmcli connection add type ethernet ifname eth1 con-name static-eth1 ipv4.addresses $FOREMAN_IP/24 ipv4.gateway $FOREMAN_IP ipv4.method manual
sudo nmcli c mod static-eth1 ipv4.never-default true
sudo nmcli c mod static-eth1 ipv6.never-default true
sudo nmcli c mod static-eth1 ipv4.dns "192.168.122.1"
sudo nmcli connection up static-eth1

# package forwarding from eth1 to eth0
sudo sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/50-enable-ipv4-forward.conf
sudo nft add table ip nat
sudo nft add chain ip nat postrouting { type nat hook postrouting priority 100 \; }
sudo nft add rule ip nat postrouting oif "eth0" masquerade
sudo nft add table ip filter
sudo nft add chain ip filter forward { type filter hook forward priority 0 \; policy accept \; }
sudo nft add rule ip filter forward iif "eth1" oif "eth0" accept
sudo nft add rule ip filter forward iif "eth0" oif "eth1" ct state established,related accept
sudo nft list ruleset | sudo tee -a /etc/sysconfig/nftables.conf # make it persistent
sudo systemctl enable --now nftables

echo "$FOREMAN_IP fleet.eu-os.internal" | sudo tee -a /etc/hosts

sudo dnf clean all
sudo dnf install -y https://yum.theforeman.org/releases/3.16/el9/x86_64/foreman-release.rpm
sudo dnf install -y https://yum.theforeman.org/katello/4.18/katello/el9/x86_64/katello-repos-latest.rpm
sudo dnf install -y https://yum.puppet.com/puppet8-release-el-9.noarch.rpm

sudo dnf repolist enabled # verify repos
sudo dnf upgrade

sudo dnf install -y foreman-installer-katello ipa-client

sudo ipa-client-install # no, yes (should find domain automatically if dns is configured to ipa host)

sudo ipa-getcert request \
  -f /etc/pki/tls/certs/foreman.crt \
  -k /etc/pki/tls/private/foreman.key \
  -F /etc/pki/tls/certs/idx.bundle.pem \
  -K HTTP/fleet.eu-os.internal \
  -D fleet.eu-os.internal \
  -u digitalSignature \
  -u nonRepudiation \
  -u keyEncipherment \
  -u dataEncipherment \
  -U id-kp-serverAuth \
  -U id-kp-clientAuth \
  -U id-kp-codeSigning \
  -U id-kp-emailProtection

sudo katello-certs-check \
  -c /etc/pki/tls/certs/foreman.crt \
  -k /etc/pki/tls/private/foreman.key \
  -b /etc/pki/tls/certs/idx.bundle.pem

sudo foreman-prepare-realm admin realm-smart-proxy
sudo cp /home/almalinux/freeipa.keytab /etc/foreman-proxy
sudo chown foreman-proxy:foreman-proxy /etc/foreman-proxy/freeipa.keytab

# proceed only if katello check is positive

FOREMAN_PASSWORD=so-simple-in-2025

# note that 192.168.122.1 is the IP of the libvirt dnsmasq DNS server

sudo foreman-installer \
  --scenario katello \
  --tuning development \
  --certs-server-cert "/etc/pki/tls/certs/foreman.crt" \
  --certs-server-key "/etc/pki/tls/private/foreman.key" \
  --certs-server-ca-cert "/etc/pki/tls/certs/idx.bundle.pem" \
  --foreman-initial-admin-username admin \
  --foreman-initial-admin-password $FOREMAN_PASSWORD \
  --foreman-initial-organization Public_Organisation \
  --foreman-initial-location Office_Location \
  --enable-foreman-proxy-plugin-remote-execution-script \
  --foreman-proxy-plugin-remote-execution-script-mode pull-mqtt \
  --enable-foreman-plugin-templates \
  --enable-foreman-plugin-discovery \
  --enable-foreman-proxy-plugin-discovery \
  --enable-foreman-cli-discovery \
  --foreman-proxy-plugin-discovery-install-images true \
  --foreman-proxy-dns true \
  --foreman-proxy-dns-forwarders 192.168.122.112 \
  --foreman-proxy-dhcp true \
  --foreman-proxy-dhcp-managed true \
  --foreman-proxy-dhcp-range "192.0.2.100 192.0.2.150" \
  --foreman-proxy-dhcp-gateway 192.0.2.1 \
  --foreman-proxy-dhcp-interface eth1 \
  --foreman-proxy-dhcp-nameservers 192.0.2.1 \
  --foreman-proxy-tftp true \
  --foreman-proxy-tftp-managed true \
  --foreman-proxy-tftp-servername 192.0.2.1 \
  --foreman-proxy-realm true \
  --foreman-proxy-realm-keytab /etc/foreman-proxy/freeipa.keytab \
  --foreman-proxy-realm-principal realm-smart-proxy@EU-OS.INTERNAL \
  --foreman-proxy-realm-provider freeipa

sudo cp /etc/ipa/ca.crt /etc/pki/ca-trust/source/anchors/ipa.crt
sudo update-ca-trust extract
sudo systemctl restart foreman-proxy

# few tests
hammer ping
sudo foreman-maintain health check

# convenience
hammer defaults add --param-name organization --param-value "Public_Organisation"
sudo vi /etc/hammer/cli.modules.d/foreman.yml
# add under :foreman:
#  :username: 'admin'
#  :password: 'so-simple-in-2025'

hammer proxy refresh-features --id 1


# setup EU OS
hammer product create --name "EU OS"
hammer repository create \
  --product "EU OS" \
  --name "EU OS Workspace Demo Images" \
  --label "demo" \
  --description "Find more information at https://gitlab.com/eu-os/workspace-images/eu-os-base-demo/" \
  --content-type docker \
  --download-policy immediate \
  --mirroring-policy mirror_content_only \
  --url "https://registry.gitlab.com" \
  --docker-upstream-name "eu-os/workspace-images/eu-os-base-demo/eu-os-demo" \
  --exclude-tags '*-cache' \
  --include-tags latest
hammer repository synchronize --product "EU OS"
hammer repository info --id 1 # check repo
hammer lifecycle-environment update --name Library --registry-unauthenticated-pull true

hammer content-view create --name "EU OS Demo Content"
hammer content-view add-repository --name "EU OS Demo Content" --product "EU OS" --repository "EU OS Workspace Demo Images"
hammer lifecycle-environment create --name "EU OS Demo" --label "demo" --prior "Library" --registry-unauthenticated-pull true
hammer content-view publish --name "EU OS Demo Content"
hammer content-view version promote --content-view "EU OS Demo Content" --version "1.0" --to-lifecycle-environment "EU OS Demo"

# keys
# hammer activation-key create --name "eu-os-demo-key" --description "Key to use with EU OS Demo" --lifecycle-environment "EU OS Demo" --content-view "EU OS Demo Content" --unlimited-hosts

# activation key
# hammer activation-key add-subscription --name "eu-os-demo-key" --quantity "1" --subscription-id "1"

hammer subnet create \
  --locations "Office_Location" \
  --name "Provisioning_LAN" \
  --network "192.0.2.0" \
  --mask "255.255.255.0" \
  --network-type "IPv4" \
  --gateway "192.0.2.1" \
  --dns-primary "192.168.122.1" \
  --boot-mode "DHCP" \
  --ipam "DHCP" \
  --domain-ids "1" \
  --tftp-id "1" \
  --dhcp-id "1" \
  --discovery-id "1"

# Foreman subscription-manager matches /etc/os-release to such os profiles, but we disable for the PoC
# see: https://github.com/candlepin/subscription-manager/blob/1563211da957e3f901d59df099cc2ede96958c75/src/rhsmlib/facts/hwprobe.py#L128
hammer settings set --name ignore_facts_for_operatingsystem --value true

hammer os create \
  --name "EU_OS" \
  --location-title "Office_Location" \
  --architectures "x86_64" \
  --description "EU OS Demo OS" \
  --family "Redhat" \
  --password-hash "SHA512" \
  --release-name "Demo Release" \
  --partition-tables "Kickstart default encrypted" \
  --major "42"
  # --provisioning-template "Kickstart PXEGrub2 EU OS"

hammer os set-parameter \
  --operatingsystem "EU OS Demo OS" \
  --name "time-zone" \
  --parameter-type string \
  --value "Europe/Brussels"

hammer os set-parameter \
  --operatingsystem "EU OS Demo OS" \
  --name "ostree-image-repo" \
  --parameter-type string \
  --value "fleet.eu-os.internal/public_organisation/eu_os"

hammer os set-parameter \
  --operatingsystem "EU OS Demo OS" \
  --name "ostree-image" \
  --parameter-type string \
  --value "demo:latest"

hammer os set-parameter \
  --operatingsystem "EU OS Demo OS" \
  --name "autopart_options" \
  --parameter-type string \
  --value "--type=btrfs"

hammer os set-parameter \
  --operatingsystem "EU OS Demo OS" \
  --name "medium_uri" \
  --parameter-type string \
  --value "http://fleet.eu-os.internal/pub/installation-media/eu_os/"

# TODO check what the meaning of path is here
hammer medium create \
  --locations "Office_Location" \
  --name "EU OS Demo Medium" \
  --os-family "Redhat" \
  --operatingsystems "EU OS Demo OS" \
  --path "http://fleet.eu-os.internal/pub/installation-media/eu_os/"

hammer template create \
  --locations "Office_Location" \
  --file "/home/almalinux/Kickstart EU OS.erb" \
  --name "Kickstart EU OS" \
  --type "provision" \
  --operatingsystems "EU OS Demo OS"

hammer template create \
  --locations "Office_Location" \
  --file "/home/almalinux/Kickstart PXEGrub2 EU OS.erb" \
  --name "Kickstart PXEGrub2 EU OS" \
  --type "PXEGrub2" \
  --operatingsystems "EU OS Demo OS"

# hammer template create \
#   --locations "Office_Location" \
#   --file "/home/almalinux/freeipa_register_ostree.erb" \
#   --name "freeipa_register_ostree" \
#   --type "snippet" \
#   --operatingsystems "EU OS Demo OS"

hammer os update --title "EU OS Demo OS" --provisioning-templates "Kickstart PXEGrub2 EU OS,Kickstart EU OS"

# check OS with previsioning templates
hammer os info --title "EU OS Demo OS"

hammer realm create \
  --name "EU-OS.INTERNAL" \
  --realm-type "FreeIPA" \
  --locations "Office_Location" \
  --realm-proxy-id 1

hammer hostgroup create \
  --name "EU OS Hostgroup" \
  --description "EU OS Hostgroup" \
  --location "Office_Location" \
  --lifecycle-environment "EU OS Demo" \
  --medium "EU OS Demo Medium" \
  --operatingsystem "EU OS Demo OS" \
  --architecture "x86_64" \
  --content-view "EU OS Demo Content" \
  --subnet "Provisioning_LAN" \
  --root-password "qwerqwer" \
  --partition-table "Kickstart default encrypted" \
  --realm "EU-OS.INTERNAL" \
  --domain "eu-os.internal" \
  --pxe-loader "Grub2 UEFI" \
  --content-source "fleet.eu-os.internal"

# this is need to avoid awkward partitions of demo device with 2nd smaller SSD
# could also be done per host, but for demo, this is faster
# hammer hostgroup set-parameter \
#   --hostgroup "EU OS Hostgroup" \
#   --name "ignoredisk_options" \
#   --parameter-type string \
#   --value "--drives=sdb"

hammer template build-pxe-default
erb
<%#
kind: PXEGrub2
name: Kickstart PXEGroup2 EU OS
model: ProvisioningTemplate
oses:
- EU_OS
description: |
  The template to render Grub2 bootloader configuration for kickstart based distributions.
  The output is deployed on the host's subnet TFTP proxy.
-%>
<%
os_major  = @host.operatingsystem.major.to_i
os_name   = @host.operatingsystem.name
%>
# This file was deployed via '<%= template_name %>' template
<%
    linuxcmd = "linuxefi"
    initrdcmd = "initrdefi"
-%>

set default=<%= host_param('default_grub_install_entry') || 3 %>
set timeout=<%= host_param('loader_timeout') || 10 %>

menuentry '<%= template_name %>' {
  <%= linuxcmd %> <%= @kernel %> inst.debug inst.stage2=<%= medium_uri %> inst.ks=<%= foreman_url('provision') %> inst.ks.sendmac ip=dhcp inst.noverifyssl
  <%= initrdcmd %> <%= @initrd %>
}
erb
<%#
kind: provision
name: Kickstart EU OS
model: ProvisioningTemplate
oses:
- EU_OS
description: |
  EU OS Provisioning Template using Red Hat anaconda installer

  This Template accepts the following parameters:
  - lang: string (default="en_US.UTF-8")
  - keyboard: string (default="gb")
  - time-zone: string (default="UTC")
-%>
<%
  install_disk = host_param('install-disk') ? host_param('install-disk') : "nvme0n1"
  ostree_image_repo = host_param('ostree-image-repo')
  os_major  = @host.operatingsystem.major.to_i
  os_name   = @host.operatingsystem.name
  ostree_image = host_param('ostree-image')
-%>

# License agreement
eula --agreed

# Use graphical install
graphical

# Keyboard layouts
keyboard --vckeymap=<%= host_param('keyboard') || 'gb' %> --xlayout=<%= host_param('keyboard') || 'gb' %>

%pre-install --erroronfail
# Prepare the SSL certificate

# Define the path to the Katello server CA certificate
KATELLO_SERVER_CA_CERT=/etc/rhsm/ca/katello-server-ca.pem
mkdir -p /etc/rhsm/ca
touch $KATELLO_SERVER_CA_CERT
chmod 644 $KATELLO_SERVER_CA_CERT

<%= snippet('ca_registration') -%>
%end

# System language
lang <%= host_param('lang') || 'en_US.UTF-8' %>

# Network
network --bootproto=dhcp

# Firewall configuration
firewall --use-system-defaults --ssh

# OSTree setup
# ostreecontainer --url=<%= ostree_image_repo %>/<%= ostree_image %> --no-signature-verification
ostreecontainer --url=<%= ostree_image_repo %>/<%= ostree_image %>

# Don't run the setup agent on first boot
#firstboot --disable

# Reboot after installation
reboot

# Generated using Blivet version 3.5.0
<%= @host.diskLayout %>

# System timezone
# timezone  <%= host_param('time-zone') || 'Europe/Brussels' %>
# timesource --ntp-disable

# Root password
rootpw --iscrypted <%= root_pass %>

# SSH Key for debugging the installation
sshkey --username=root "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAykgY+AnIUy3wzTFtQOtvwE/DQjcv5JKKu6eb0T/pvRqmIedVB2RVObI5DhZQGEpLKLF2Hugr1fy03bG98dNL/kFiStmH5nJ9vLGs9fGIHRUq+NpRJPB37vx19wGqd0YzAJeTanXDd8ZF2ZoBzCWaO5qqR/9c41m/DZ3GthxoryoEf0eJ04J4LFMhDcT5rC9c5PrcLqiwHcw80z9OZW5c5Npt7oL9G9Ymy7zCSysGcDJjDRXiAaPUSI59TztzB6W+MSungzsbQ8g3NPQJkPdMCtciJZ1dOFo+4+M+LCjvDG2lsdUxO8l4U34YxEhAhmZS0JrF0LoP9Ga+E5tf3Wcnaw== rriemann@earth"

<%#
Main post script, if it fails the last post is still executed.
%>
%post
exec < /dev/tty3 > /dev/tty3
chvt 3
(
logger "Starting anaconda <%= @host %> postinstall"

# Enforce local timezone link
ln -sf /usr/share/zoneinfo/<%= host_param('time-zone') || 'Europe/Brussels' %> /etc/localtime

<% if host_enc['parameters']['realm'] && @host.realm && (@host.realm.realm_type == 'FreeIPA' || @host.realm.realm_type == 'Red Hat Identity Management') -%>
  ##
  ## IPA Client Installation
  ##
  echo "Realm found: setup ipa-client"
  /usr/libexec/openssh/sshd-keygen rsa

  # HOTFIX for https://github.com/fedora-silverblue/issue-tracker/issues/427
  mkdir -p /var/lib/ipa-client/sysrestore
  mkdir -p /var/lib/ipa-client/pki
  mkdir -p /var/log/certmonger
  rm -f /etc/krb5.conf.d/*
  touch /var/log/ipaclient-install.log

  # One-time password will be requested at install time. Otherwise, $HOST[OTP] is used as a placeholder value.
  env DBUS_SYSTEM_BUS_ADDRESS=unix:path=/dev/null /usr/sbin/ipa-client-install -w '<%= @host.otp || "$HOST[OTP]" %>' --realm=<%= @host.realm %> -U --mkhomedir

  # Modify sssd.conf, add 'selinux_provider = none'
  sed -i 's/\(\[domain\/.*\]\)/\1\nselinux_provider = none/' /etc/sssd/sssd.conf

  # Set static hostname
  hostnamectl set-hostname --static <%= @host.name %>
<% end -%>
# Prepare the SSL certificate

# Define the path to the Katello server CA certificate
KATELLO_SERVER_CA_CERT=/etc/rhsm/ca/katello-server-ca.pem
mkdir -p /etc/rhsm/ca
touch $KATELLO_SERVER_CA_CERT
chmod 644 $KATELLO_SERVER_CA_CERT

<%= snippet('ca_registration') -%>
RHSM_CFG="/etc/rhsm/rhsm.conf"
test -f $RHSM_CFG.bak || cp "$RHSM_CFG" "$RHSM_CFG.bak"
subscription-manager config \
  --server.hostname="<%= @host.content_source.registration_url.host %>" \
  --server.port="<%= @host.content_source.rhsm_url.port %>" \
  --server.prefix="<%= @host.content_source.rhsm_url.path %>" \
  --rhsm.repo_ca_cert="$KATELLO_SERVER_CA_CERT" \
  --rhsm.baseurl="<%= @host.content_source.load_balancer_pulp_content_url %>"

<% if host_param('kt_activation_keys') -%>
  echo "Activation key found: Registering to subscription manager"
  subscription-manager register --name="<%= @host.name %>" --org='<%= host_param('subscription_manager_org') || @host.rhsm_organization_label %>' --activationkey='<%= host_param('kt_activation_keys') %>'
<% elsif host_param('subscription_manager_username') && host_param('subscription_manager_password') -%>
  echo "Credentials found: Registering to subscription manager"
  subscription-manager register --name="<%= @host.name %>" --username='<%= host_param("subscription_manager_username") %>' --password='<%= host_param("subscription_manager_password") %>' --environments "Library"
<% else -%>
  echo "Not registering to subscription manager"
<% end -%>

<%= snippet "blacklist_kernel_modules" %>

<%= snippet 'efibootmgr_netboot' %>

chvt 6

touch /tmp/foreman_built

) 2>&1 | tee /root/install.post.log
%end

# The last post section halts Anaconda to prevent endless loop in case HTTP request fails
%post --erroronfail --log=/root/install-callhome.post.log

if test -f /tmp/foreman_built; then
  echo "calling home: build is done!"
  <%= indent(2, skip1: true) { snippet('built', :variables => { :endpoint => 'built', :method => 'POST', :body_file => '/root/install.post.log' }) } -%>
else
  echo "calling home: build failed!"
  <%= indent(2, skip1: true) { snippet('built', :variables => { :endpoint => 'failed', :method => 'POST', :body_file => '/root/install.post.log' }) } -%>
fi

sync
%end

Provisioning with PXE requires that the Laptop can find some files from the ISO to boot into Anaconda installer. The ISO needs to be built first, so that the files can be extracted. As EU OS collaborators look into an automated way, the already extracted files can be downloaded as follows:

sh
cd $HOME
curl -LO https://blog.riemann.cc/files/eu-os/eu-os-foreman-pub-files.tar.gz
cd /
sudo tar -xvf $HOME/eu-os-foreman-pub-files.tar.gz
  • skopeo inspect --raw docker://fleet.eu-os.internal/public_organisation/eu_os/demo to check container image with skopeo
  • foreman-tail to see all kinds of logs
  • hammer template dump --id "Katello Kickstart Default" > template1.txt to export templates. Syncing to a git repo is also possible.

References:

Enroll Fleet Computers ​

System administrators can manually enroll computers to FreeIPA or use automatic enrollment during provisioning using the Kickstart EU OS.erb imported previously. Find more information in the subsequent section on Provisioning.

Manage Configuration and Execute Commands ​

WARNING

This page is work in progress. Related issue: #43

  • foreman comes with some remote execution scripts for bootc preconfigured (bootc-update or so)
  • TODO

References:

Cockpit ​

TODO ​