Given the choice I would treat all my homelab servers as cattle1 however there are components that do need bespoke configuration. My main home server acts as my Kubernetes Master, my on-site backup server and my Salt master. My ArchLinux home server acts as a package mirror for my desktop and laptop while also building my AUR packages to ensure they stay up to date and consistent between machines.

On the other hand most of my machine configuration is fairly identical between my homelab machines. I want the same CLI tools available, similar backup configurations and want to make sure any new host can be added to Kubernetes without issue if applicable. Between these common configurations and my preference for Programmatic Configuration I knew I needed some sort of Configuration Management tool. Even my pet servers become more cattle-like since their configuration is easily reproduced on a new machine in case of a failure.

Why Salt?

I’m well aware of alternate Configuration Management tools such as Ansible and Puppet. I’d previously used Puppet at my job and it worked well, but day to day I use YAML (and jinja templates) more often than Ruby so sticking with those felt more sensible. Ansible on the other hand I have never used.

I remember being drawn to Salt years ago before I ever used anything for Configuration Management professionally. It might have been due to it supporting both Windows and Linux when I was dual-booting my desktop.

If I ever run into situations where Salt doesn’t satisfy my needs I will revisit my choice and likely switch to Ansible (if only due to its popularity) but for the time being I see no reason to choose anything else.

My Two Salt Setups

I currently have 2 Salt configurations. The first one is primarily meant for bootstrapping my homelab in case of a future rebuild while the second is meant to manage any updates and configuration changes over time. I need these distinct setups since I leverage Tailscale for my inter-machine connectivity and need to be able to redeploy my Headscale control server in case of failure.

For those unfamiliar with Salt a few terms:

State
One or more configurations to be applied to target machines
Pillar
Configuration data to be processed by Salt States. This allows for re-using the same state with different configuration based on how the minion is configured.
Grains
Per-minion configuration that can be used to identify what states need to be applied.
Master
The control node for a Salt setup. It orchestrates and tracks the state of minions.
Masterless
A Salt installation that does not have any master node. The configuration is applied and tracked locally.
Minion
A node that is managed by the master.

Headscale Masterless

This is a very bare-bones setup to be hopefully only ever run once that installs Salt, decrypts my age key so that encrypted Salt data can be used and then runs Salt locally to create a new Headscale node.

I use age for encryption similarly to how I do for my dotfiles, I generate a keypair to be used for keeping data secure, encrypt the private key with age using a complex password so that I can keep it in the repository then decrypt it during setup so that subsequent operations can leverage it to view the sensitive data.

Homelab Saltstack

This is a more complete and typical Salt setup. I configure blackstaff as the Salt Master, as well as as a minion so that it can self-configure and then point all my other machines to it for registration. As in my Masterless setup for Headscale I use an age renderer (below) to manage encrypted data but only the master node stores the decryption keys, everything else gets the data via Salt.

Both the Salt State and Pillar data get stored in the same git repository for simplicity with distinct git configuration to ensure the correct directories are references. As long as I have a copy of my repository I can rebuild the majority of my homelab, there are a few aspects that aren’t integrated yet (such as my Caddy configuration) but they will be in the near future. I do also intend to move my Headscale configuration into the Salt Master so that the node can be migrated over to use the same configuration post-deployment. This will allow my bootstrap setup to be as minimal as possible before the complete setup gets managed in a consistent manner.

I currently leverage the Salt Mine for two pieces of data. I need the SSH Public Keys for the root user on each node to add to my Borg backup target .ssh/authorized_keys so that new nodes can perform their backups as expected. The other data in the mine is purely for K3s on-boarding. The master node has an authentication token that gets generated on initial install. After starting my cluster setup I re-trigger my Salt state so that the Agent nodes can connect securely. My next post will go further in depth to how I have K3s configured.

  • Age Decryption Renderer: salt/_renderers/age.py 
     1import logging
 2import subprocess
 3from pathlib import Path
 4
 5import yaml
 6
 7log = logging.getLogger(__name__)
 8
 9AGE_HEADER = "-----BEGIN AGE ENCRYPTED FILE-----"
10
11
12def _decrypt_block(encrypted_data, key_file):
13    """Decrypt a single age-encrypted block."""
14    try:
15        process = subprocess.Popen(
16            ["age", "-d", "-i", key_file],
17            stdin=subprocess.PIPE,
18            stdout=subprocess.PIPE,
19            stderr=subprocess.PIPE,
20        )
21
22        decrypted_data, stderr = process.communicate(input=encrypted_data.encode())
23
24        if process.returncode != 0:
25            raise Exception(f"Decryption failed: {stderr.decode()}")
26
27        return decrypted_data.decode().strip()
28
29    except Exception as e:
30        log.error("Age decryption failed: %s", str(e))
31        raise
32
33
34def _process_encrypted_blocks(data, key_file):
35    """Recursively process dictionary looking for encrypted blocks."""
36    if isinstance(data, dict):
37        return {k: _process_encrypted_blocks(v, key_file) for k, v in data.items()}
38    elif isinstance(data, list):
39        return [_process_encrypted_blocks(item, key_file) for item in data]
40    elif isinstance(data, str):
41        if AGE_HEADER in data:
42            return _decrypt_block(data, key_file)
43        return data
44    return data
45
46
47def render(data, saltenv="base", sls="", **kwargs):
48    """
49    Handle age decryption in the render pipeline.
50
51    Expects:
52​    - Individual values to be encrypted, not the entire file
53​    - Age armored format with BEGIN/END markers
54    """
55
56    # Get key file location from config or use default
57    key_file = __opts__.get("age.key_file", "/etc/salt/age/key.txt")
58
59    if not Path(key_file).exists():
60        raise Exception(f"Age key file not found: {key_file}")
61
62    # If data is already parsed (e.g., by YAML renderer), process it
63    if isinstance(data, (dict, list)):
64        return _process_encrypted_blocks(data, key_file)
65
66    # If data is a string (raw file content), parse it first
67    try:
68        parsed_data = yaml.safe_load(data)
69        return _process_encrypted_blocks(parsed_data, key_file)
70    except Exception as e:
71        log.error("Failed to parse data: %s", str(e))
72        raise
73
74
75def __virtual__():
76    """
77    Verify age is installed
78    """
79    try:
80        subprocess.run(["age", "--version"], capture_output=True)
81        return True
82    except FileNotFoundError:
83        return False, "age encryption tool not found"
  • Salt Master config 
    ## GITFS configuration
fileserver_backend:
​  - gitfs
​  - roots

gitfs_pubkey: /opt/saltstack/salt/.ssh/id_ed25519.pub
gitfs_privkey: /opt/saltstack/salt/.ssh/id_ed25519
gitfs_provider: pygit2
git_ssh: /opt/saltstack/salt/.ssh
gitfs_global_lock: False

# State file configuration
gitfs_update_interval: 900          # 15 minutes instead of 60 seconds
gitfs_remotes:
​  - [email protected]:jleechpe/saltstack.git:
​    - root: salt
​    - base: main

# External pillar
git_pillar_global_lock: False
git_pillar_update_interval: 900     # 15 minutes
ext_pillar:
​  - git:
​    - main [email protected]:jleechpe/saltstack.git:
​      - root: pillar
​      - env: base
​      - pubkey: /opt/saltstack/salt/.ssh/id_ed25519.pub
​      - privkey: /opt/saltstack/salt/.ssh/id_ed25519

## Merge rules
pillar_merge_lists: True

## Renderer
renderer_whitelist:
​  - yaml
​  - age
​  - jinja

age.key_file: /etc/salt/age/key.txt
  • Salt Mine settings 
    1mine_functions:
2  ssh.user_keys:
3​    - prvfile: False
4  network.ip_addrs: []
5  grains.items: []
6  k3s.get_token:
7​    - mine_function: cmd.run
8​    - 'cat /var/lib/rancher/k3s/server/node-token'
9​    - python_shell: true

  1. “Cattle not Pets”. Treat your servers as replaceable and generic (cattle) rather than special unique machines that would be painful to lose and replace (pets). ↩︎