Secure Homelab Connectivity: How Headscale Handles my Needs

As I mentioned previously, I have a mixture of local and remote machines in my homelab as well as a laptop and phone that can and will connect to my various systems and applications. This means I need some sort of VPN or SD-WAN¹ to ensure that the traffic remains encrypted and to provide a reliable way to connect without having to expose everything to the public internet.

Ideally I wanted this solution to also work when connecting locally so I could use consistent naming and DNS resolution. Network performance, especially between co-located nodes, was important, there was no benefit to keeping traffic on the encrypted tunnel if it resulted in horrible bandwidth.

Previous setups

Early in my career I would expose my SSH server and then use SSH tunneling to connect remotely to applications running at home, or to act as a proxy for web browsing. It wasn’t ideal but it worked when I only had to worry about one or two target devices that I only connected to sporadically.

Zerotier

Several years ago I was introduced to Zerotier and I used it for my encrypted networking mesh up until fairly recently. It is quite easy to get configured, you install the zerotier binary, connect the node to your network then from the admin control panel you approve the new node and everything works. There are routing rules and permissions that can be applied but the defaults did everything I needed so I never investigated.

Unfortunately there were a handful of limitations I ran into that led me to look for alternatives. Bandwidth often ended up being quite poor. Searching for zerotier throughput and bandwidth shows multiple cases with similar issues and solutions. At times everything worked out nicely but then I would have my two machines that were side-by-side only managing a fraction of their expected bandwidth (Gigabit switch and ports but getting less than 10 megabits/s). The inconsistency was the biggest issue here.

DNS with Zerotier worked most of the time using their zeronsd offering. My android phone frequently failed to retrieve the DNS records and even my other machines sometimes decided they couldn’t find the right information. DNS performance did get better with version updates (originally there was no direct support for DNS) but still left something to be desired.

OpenZiti

Late last year when looking to move on from Zerotier I came across OpenZiti, the open source SD-WAN underlying NetFoundry. I really liked what it did and will keep it in mind as a tool for future projects but once again ran into quirks that led me to keep looking.

Applying new configuration for new nodes or services sometimes failed to work, even though the equivalent settings worked for a previous service. Deleting and re-creating the service sometimes fixed that issue with no apparent reason why the subsequent attempt worked.

I’ll be the first to admit that I was likely to blame for the issues I had with OpenZiti. But I also did not need the granularity it provided. I wanted a network where I could treat all my devices as if they were in the same LAN, I didn’t need network segmentation or security rules.

Migrating to Headscale

Although I had had fun experimenting with OpenZiti I felt it wasn’t the right fit for my current needs. I knew that WireGuard was extremely performant on Linux and that Tailscale simplified the VPN configuration by handling all the inter-node routing. Since I was looking to self-host as much as possible this led me to Headscale, an open source, self-hosted implementation of the Tailscale control server.

For the most part my initial setup follows the official installation guide. I am using Salt in masterless to handle the bootstrapping and configuration (see the next post in this series for more details on how I’m using Salt) and to ensure my initial configuration is consistent. I also add the Headscale server as a Tailscale client so that the only exposed ports are those needed for the Tailscale control plane to operate.

Once a machine is registered everything “just works” with this setup. I haven’t needed exit nodes yet or any fancy ACLs² since I am the only user on the network and I want all my nodes to be able to communicate. DNS always works as long as I have my Linux network services configured properly (Resolvers have occasionally activated out of order because I had my systemd services running incorrectly, but I think I finally have it working). Custom DNS records are added to the Headscale configuration file and get propagated automatically.

Addendum: The DNS Issue

I’m using NetworkManager to handle my networking since it is the default on EndeavourOS. If the connection resets after Tailscale comes up then it overwrites the Tailscale DNS settings. I was finally able to resolve this ordering issue by enabling systemd-resolved and then setting NetworkManager to leverage it for DNS.

1[main]
2dns=systemd-resolved
3rc-manager=symlink

Manually adding the symbolic link fixes the ordering issue without having to reboot.

1sudo systemctl enable --now systemd-resolved
2sudo systemctl restart NetworkManager tailscaled
3sudo systemctl restart k3s-agent # A future post will cover K3s
4sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf