Kubernetes

I’ve been in the process of upgrading my homelab for the last few months. I will be writing a more in depth series of posts about it in the near future. When I first deployed my new homelab (see my previous post on why I had to redeploy) I went with Uptime Kuma for status monitoring. I quite liked the UI for the statuses themselves as well as the dashboards I could create. Unfortunately all the persistent storage data for these status checks was lost during the rebuild and when looking for a more programmatic way to re-create the checks I found out that there is no official API to do so. Now I could install the Unofficial UptimeKuma API Wrapper and use Python to create all monitor endpoints, but then I would need to ensure I kept making these updates each time I added a new application. Instead I was able to settle on Gatus by browsing the list of awesome-status-pages and then using a combination of Claude Sonnet, Reddit and Google to determine a path forward. ...

A few weeks ago my Kubernetes-based Homelab suffered a catastrophic failure. The internal routing no longer worked nor did any DNS resolution for in-cluster services. Since I had almost everything defined in yaml files and had all my persistent data stored in Longhorn Volumes, which were they themselves backed up, I felt safe destroying and rebuilding the cluster. Unfortunately while doing so the Longhorn volumes turned out to either be missing (only 9 of roughly 17 were found) or were orphans that refused to attach to Kubernetes. I was able to find the remaining volumes in a backup so started restoring but when trying to mount them individually using a Longhorn Docker container the contents were corrupt. Working through my Borg backups of the volumes only found more corruption except for my PGDump backups, which luckily contained the vast majority of the data I would hate to lose. ...