🤖 Self-Healing DNS, Auto-Generated Diagrams, and Automation That Runs While I Sleep
Fourth post in the k3s homelab series. Previously: CGNAT tunneling, LUKS + Dropbear + RAID6, and multi-arch scheduling.
What happens when I deploy a new service at midnight? The DNS updates itself, the dashboard rebuilds, the cluster diagram regenerates, and a notification hits my phone. I wake up and everything just works. That's the goal, anyway.
It wasn't always like this. I used to manually update DNS records after adding services, forget to mirror TLS certs to new namespaces, and discover weeks later that orphaned PVs were eating 200 GB of RAID storage. Every automation here exists because I got burned by doing it by hand 🔥.
Event-watcher: the cluster has eyes
The event-watcher is a small deployment that watches for Ingress ADDED and DELETED events across all namespaces. When it sees one, it triggers the dns-updater CronJob immediately.
There's a cooldown window (300 seconds) to prevent rapid-fire triggers during a big deployment, and a startup grace period (15 seconds) so it doesn't go wild when the pod first starts and replays the existing ingress list. It tracks a "known ingress set" in a tmpfs volume to distinguish real changes from reconnection replays.
The event-watcher itself is tiny, 5m CPU and 32 MB of memory. It just sits there, watching, waiting. Like a very boring security guard who is extremely good at their job.
DNS-updater: push-based split-horizon DNS
The dns-updater is where the real work happens. It runs daily at 5 AM as a CronJob, plus on-demand when event-watcher triggers it. Its job: discover every domain the cluster serves and push DNS records to two places.
First, it discovers all domains from three sources: standard Ingress resources, Traefik IngressRoute CRDs (parsing Host() matchers), and Garage S3 bucket names (for web-hosted static sites). That gives it the complete list of every hostname the cluster responds to.
Then it pushes to two targets:
Home router (split-horizon DNS): hoth, my router running kresd, gets an updated /etc/hosts via SSH. Every discovered domain points to MetalLB's LoadBalancer IP on the LAN. From inside the house, cluster services resolve to the local IP and traffic stays on the network. From outside, the same domains resolve to the public VPS via normal DNS. Split-horizon, no hairpin NAT needed.
Headscale DNS: The same domain list goes to the edge VPS as Headscale extra DNS records. Tailscale clients anywhere in the world can resolve cluster services and traffic routes over the encrypted overlay instead of bouncing through the public internet.
The whole thing is pull-based from the cluster's perspective: it discovers what it's running and pushes config outward. The router and Headscale don't need to know anything about Kubernetes. They just get a hosts file and a JSON blob.
Reloader: because I kept forgetting to restart pods
Here's a scenario that happened more times than I'd like to admit: I'd update a ConfigMap, run install <chart>, and then spend 20 minutes wondering why the change wasn't taking effect. The pod was still running the old config because Kubernetes doesn't restart pods when their mounted ConfigMap changes. You have to do it manually. Every. Single. Time.
Stakater Reloader fixes this. It watches all ConfigMaps and Secrets across the cluster. When one changes, it finds the Deployments and StatefulSets that mount it and triggers a rolling restart. No annotations needed, it just works globally.
This is especially useful for Glance (the dashboard). All its config lives in a ConfigMap. I edit the layout, run install glance, and Reloader restarts the pod automatically. I don't even think about it anymore. Same pattern for any chart where config lives in a mounted ConfigMap.
Reflector: the end of my jankiest CronJobs
Emberstack Reflector solves a common Kubernetes annoyance: you need the same Secret in multiple namespaces. The TLS wildcard cert is issued by cert-manager in the cert-manager namespace, but every namespace needs it for Ingress TLS termination.
Before Reflector, I had a pair of CronJobs that ran kubectl get secret ... -o json | kubectl apply every hour. They worked.. most of the time. Then I'd create a new namespace, forget to add it to the script, and spend an evening debugging why Traefik was serving the default self-signed cert for one specific service. Not fun 😅.
Reflector watches for Secrets with specific annotations and mirrors them to other namespaces automatically:
1# cert-manager Certificate with Reflector annotations
2secretTemplate:
3 annotations:
4 reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
5 reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
Same pattern for the Gitea container registry secret, which needs to exist in every namespace that pulls images from the private registry. New namespace? Reflector mirrors the secrets there instantly. No script to update, no CronJob to wait for.
Gotify notifications: the 3 AM wake-up call
What's the point of monitoring if you don't know when things break? Alertmanager routes alerts to Gotify, which runs in-cluster and has its own Android app. My phone buzzes, I see what's wrong, I decide if it can wait until morning. Usually it can. Sometimes it can't 😅.
1# Alertmanager routing (simplified)
2routes:
3 - receiver: gotify
4 group_by:
5 matchers:
6 - severity =~ "critical|warning"
Critical alerts repeat every 15 minutes, warnings every 2 hours. Inhibition rules prevent duplicate noise: a critical alert for the same alertname suppresses the corresponding warning. So if NodeDown fires as critical, the warning-level version doesn't pile up in my notification shade.
The alertmanager-gotify-bridge deployment translates Alertmanager webhooks into Gotify API calls. Lightweight Go binary, barely any resources. The most expensive part of this whole pipeline is my sleep quality.
PVC reclaimer: the 200 GB I didn't know I was wasting
One day I noticed my RAID was at 30% usage when I expected 25%. I ran kubectl get pv and found 47 Released PersistentVolumes hanging around, holding NFS directories hostage. Some were from services I'd uninstalled months ago. 200 GB, just sitting there.
The problem: when a PVC is deleted but the PV has Retain reclaim policy (which I use for everything, because data), the PV enters a Released state and sits there forever. Nobody cleans it up. Nobody tells you.
The pvc-reclaimer CronJob runs daily at 5 AM and deletes all Released PVs. Simple, but I wish I'd written it 6 months earlier.
Job cleanup: history limits
CronJobs generate completed Job pods that pile up if you're not careful. On kamino (the RPi that runs event-watcher), a burst of ingress events would create a dozen dns-updater Jobs sitting around as Completed pods, eating into the node's pod capacity.
The fix is CronJob history limits:
1# Every CronJob
2spec:
3 successfulJobsHistoryLimit: 1
4 failedJobsHistoryLimit: 3
Only the last successful run and the last 3 failures are kept. Kubernetes garbage-collects the rest. I initially used ttlSecondsAfterFinished for this, but history limits are cleaner, they keep a small window of recent jobs around for debugging without letting things pile up indefinitely.
The descheduler: rebalancing every 5 minutes
I covered this in the scheduling post, but it's worth mentioning here as part of the automation story. The descheduler runs every 5 minutes, evicting pods from overutilized nodes so the scheduler can place them somewhere better. It respects priority classes (critical pods are untouchable), handles pods with PVCs, and catches crashlooping pods that have restarted too many times.
The descheduler is the piece that ties everything together. Combined with the event-watcher, Reloader, Reflector, and the PVC reclaimer, the cluster is constantly adjusting itself: new services get DNS entries, pods get rebalanced, config changes trigger restarts, orphaned storage gets reclaimed. It's not one big brain running the show — it's a dozen small automations that each handle one thing, and the compound effect is a cluster that takes care of itself.
The result
All the ArgoCD applications synced and healthy, DNS that updates itself when services come and go, a cluster diagram that regenerates hourly from live state, secrets that mirror across namespaces, pods that restart when their config changes, alerts that reach my phone, storage that cleans up after itself.
None of this is flashy. Most of it is CronJobs and watchers, small scripts that run on schedule or in response to events. Nothing that would make the front page of Hacker News. But I deploy a new service, push to main, and go to bed. By morning the DNS is updated, the diagram shows the new pod, the TLS cert is mirrored, and everything is monitored. The only thing I need to do is write the Helm chart.
Every automation here started as a manual task I did once too often. The best time to automate something is right after you've been burned by not automating it 🔥.
Next up: GitOps at home, ArgoCD + Gitea, and the monorepo that runs everything.