07 — Keep the homelab working and recoverable

Chapter TL;DR: Monitor the things people use, apply small tested updates, keep complete handoffs, and rehearse failures. Consider Kubernetes only after one-host Docker creates a measured problem.

Unattended operation requires visible failures and a recovery path that still works when the failed service is down. This chapter covers monitoring, updates, handoffs, drills, and the conditions for moving from Docker Compose to k3s.

Entry conditions#

TL;DR: Automate recovery only after local and remote access, storage restore, ownership, desired state, agent limits, and independent rollback are proven.

Do not automate recovery until all of these are true:

• Tier 0 local access has been tested with WAN and DNS unavailable.
• Tier 1 remote entry has been tested from a fresh cellular connection.
• storage datasets, owners, snapshots, and independent backups are documented;
• at least one representative restore has succeeded;
• every service has an exposure classification and named owner;
• the operations repository describes desired state and current validators;
• agent identities are narrower than human break-glass identities;
• rollback does not rely solely on the service being changed.

Automation applied to an unproven recovery path makes failure faster and less legible.

The operating loop#

TL;DR: Every intervention must observe, compare, classify, propose, authorize, protect, change, validate, record, and notify in that order.

Every managed service should fit the same loop:

1. Observe: gather live state using read-only identity and timestamp it.
2. Compare: distinguish desired state, last-known-good state, and observed state.
3. Classify: determine affected service tier, users, dependencies, and urgency.
4. Propose: choose the smallest reversible intervention and define success.
5. Authorize: reuse an existing phase grant or stop at a genuine authority boundary.
6. Protect: establish rollback and preserve evidence.
7. Change: perform one coherent, bounded mutation.
8. Validate: test from the consumer side, not only from inside the service.
9. Record: journal the source revision, target, facts, action, validator, and result.
10. Notify: tell the human only when action, decision, or materially changed risk warrants attention.

The model may perform the loop. The infrastructure decides which steps its identity can execute.

Monitor what household members actually use#

TL;DR: Test household-visible outcomes and dependencies rather than trusting running processes or green dashboards.

A process can be running while DNS returns the wrong answer, storage is read-only, a certificate is expired, or remote clients cannot connect. Start with the household promise at each tier.

Prometheus recommends complementing internal white-box metrics with black-box checks of user-visible behavior and alerting on symptoms that require action. Its alerting guidance is more valuable here than a giant metric catalog.

Check remote access from outside the house#

TL;DR: Use an external, credential-free heartbeat or probe so total home failure remains observable.

An in-house monitor cannot report total power loss, ISP failure, router failure, or its own death. Start with:

• an outbound dead-man heartbeat from home to an external destination;
• an external HTTPS probe for deliberately public applications;
• an external overlay member with monitor-only grants for private checks when warranted;
• local probes for direct-IP Tier 0 behavior;
• UPS events for on-battery, low-battery, and communications loss.

The Prometheus blackbox exporter supports HTTP(S), DNS, TCP, ICMP, and gRPC probes. It is an option, not a requirement. A small hosted health-check service is often the better first implementation because it remains outside the failure domain.

Give an external monitor no administration credential. Its identity should reach only the exact probe endpoints.

Track one incident instead of sending repeated alerts#

TL;DR: Give alerts identity, evidence, ownership, acknowledgement, dependency suppression, action, and a defined recovery condition.

Every alert needs:

• a stable identity;
• a start time and last-observed time;
• affected tier and promise;
• current evidence;
• dependency/parent incident;
• owner;
• actionable next step;
• acknowledgement;
• resolution condition;
• recovery notification.

Use grouping, deduplication, inhibition, and silence mechanisms rather than asking an LLM to remember which messages are stale. Alertmanager provides these concepts even if another implementation is used.

When “home unreachable” is active, inhibit the predictable cascade of “Pi-hole down,” “Plex down,” “NAS down,” and “certificate probe failed.” Those are evidence beneath one incident, not four additional pages.

Notify the human when:

• a Tier 0 or Tier 1 promise fails;
• unique data or backup recoverability is at risk;
• a security boundary may have changed;
• a bounded automated recovery failed;
• capacity will cross a decision threshold before the next review;
• physical work or a value judgment is required.

Do not notify merely because:

• a container restarted once and recovered;
• a scheduled job succeeded;
• a metric briefly crossed a threshold without user impact;
• an optional service is down during its maintenance window;
• an old alert appears in a transcript or log.

Always revalidate live state before acting on an alert. Alert text is untrusted operational input and may be stale or malicious.

Controlled automated recovery#

TL;DR: Start with bounded native supervision and scheduled fixed actions; do not give an alert-driven LLM general privileged remediation.

The day-one baseline grants no agent a privileged recovery verb. Separate native service supervision from agent-directed remediation:

• A root-owned systemd unit may use Restart=on-failure, RestartSec, StartLimitIntervalSec, and StartLimitBurst for one named native service. Its unit file, limits, and failure notification are installed in the service's human packet and are not writable by the agent.
• A container restart policy may recover a process that exits. Docker health status alone does not restart a running-but-unhealthy container, and this guide does not pretend otherwise.
• Root-owned timers may run an already configured backup, snapshot, probe, or UPS shutdown action only after their owner, installation, schedule, limits, evidence, and alert route are recorded and tested. The agent observes their result; it does not acquire the service credential or a generic systemctl, Docker, snapshot, or power verb.

These are fixed native mechanisms configured with the service, not an LLM deciding to run privileged commands from an alert. For a process that remains running but fails a health check, notify the owner and use the prepared human recovery packet. Add unattended health-triggered remediation only just in time, after a recurring failure justifies a concrete root-owned mechanism with a fixed target, protected configuration, bounded attempts, evidence capture, post-check, disable switch, and independent review.

Safe day-one automatic behavior is therefore limited to:

• retry a read-only probe;
• let the native supervisor restart one named noncritical process after it exits, within its configured rate limit;
• run a root-owned scheduled snapshot or backup whose writer, retention, installation, evidence, and alert boundaries were established and tested during storage setup;
• perform the exact UPS shutdown or preconfigured load-shed action already owned by the power service.

Do not initially automate:

• router, switch, DNS, VPN, storage, directory, or control-host restart;
• filesystem repair;
• storage topology changes;
• backup deletion/pruning;
• public DNS/firewall/ingress policy;
• blanket PDU actions;
• certificate-authority changes;
• identity or privilege policy.

For any later health-triggered automated restart, require all of these before enabling it:

1. repeated failed consumer-side checks;
2. confirmation that the parent dependency is healthy;
3. a cooldown and daily attempt limit;
4. capture of relevant evidence before restart;
5. post-restart validation;
6. escalation after the final allowed attempt;
7. an audit record.

Recovery that repeats without a limit is an outage amplifier. Until the concrete mechanism passes these tests, “restart after health failure” remains a human-run packet, not an exit requirement.

Handle updates in small, tested changes#

TL;DR: Discover continuously, but classify and install updates as bounded, recoverable changes with consumer-side verification.

Work profile: operational size large; quota large; human effort medium; agent effort large; wait large; outage change-specific; service/network/storage/household disruption declared per update lane; clock duration unknown until target-specific evidence exists.

An agent may continuously discover available updates with read-only access. Installation is a separate operation.

The homelab policy is neither “never upgrade” nor “treat every package like an enterprise release train.” The owner may run ordinary tools such as brew upgrade or apt upgrade. The safety requirement is to know the affected layer, preserve a real recovery point, change a bounded set, and test the outcome. Do not claim a staging environment that does not exist; use a disposable or lower-tier test only when it genuinely represents the dependency or migration at risk.

Classify an update:

For every update, one transaction artifact performs this sequence. It records the complete package, binary, image, configuration, and migration closure; runs relevant Bash/Python and regex fixtures; runs native checks; installs the bounded set; exercises real consumers and expected denials; records evidence; and reports success only after the final observation:

1. read upstream release notes and security advisories;
2. identify configuration/schema/default changes;
3. map dependencies and affected tiers;
4. verify backup and prepare a rollback or rebuild path, including limits on downgrade or data-schema compatibility;
5. test on disposable or lower-tier state where practical;
6. install one bounded set;
7. run native validators and consumer-side scenarios;
8. observe startup and a representative normal workload; any clock duration must be measured, sourced, or explicitly labeled a guess with basis and uncertainty;
9. record the observed version and result.

Do not let “latest” become desired state. Pin versions or channels deliberately and let a separate process propose movement.

Keep four update lanes separate: control-host applications and package-manager binaries; Linux operating-system packages and kernels; container/application releases and data migrations; firmware or storage-format changes. Combining them makes failure attribution and recovery needlessly hard. Automatic distro security patch installation is reasonable; schedule reboots deliberately. Rerun model/client admission, permission, transcript-capture, sandbox, and denial tests after meaningful agent-client or model changes.

If the owner upgraded outside the planned window, do not hide or scold the drift. Capture the package transaction and before/after versions if recoverable, run the same native and consumer checks, update the change record, and decide whether to accept the new state or return to last known good. After any failed update, the agent freezes mutation and recommends rollback as the SRE default, but it always states rollback risk and asks the owner before executing it.

Keep Docker Compose while one host is enough#

TL;DR: Keep Compose while one host remains understandable and recoverable; harden it before adding orchestration.

Docker calls single-server Compose the easiest production deployment model and recommends production-specific configuration, restart policies, and avoiding code bind mounts in production. See Compose in production.

Stay with Compose while:

• workloads fit on one host;
• brief host-level downtime is acceptable;
• processes exit on unrecoverable failure so restart policies can act, while health checks expose degraded-but-running state to monitoring or a separately bounded remediator;
• deployments are infrequent;
• service dependencies are understandable;
• storage is local or mounted before Docker starts;
• the owner does not need automatic placement across nodes.

Make Compose reliable before outgrowing it:

• keep the fully resolved configuration in Git;
• validate with docker compose config;
• define real health checks;
• remember that depends_on orders creation, not readiness, unless paired with a health condition as documented in startup order;
• use explicit restart policies;
• make host mounts and network prerequisites system services with ordering and failure behavior;
• pin images by deliberate version;
• separate application definition, secrets, durable state, and backup;
• verify recovery after host reboot.

Do not assume an unhealthy container will be restarted. A Docker health check records health status; a restart policy reacts to a container exit or daemon lifecycle, not to health status by itself. Either design the application to exit on an unrecoverable condition or use a separately authorized remediator with repeated consumer-side checks, cooldowns, attempt limits, evidence capture, and post-restart validation.

Do not hand a general deployer the Docker socket casually. Docker states that daemon control is effectively host-root authority in its security guidance. Retain the human boundary unless a constrained deployment mechanism actually exists and has been independently reviewed.

The k3s decision#

TL;DR: Adopt Kubernetes only when measured recurring problems justify the cluster's networking, storage, identity, and recovery burden.

Kubernetes becomes worthwhile when its reconciliation model removes more operational complexity than the cluster adds.

Move when several of these are recurring, observed problems:

• more than one workload node needs deliberate placement;
• a failed node should reschedule stateless work automatically;
• startup/dependency races recur despite correct host and application health design;
• repeatable rolling deployments matter;
• service discovery and ingress configuration are duplicated across stacks;
• Compose recovery and cross-host dependency toil consume material time;
• the operator can support cluster networking, storage, backup, and control-plane recovery.

Do not move because:

• Kubernetes is considered the “proper” homelab destination;
• two machines exist;
• restarting containers is mistaken for high availability;
• an agent can generate manifests;
• one broken Compose deployment has not yet been understood.

Kubernetes controllers reconcile declared state and replace failed pods, but they do not repair application corruption or missing storage. See Kubernetes self-healing. Incorrect liveness probes can themselves cause cascading failure; use startup, readiness, and liveness probes for their distinct purposes as described in Kubernetes probe guidance.

Set Kubernetes access limits before moving any service#

TL;DR: Enforce weak identities, admission restrictions, secret isolation, and default-deny networking before giving any deployment capability.

Kubernetes makes a subtle privilege-escalation path easy to miss: permission to create or change a workload can also confer the powers of its service account and access to any secret, host path, device, or node feature that the workload may request. Namespace-scoped RBAC alone is therefore not containment. Kubernetes documents these risks in its RBAC good practices.

Build and test this boundary before giving an agent any cluster write path:

1. Keep cluster administration human and break-glass. Never give the agent the k3s server token, node/admin kubeconfig, system:masters, cluster-admin, the Docker/containerd socket, or permission to mint credentials. The agent may inspect through a separate observer identity.
2. Separate namespaces by trust and failure boundary. Start with one application or tightly coupled stack per namespace. The deployer cannot create, relabel, or delete namespaces. Enforce the Kubernetes Pod Security Standards restricted profile by default; the agent cannot weaken or exempt it.
3. Scope RBAC to the exact job. A future protected deployment mechanism receives only the verbs and namespaced resource types its fixed job needs. Deny creation or mutation of Roles, RoleBindings, ClusterRoles, ClusterRoleBindings, service accounts, namespaces, CRDs, admission webhooks, nodes, certificate-signing requests, storage classes, and persistent volumes. Use resource names where the API supports them, and keep read-only observation separate from deployment. Until that mechanism exists, the agent receives no writer kubeconfig and the human applies the reviewed manifest.
4. Reject escape-capable workload settings at admission. Reject privileged containers, added capabilities, privilege escalation, host PID/IPC/network namespaces, hostPath, unapproved volume types, root execution, and arbitrary device, device-plugin, or runtime-class requests. Plex GPU access or a home-automation USB/radio may require an exception; put it in a dedicated exception namespace with a separately reviewed manifest and deployment path rather than weakening the default.
5. Make service accounts explicit and weak. Set automountServiceAccountToken: false by default. Create a narrowly privileged service account only for a workload that genuinely calls the Kubernetes API, and prevent the deployer from selecting any service account other than an allowlisted one.
6. Keep secret values outside the model. The observer and any future deploy path cannot get, list, or watch Secrets. That is not sufficient by itself: anyone who can create a Pod may be able to mount a Secret and read it through the workload. The protected admission/deployment path must allowlist secret references per workload, inject only the required values, and prevent the agent from changing the service account, command, image, or volume to extract them.
7. Default-deny network traffic. Apply default-deny ingress and egress in every workload namespace, then allow DNS and named dependencies explicitly. Confirm that the installed k3s CNI and policy controller actually enforce both directions; a stored NetworkPolicy object is not proof. Test denial from a disposable pod, including cross-namespace and unexpected Internet destinations. See Kubernetes Network Policies.
8. Keep the writer human until a real control plane earns trust. Let the agent propose and validate a fully rendered change; the human reviews and applies it with a separately held credential. Later, a protected GitOps controller or independently reviewed fixed deployment service may verify an independent human grant, target namespace, resource types, image, service account, secret references, volumes, devices, policy, and server-side validation before applying with its own credential. The agent cannot modify that controller, its policy, credential, or approval path. A custom root program drafted during the migration is not automatically such a control plane.

Admission-test the boundary, not just the prompt. A forbidden privileged or hostPath Pod must be rejected; secret reads, service-account substitution, cross-namespace writes, and namespace relabeling must be denied; and a controlled network probe must demonstrate default-deny behavior. A successful harmless deployment does not prove that the forbidden paths are closed.

What highly available k3s requires#

TL;DR: Three control-plane nodes address quorum only; enumerate every remaining storage, ingress, network, power, and administrative failure domain.

A default single k3s server uses SQLite and remains a control-plane single point of failure. One server plus one agent adds workload capacity; it does not create an embedded-etcd high-availability control plane.

K3s requires at least three server nodes and an odd count for embedded-etcd quorum. See k3s architecture and embedded-etcd HA.

Even then, list remaining failure domains:

• local persistent volumes;
• single-instance databases;
• one ingress address or load balancer;
• one router/Internet connection;
• one storage server;
• DNS outside or inside the cluster;
• applications unable to run concurrently;
• the rack, room, UPS, and human administrative domain.

Do not label the result “HA” without saying which failure it survives.

K3s recovery requires both datastore backup and the original server token because the token protects encrypted bootstrap data. Preserve and test both using the current k3s backup and restore procedure.

Migration from Compose to k3s#

TL;DR: Build alongside the working system and migrate from disposable stateless workloads to stateful services only after recovery proof.

Prerequisites:

• resolved Compose state in version control;
• every port, hostname, dependency, UID/GID, secret reference, dataset, and health check inventoried;
• application-level backup and restore tested;
• the Kubernetes observer/deployer identities, namespace and admission policy, service-account and secret boundary, and default-deny network policy are enforced and their denial tests pass;
• target cluster storage and node-placement model decided;
• ingress collision understood—k3s installs Traefik by default and ServiceLB commonly uses ports 80/443, as described in k3s networking services;
• old deployment retained as rollback;
• maintenance window and success tests defined.

Build the cluster alongside the working Compose host. Migrate in this order:

1. stateless internal test service;
2. stateless production service;
3. reconstructable service with noncritical cache/state;
4. read-mostly application whose state has a proven restore;
5. stateful service last.

For each stateful cutover:

1. prove target storage and permissions with disposable data;
2. stop or quiesce writes at the source;
3. take an application-consistent backup/snapshot;
4. copy once;
5. validate ownership and application data;
6. start only the target;
7. test user behavior;
8. retain the source deployment stopped and unchanged for rollback;
9. never permit two writers to the same database.

Tools such as Kompose can provide scaffolding, but its own documentation describes behavior changes around volumes and service naming. Generated manifests require design review; they are not a migration plan.

Keep recovery and networking outside Kubernetes#

TL;DR: Keep DNS, VPN, power, storage, backup, console, repository, and external monitoring outside the cluster they must recover.

Do not move these into the cluster merely for consistency:

• the only household DNS resolver;
• the only VPN entry point;
• UPS communications and shutdown coordinator;
• storage pool management;
• backup repository and retention authority;
• BMC/PiKVM/PDU control;
• the only operations repository;
• the only external monitor.

The cluster may consume those services. It should not be required to repair them or itself.

Prove a host can be replaced#

TL;DR: Call a host replaceable only when desired state, recoverable data, external identity, and a tested rebuild path exist elsewhere.

A host is replaceable only when:

1. desired state exists elsewhere;
2. unique data is separated and recoverable;
3. identity and secrets can be reissued;
4. a replacement has actually been built or rehearsed.

Without those proofs, “cattle” is only a label attached to a pet.

Use a rebuild drill:

• provision a clean VM or spare machine;
• assign a new infrastructure identity;
• apply desired state from the repository;
• restore selected application data;
• run service validators;
• switch a test client;
• destroy the replacement.

Record manual steps. Reduce them in the next iteration without granting the agent general privilege.

Failure drills#

TL;DR: Exercise recovery claims from nondisruptive evidence to attended high-blast-radius tests, using risk-based rather than ceremonial cadence.

Work profile: operational size XL; quota large; human effort large; agent effort large; wait large; outage planned by drill; network/service/power/storage effects require separate attended gates; clock duration unknown until target-specific evidence exists.

Run the drills in increasing blast radius.

Routine checks that do not cause an outage#

TL;DR: Automate safe reachability, DNS, heartbeat, backup, restore, expiry, configuration, and denial checks without paging on success.

• connect to the Mac and one server by their recorded direct IPs without stopping household services;
• establish a fresh remote VPN session from cellular;
• verify management surfaces remain unavailable publicly;
• confirm the external heartbeat detects and resolves a simulated miss that does not interrupt the home.

Collect these automatically where possible. A month is a reasonable initial maximum interval for a fresh cellular-path check, not a reason to page the owner or cause an outage.

Attended Tier 0/1 drill when risk or change warrants it#

TL;DR: Deliberately fail DNS or access only in a maintenance window and prove recovery by the independent path.

During a declared maintenance window, stop Pi-hole and use the direct-IP recovery card. Run this when commissioning the recovery claim, after a material DNS/network/access change, after a routine check fails, before important travel, or when the recorded maximum risk interval arrives. Restore DNS and verify ordinary household resolution before closing the window.

At the recorded restore and identity-test interval#

TL;DR: Restore representative data and application state, then verify ownership, ACLs, identities, and actual usability.

Work profile: operational size large; quota medium; human effort medium; agent effort large; wait large; outage none when staged correctly; backup/restore load; production data unchanged; clock duration unknown until target-specific evidence exists.

• restore representative family data to scratch;
• restore one application from its documented backup;
• reboot the control host and verify the entire post-boot path;
• shut down one optional app host and observe alert inhibition;
• prove downloader VPN containment if that stack exists;
• confirm current UID/GID ownership on representative datasets.

After power/storage changes or at their recorded maximum interval#

TL;DR: Retest shutdown, startup order, pool import, degraded behavior, replacement identification, and measured power margin.

Work profile: operational size XL; quota medium; human effort large; agent effort medium; wait large; outage planned power/storage interruption; attended household disruption and data-risk preflight; clock duration unknown until target-specific evidence exists.

• controlled UPS outage and graceful shutdown;
• cold boot in dependency order;
• identify a chosen disk by serial and physical bay;
• rehearse replacing a failed noncritical drive with a spare or a documented dry run;
• retrieve offsite recovery material without relying on the primary password manager or homelab.

Before extended travel#

TL;DR: Verify the owner's access, account recovery, monitoring, backup, capacity, and update state before departure; check any other holder or helper only when one exists.

Work profile: operational size medium; quota medium; human effort medium; agent effort medium; wait medium; outage none unless a failed check requires maintenance; no planned household disruption; clock duration unknown until target-specific evidence exists.

• verify the owner's recovery access;
• if another household member holds access, verify that person's access;
• if a local helper exists, leave only the instructions and authority that helper actually needs;
• confirm FileVault/cold-boot implications for the exact Mac;
• check UPS battery state;
• confirm external alert delivery and recovery;
• remove expired grants and stale devices;
• freeze nonessential platform changes.

Do not run a destructive drill against unique data. Use disposable fixtures or validated backups.

What the agent does during an incident#

TL;DR: Preserve access and evidence, diagnose the earliest failed dependency read-only, and propose the smallest authorized reversible recovery.

In an incident, tell the operator agent:

Preserve access and evidence. Begin read-only. Determine the highest affected service tier and identify the earliest failed dependency. Do not work around a child symptom while its parent is unhealthy. Do not change identity, firewall, DNS, storage topology, or backup retention. Propose the smallest reversible recovery with verifier and a prepared rollback; recommend rollback before fix-forward when evidence supports it, but always ask me before backtracking. Continue through the already-authorized incident runbook; stop for rollback, destructive recovery, new authority, conflicting evidence, or a Tier 0/1 power or network change. Report only current state, evidence, action, and blocker. End with NEXT ACTION naming the actor, host or UI surface, account, path, exact invocation or wait event, expected result, and resume condition.

The incident transcript is not the incident record. Persist observations and actions with timestamps as they occur.

Exit criteria#

TL;DR: Claim lights-out readiness only when failures are externally visible, recoverable by tested paths, durably recorded, and bounded by enforced agent authority.

Work profile: operational size large; quota medium; human effort medium; agent effort large; wait large; outage representative drill outages; all affected tiers and household effects must be evidenced; clock duration unknown until target-specific evidence exists.

The homelab has reached the first useful lights-out operating state when:

• a monitor outside the house checks the Tier 1 promise;
• alerts group parent and child failures and send recovery;
• at least one selected noncritical service has proved its root-owned native supervisor can recover an exited process within a recorded rate limit and alert after final failure; health-triggered privileged remediation remains disabled unless its separate just-in-time gate has passed;
• no automatic actor controls Tier 0/1 power, identity, storage destruction, or backup deletion;
• ordinary updates follow a repeatable classify/protect/change/validate loop;
• a fresh agent can reconstruct current intent without prior conversation;
• representative local, remote, restore, reboot, and UPS drills have passed;
• the owner knows which specific observed pain—not aspiration—would justify k3s.