03 — Set up the first Linux server

Chapter TL;DR: Install a small Linux base, test the used hardware, give people and the LLM separate access, and prove the server can reboot, update, and recover over both the home network and a direct cable.

Before adding Docker, storage, or applications, make the server predictable to boot, identify, reach, and rebuild. This chapter establishes local and SSH access, a direct-network fallback, limited LLM access, updates, firewall behavior, and a rebuild record.

Recommended baseline:

• the current Ubuntu Server LTS point release, installed without a desktop;
• one small SSD or NVMe device dedicated to the operating system;
• UEFI boot, direct disk access, and no motherboard “RAID” layer;
• a plain ext4 root filesystem unless a recorded requirement justifies another layout;
• wired Ethernet with DHCP initially and a reservation at the router;
• a human administrator with key-based SSH and a separate local recovery credential;
• a distinct unprivileged agent account with its own narrow, source-constrained, noninteractive key;
• a tested direct-Ethernet recovery address with no gateway or DNS dependency;
• automatic security updates, but deliberate reboots;
• no Docker, ZFS data pool, SSO, private CA, Kubernetes, or general-purpose agent sudo yet.

As of September 2026, Ubuntu publishes 26.04.1 as its current Server LTS. The durable rule is “current supported LTS point release,” not “the newest six-month release.” Ubuntu recommends LTS for servers because it receives five years of standard support; interim releases receive nine months. If a required controller, application, or vendor supports only the preceding LTS, use that still-supported LTS and record the exception instead of improvising around it. See the Ubuntu release list and Server upgrade policy.

What this chapter must prove#

TL;DR: Finish with a predictably bootable, independently reachable, reproducible Linux base whose agent identity lacks privilege before storage or services depend on it.

Do not proceed to the storage chapter until all of these are true:

• The machine boots twice from cold power and returns after AC loss in the intended state.
• The OS disk is physically and logically distinguishable from every data disk.
• A human can log in locally and by SSH without a vendor cloud service.
• The agent can log in under its own identity without borrowing a human key.
• The agent has no sudo, Docker, lxd, libvirt, disk-management, or unrestricted journal privilege.
• The host firewall permits only the initial named management paths.
• The normal LAN path and the direct-cable path both work by IP address.
• Time synchronizes, security updates run, disk and temperature observations are available, and logs survive a reboot.
• The owner has a dated build record and can reinstall the boot disk without guessing where household data lives.

This is the first cattle test. It does not mean the physical machine is disposable. It means the operating system installation is no longer the only copy of its own design.

Step 1 — Make the installer choices that are costly to change#

TL;DR: Set the supported distribution, boot mode, OS disk, filesystem, encryption, addressing, and recovery defaults before installation makes them costly to reverse.

Work profile: operational size medium; quota medium; human effort medium; agent effort medium; wait small; outage none; no service/network/storage/household disruption; clock duration unknown until target-specific evidence exists.

The human decides consequences. The agent discovers models, interfaces, and current state. Complete this table before writing installation media.

Why plain ext4 on the boot device#

TL;DR: Use a commonly recoverable ext4 boot filesystem unless encryption, volume management, or another layer solves a recorded requirement with a tested unlock path.

The data pool will later provide checksums, redundancy, snapshots, and dataset policy. The boot device has a different job: contain a replaceable OS that common rescue tools can mount. Plain ext4 minimizes the number of storage layers a beginner must reconstruct while standing at a failed machine.

This is not an argument that LVM, encrypted LUKS, or ZFS root are bad. It is an argument that every layer needs a requirement. “The installer offered it” is not one.

If host encryption is selected, the decision record must name:

• what threat it mitigates;
• who can unlock after a remote reboot;
• whether the console path works when WAN and SSO do not;
• where the recovery key is escrowed;
• how an unattended kernel update and reboot is recovered.

Do not claim lights-out operation until that unlock path has been drilled.

Step 2 — Disconnect or protect every data disk before installation#

TL;DR: Inventory and physically disconnect every data disk so the installer can erase only the serial-number-verified OS device.

Work profile: operational size medium; quota small; human effort large; agent effort small; wait small; outage new server powered down; physical storage handling; household services unaffected; clock duration unknown until target-specific evidence exists.

An installer warning is not a guardrail. Physical absence is.

1. Inventory the motherboard, CPU, memory modules, NICs, storage controllers, and every attached drive by model and serial number.
2. Photograph the cabling and label each port, bay, and drive.
3. The owner decides which existing files must survive. The agent may inventory names, sizes, and locations, but it does not decide that a file is disposable.
4. Copy every file that must survive to another drive or machine that will not be erased with the OS drive. For the independent copy, use a separate device and, when practical, do not keep every copy in the same enclosure or on the same power path. Record an opaque destination reference and confirm that the copy will remain available after the OS device is wiped; an offline or removable copy is valid. If the agent already has permission to read the source and write the destination, it performs and records the copy. Otherwise it prepares one complete copy-and-verify packet for the owner to run.
5. The agent compares source and destination counts, sizes, copy errors, and hashes or an application-native verification appropriate to the data. The owner then opens or parses a meaningful sample from each folder or data set being kept. A successful copy command, a count alone, or one readable sample is not enough. If this would leave the destination as the only wanted copy, make another independent copy.
6. If the owner intentionally discards anything previously marked must-survive, record a separate explicit owner decision acknowledging irreversible loss, reclassify that item as disposable, and issue a fresh destructive authorization. That decision is never a successful preservation result.
7. Power the machine down and disconnect every intended data drive at the drive end.
8. Leave connected only the dedicated OS device and installation media.
9. In firmware, confirm that the OS device is the only plausible installation target.

Do not start installation or erase the OS drive until the copies of everything still marked must-survive have passed these checks. Missing evidence means stop, not assume the copy worked.

If there is no separate OS device, buy one before continuing. A modest reliable SSD is cheaper than designing every future storage change around an OS mixed into the data array.

The agent may prepare the inventory, compare serials, and record the preservation evidence. The owner decides what must survive, performs the physical disconnect, opens the representative files from the copy, and confirms the one device named by the separate current destructive authorization. The Linux bootstrap record describes the intended host; it is not that authorization. No conversational phrase such as “use the empty disk” substitutes for a live match of the protected serial, WWN/EUI, by-id path, signatures, and known contents.

Ubuntu's own installation guide warns that partitioning is destructive and that mistakes can lose the disk's contents. Follow its basic installation guide for the current screens, not an old screenshot blog post.

Step 3 — Test the used hardware#

TL;DR: Update and record firmware, then test memory, sustained load, cooling, ports, and drive health before working around hardware instability in software.

Work profile: operational size large; quota small; human effort medium; agent effort medium; wait XL; outage new server unavailable; no household/storage service disruption; hardware under stress; clock duration unknown until target-specific evidence exists.

Used gaming-PC parts are perfectly reasonable server parts, but prior success at a different workload is not a burn-in result.

Before installing:

• Update motherboard firmware from the manufacturer while the machine is local and empty.
• Replace a dead or suspect CMOS battery; verify that time and firmware settings survive loss of AC.
• Load firmware defaults, then record every change.
• Select AHCI or a true HBA/JBOD mode. Do not create firmware or motherboard “fake RAID” volumes in front of the future software-defined pool.
• Enable virtualization extensions and IOMMU if stable on this platform.
• Set “restore on AC power loss” to the chosen behavior, but drill it only after clean shutdown works.
• Disable network boot unless it is deliberately used.
• Run at least one full memory test and investigate any error. Memory errors are not acceptable because a filesystem is redundant.
• Run a sustained CPU/memory load while observing temperatures, throttling, and unexpected resets.
• Exercise every NIC and storage port that will be used.
• Obtain complete SMART/NVMe health and self-test results for each drive before it becomes authoritative storage.

smartctl and smartd cover ATA, SCSI, and NVMe devices, but their output is evidence rather than an oracle. A drive can fail with a clean SMART summary; a nonzero raw attribute is not automatically fatal without model-specific meaning. Preserve the raw report and self-test history so a later change is visible. The smartmontools documentation describes the two tools and supported device families.

Stop if memory errors, link resets, thermal shutdown, unexplained kernel errors, or unstable firmware appear. An LLM must not “work around” suspect hardware with service restart loops.

Step 4 — Install the smallest useful server#

TL;DR: Verify official media, install a minimal UEFI system onto the isolated OS disk, and capture a complete baseline before adding packages or services.

Work profile: operational size large; quota medium; human effort large; agent effort medium; wait large; outage new server unavailable; authorized OS disk erased; data disks disconnected; clock duration unknown until target-specific evidence exists.

Use official installation media. Verify both the signed checksum list and the ISO checksum; Canonical provides a current image-verification procedure.

During installation:

1. Boot the USB installer explicitly in UEFI mode.
2. Use wired DHCP. Do not invent a static LAN address in the installer.
3. Do not configure a proxy, custom mirror, or third-party repository unless the discovered network requires it.
4. Select only the serial-number-verified OS device.
5. Use the whole OS device with a plain ext4 root filesystem. Review the final destructive summary one last time.
6. Give the host a functional, durable name such as node1, not a joke, a person's name, or the application it happens to run first.
7. Create one named human administrator. Do not call it admin, root, claude, codex, or agent.
8. Install the OpenSSH server only if the installer can do so without importing an account-wide key set you have not reviewed.
9. Select no optional application snaps or convenience stacks.
10. Reboot, remove the installer, and confirm it boots from the intended device.

The host name identifies the machine. DNS aliases later identify services. If node1 initially serves files and later becomes a compute node, the machine name remains honest while files.home.arpa moves.

After the first login, record before changing:

OS release and kernel
firmware/BIOS version
boot mode and Secure Boot state
motherboard, CPU, RAM quantity
NIC name, MAC, driver, and negotiated speed
OS-disk model, serial, by-id path, partition UUIDs, and filesystem UUID
default route, DHCP address, DNS servers, and time-sync state
installed package sources
failed systemd units and high-priority boot errors

The exact collection commands belong in a generated, reviewed discovery packet. Typical read-only sources include hostnamectl, timedatectl, ip, ss, lsblk, findmnt, lspci, ethtool, systemctl, and journalctl. Capture structured or complete output; do not pipe everything through a fragile regular expression and then treat absence from filtered output as proof.

Step 5 — Learn how to inspect the host#

TL;DR: Recognize the systemd, journal, socket, mount, disk, route, and time evidence needed to distinguish running state from healthy, durable configuration.

Work profile: operational size small; quota small; human effort medium; agent effort small; wait none; outage none; no service/network/storage/household disruption; clock duration unknown until target-specific evidence exists.

The owner does not need to become a Linux administrator before the agent helps. The owner does need to recognize the evidence an agent should present.

systemd is the supervisor that starts services and orders much of boot. A unit's active and enabled states answer different questions; neither proves health. journalctl reads the structured system journal and can filter by unit and boot. See the upstream journalctl manual.

When changing a packaged service, put all of the following inside one reviewed execution artifact. The owner runs one command; the artifact performs the checks, change, reboot test where required, and evidence capture before it reports success:

• inspect the effective definition with systemctl cat NAME;
• use the application's supported .d fragment or a systemd drop-in;
• never edit the vendor unit under /usr/lib/systemd/system or /lib/systemd/system;
• validate the candidate syntax before reload;
• compare the effective post-change state;
• prove behavior after reboot.

This pattern is less brittle than search-and-replace inside a vendor file. Ubuntu also recommends package-specific configuration fragments and systemd overrides rather than altering packaged files; see Changing package files.

Step 6 — Install security updates automatically and schedule reboots#

TL;DR: Keep automatic security updates, use deliberate recovery-ready reboot windows, and treat every third-party package source as a recorded maintenance dependency.

Work profile: operational size medium; quota medium; human effort medium; agent effort medium; wait medium; outage planned server reboot; new server only; later use declares affected services; clock duration unknown until target-specific evidence exists.

First update the official Ubuntu packages and reboot once while the server is still empty. Use one bounded update artifact that records the full package transaction, preserves the previous boot option, installs the update, arranges its own post-reboot continuation, and checks automatic-update state plus the real access and service outcomes before reporting success.

Ubuntu Server includes unattended-upgrades and applies security updates automatically. That is the default to keep. Do not disable security updates merely to avoid operational thinking. Do not enable automatic reboot yet: a reboot can stop household access, require disk unlock, or expose boot-order defects. Ubuntu documents the default behavior and drop-in configuration in Security updates.

The operating rule is:

• install security updates automatically;
• surface failed update jobs and a pending reboot as actionable state;
• reboot within a defined maintenance window after checking lower-tier dependencies;
• capture the previous kernel and boot entry before reboot;
• confirm SSH, time, firewall, mounts, and failed units afterward;
• have console or direct-cable recovery available for any kernel, bootloader, firmware, or network change.

Use official distribution packages by default. Every PPA, third-party APT repository, downloaded .deb, curl-to-shell installer, and vendor bootstrap script adds another party that can execute privileged code during updates. A third-party source needs a recorded owner, purpose, signing-key fingerprint, supported releases, update path, and removal procedure. “The README said to paste this” is not approval.

This homelab does not pretend to have an enterprise staging system. The owner may run apt upgrade. Planned updates still use one execution artifact, and that artifact carries out this sequence internally:

1. Separate read-only discovery of available updates from installation.
2. Record current package sources, versions, configuration, kernel/boot entry, and the proposed package transaction.
3. Classify urgent security work separately from routine, feature, platform, and firmware updates.
4. Establish the real recovery point—snapshot where supported, restorable configuration/data, retained package or image, recovery media, and independent console/direct-cable access.
5. Declare affected tiers and expected service, network, storage, and household disruption.
6. Update one bounded layer, then run package-manager health, failed-unit, listener, SSH, time, firewall, mount, and representative consumer tests before touching the next layer.
7. If the package manager cannot reliably downgrade, state whether recovery means booting the previous kernel, reinstalling a pinned package, restoring a snapshot, rebuilding from desired state, or asking for a reviewed fix-forward.

If the owner performs an unplanned upgrade, reconcile it without blame: capture the new versions and transaction log, re-run the same admission and outcome tests, record drift, and decide whether the observed state should become desired state or be returned to the last known good state. Rollback remains the default recommendation after a failed update, but the agent preserves evidence and always asks before invoking it.

Step 7 — Test human SSH before disabling password login#

TL;DR: Prove a dedicated human key and local recovery credential before disabling remote passwords, then validate and reload SSH without closing the known-good session.

Work profile: operational size medium; quota small; human effort medium; agent effort medium; wait small; outage none; server management path at risk; console remains available; clock duration unknown until target-specific evidence exists.

Keep a local console available and the existing session open. Steps 1–5 are one complete human key-setup packet. Steps 6–9 run inside one lockout-safe configuration artifact and one invocation; it seals the full SSH include set, runs Bash and fixture checks where used, runs sshd -t and sshd -T, reloads SSH, proves a new allowed login and a denied password login, and only then reports success.

1. Generate or select a dedicated human homelab key on the Mac mini. Do not reuse an identity that grants broad work or cloud access.
2. Add only its public key to the named human administrator.
3. Connect from the Mac mini using the server's direct LAN IP and verify the expected host-key fingerprint out of band.
4. Open a second SSH session and prove key authentication through the intended human credential flow. A deliberate local password-manager or hardware-key approval is acceptable for human administration; an online-only prompt or any prompt that an unattended agent must satisfy is not.
5. Store the server host-key fingerprints in the recovery record.
6. Only then stage an SSH server drop-in that disables remote root login and password/keyboard-interactive authentication and limits login to named accounts.
7. Validate the complete effective configuration with sshd -t and inspect it with sshd -T.
8. Reload SSH; do not kill the proven session.
9. Prove a new human session works and a password login fails before closing the old one.

The intended policy is equivalent to:

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowUsers <human-admin> <agent-account>

The exact fragment and file order depend on the installed release, so the agent must inspect the effective configuration before generating it. OpenSSH documents these controls in the upstream sshd_config manual.

Disabling SSH passwords does not remove the local console password. The owner still needs a non-cloud local recovery credential stored outside the server. Test it at the physical console before considering SSH hardened.

Do not expose TCP 22 on the broadband router. Remote SSH will arrive later over the authenticated overlay network.

Step 8 — Give the agent its own account#

TL;DR: Give the agent a dedicated locked-password account and constrained key with no administrative groups, human secrets, protected data, or broad forwarding.

Work profile: operational size medium; quota medium; human effort medium; agent effort medium; wait small; outage none; server identity/SSH policy only; clock duration unknown until target-specific evidence exists.

Create a separate local account such as homelab-agent. Its password is locked; it authenticates with a dedicated key held by the always-on control host.

Keep the initial group membership simple:

• its own primary group;
• no sudo or wheel;
• no docker, lxd, libvirt, disk, or similar control group;
• no broad human-share group;
• no adm or unrestricted journal group merely for convenience;
• no access to the human's home directory, SSH agent, password manager, browser profile, or private keys;
• no secret-bearing environment file.

Prefer a root-owned observation exporter that returns bounded, redacted fields. During bootstrap, the agent may instead need an unprivileged shell, a workspace it owns, and read access to the deliberately model-safe parts of desired state. That shell is not read-only: it can change its own files and use whatever outbound network the host permits. It does not need package-install authority or arbitrary web access on the server. Many useful observations—interfaces, routes, mounts, processes, package versions, socket state, filesystem capacity, and public service status—do not require root.

Constrain its SSH authorized key to the control host's expected source and disable TCP, agent, and X11 forwarding and PTY allocation unless an observed requirement justifies one of them. If a forced exporter is used, the key invokes only that root-owned exporter. If a general shell is temporarily used, record the residual egress and data-exposure risk, keep all target permissions weak and revocable, and keep exact device and household identity mappings outside its readable files.

Privileged observations that reveal secrets or protected logs should later be exposed through a root-owned observation interface that returns bounded, redacted fields. Privileged changes should later use typed operations such as “deploy approved release X to service Y” or “reload validated Caddy candidate Z,” not sudo bash, arbitrary systemctl, arbitrary file editors, or a writable Docker socket.

During this bootstrap gap, the human may run one pre-reviewed privileged discovery packet and return its output. That is preferable to granting a permanent broad exception because the typed broker has not been built yet.

The first admission test#

TL;DR: Demonstrate both useful workspace access and enforced denial of sudo, control sockets, secrets, devices, forwarding, and system writes, including revocation after reboot.

Work profile: operational size medium; quota medium; human effort small; agent effort medium; wait small; outage none; no household disruption; allowed and denied server paths tested; clock duration unknown until target-specific evidence exists.

From the agent account, prove that:

• SSH works noninteractively from the Mac mini;
• sudo -n true fails;
• Docker/LXD/libvirt sockets are absent or inaccessible;
• the human home and secret directories are inaccessible;
• raw block devices cannot be opened for writing;
• protected system logs are not broadly readable;
• the agent can create and delete files in its workspace but cannot write system configuration, service data, or human shares;
• SSH forwarding attempts fail, and a forced exporter—if used—cannot be bypassed with a requested command;
• a new login still works after reboot;
• removing the agent's authorized key revokes access without changing the human account.

Record both allowed and denied results. A permission model has not been tested if only the happy path was exercised.

Step 9 — Apply a host firewall without locking out recovery#

TL;DR: Default-deny inbound traffic, allow SSH only from named recovery sources, and test both allowed and denied paths while an independent console remains available.

Work profile: operational size large; quota medium; human effort medium; agent effort large; wait small; outage possible management interruption; server network path at risk; console/direct cable required; clock duration unknown until target-specific evidence exists.

The consumer router is not the server's only inbound boundary. Devices on a flat household LAN can reach each other, and a compromised adult laptop should not automatically become a server administrator.

Ubuntu's default host-firewall frontend is UFW, and it begins disabled. The initial policy should deny unsolicited inbound traffic and allow outbound traffic. Add source-constrained rules before enabling it:

• SSH from the Mac mini's reserved LAN address;
• SSH from the one direct-cable recovery address on the intended interface;
• later, SSH from the overlay interface or exact owner identity path;
• no application ports yet.

Do not use a broad “allow SSH from anywhere” rule. Keep a console or direct path open. One timed, lockout-safe firewall artifact must check its Bash and any input fixtures, run the UFW dry-run, apply the complete ruleset, probe an allowed source and a denied household source, capture the result, and report success only after both probes pass. The owner approves any timed return-to-old-rules behavior before the invocation. Ubuntu documents address- and subnet-constrained rules and --dry-run in its firewall guide.

UFW is not the future container boundary by itself. Docker-published ports can interact with packet filtering in ways that bypass an expected UFW path. The service chapter therefore binds ordinary application backends to loopback, centralizes ingress, and validates reachability from an external client.

Step 10 — Build the direct-Ethernet recovery path now#

TL;DR: Make the labeled cable, adapter, collision-free addresses, firewall rule, credentials, and reboot drill real before router or switch failure removes ordinary access.

Work profile: operational size medium; quota medium; human effort large; agent effort medium; wait medium; outage planned server network interruption; household network unaffected; clock duration unknown until target-specific evidence exists.

“The recovery card says direct link” is not a recovery path. The cable, adapter, addresses, firewall rule, keys, and test must all exist before the router fails.

Hardware to label and reserve#

TL;DR: Reserve and label a known-good cable, Mac adapter, server port, and port-to-MAC mapping for direct recovery.

• One known-good Ethernet cable long enough to reach the server.
• A USB-C/Thunderbolt Ethernet adapter for the keyboarded Mac if it lacks a spare Ethernet port.
• The exact server NIC or port used for recovery.
• A printed mapping from server port label to MAC address.

Modern Ethernet ports use auto-MDI/MDIX; a normal patch cable is sufficient. Apple likewise documents using a standard cable rather than a crossover cable when connecting Macs directly in its direct Ethernet guide.

Address plan#

TL;DR: Use a collision-free private recovery subnet with no gateway, DNS server, or search domain so it cannot attract ordinary traffic or depend on name resolution.

During network discovery, reserve one RFC 1918 subnet that does not overlap:

• the home LAN;
• the Tailscale range;
• any site-to-site or work VPN;
• Docker/Kubernetes networks you expect to use;
• another recovery network the laptop may encounter.

For illustration only, the record might assign 10.203.0.1/24 to the Mac recovery adapter and 10.203.0.10/24 to node1. Use the discovered collision-free values, not these literals.

The recovery profile has no default gateway, no DNS server, and no search domain. It should never attract Internet traffic or make recovery depend on name resolution.

Server configuration#

TL;DR: Bind the recovery address to the verified NIC, validate the complete Netplan candidate safely, and permit SSH only from the exact Mac recovery source.

Prefer a second physical NIC if the motherboard already has one or a supported inexpensive adapter is available. Configure that interface by MAC address, because Linux interface names can change when hardware moves. A Netplan candidate has this shape:

network:
  version: 2
  renderer: networkd
  ethernets:
    recovery-link:
      match:
        macaddress: "<RECOVERY_NIC_MAC>"
      set-name: rescue0
      dhcp4: false
      dhcp6: false
      accept-ra: false
      link-local: []
      addresses:
        - "<SERVER_RECOVERY_IP>/<PREFIX>"
      optional: true

There is deliberately no routes or nameservers block. Netplan documents matching an interface by MAC and assigning static addresses in its YAML reference and interface-matching guide.

If the server has only one NIC, add the collision-free recovery address as a secondary static address on the normal DHCP interface. Record that a switch failure requires moving that cable directly to the Mac. Do not replace the normal DHCP configuration with a static-only profile merely for the drill.

The agent renders the complete candidate from discovered interface state and shows the semantic diff. The Netplan file, firewall rule, Bash/Python helpers, and every include form one sealed dependency set. One lockout-safe artifact runs YAML and input fixtures, netplan generate, the explicitly owner-approved netplan try transaction, live address and route checks, plus allowed and denied SSH probes. It verifies the actual state after confirmation or timeout and reports success only when every result matches. Netplan itself warns that rollback still needs checking. See netplan try.

Match the firewall rule to the actual design:

• With a dedicated recovery NIC, allow TCP 22 on rescue0 only from the Mac recovery address.
• With one NIC, allow TCP 22 on the normal interface only from the Mac recovery address to the server's secondary recovery destination address. Do not write a rule for nonexistent rescue0, and do not turn the whole recovery subnet into an SSH source.

Do not allow the recovery subnet on application ports. Test the selected rule while the ordinary LAN path is still open, then prove the Mac source succeeds and a second source fails.

Mac configuration#

TL;DR: Give the dedicated rescue adapter a manual address with no router or DNS and verify that recovery traffic actually traverses the cable.

On the dedicated adapter's Ethernet service, choose manual IPv4 and enter the Mac recovery address and mask. Leave router and DNS blank. Give the network service a recognizable name such as “Homelab Rescue.” Apple documents manual IPv4 entry under Ethernet in TCP/IP settings.

Keep Wi-Fi enabled during an ordinary test so the Mac retains Internet access, but prove that SSH traffic to the recovery address traverses the cable. During the destructive drill, disable Wi-Fi and disconnect the server from the switch.

The drill#

TL;DR: Remove ordinary networking, connect the labeled rescue path, verify SSH and host identity across restart, then restore normal routing without leakage.

Work profile: operational size medium; quota tiny; human effort medium; agent effort small; wait medium; outage planned server network interruption; household network unaffected; clock duration unknown until target-specific evidence exists.

1. Record the current server and Mac routes.
2. Disable the Mac's Wi-Fi.
3. Disconnect the server's normal LAN cable or power off the switch used by it.
4. Connect the labeled Mac adapter directly to the labeled server port.
5. Ping the server's recovery IP.
6. Verify the recorded SSH host-key fingerprint and open a human session.
7. Verify that DNS lookup and Internet access are not being supplied through the rescue interface.
8. Restart SSH and confirm a second connection.
9. Reboot the server once and reconnect by recovery IP.
10. Restore normal cabling and confirm the rescue interface did not become the default route.

Do not defer this drill until after the rack is difficult to reach. A failed test now is cheap information.

Step 11 — Set mount and service dependencies before adding services#

TL;DR: Identify mounts by expected source and type, make dependent workloads fail closed, and test absence so an empty directory cannot silently collect writes.

Work profile: operational size medium; quota medium; human effort small; agent effort medium; wait small; outage none; no live storage/service disruption yet; clock duration unknown until target-specific evidence exists.

The most dangerous mount failure is a believable empty directory. If /srv/media is normally a data-pool or NFS mount but is absent during boot, Linux still has an ordinary directory at that path. A container can start, write to the OS disk, fill it, and later hide those writes when the intended filesystem appears.

Adopt these rules now:

• A directory's existence never proves that its expected filesystem is mounted.
• A required local or network filesystem is identified by expected source, filesystem type, and mount identity.
• A workload that requires a mount fails closed when that mount or a read/write probe fails.
• Network storage is declared as network storage; _netdev exists for filesystems whose type alone does not convey that fact.
• nofail means the host may continue booting; it does not mean dependent applications may use the empty path.
• Dependent systemd units use RequiresMountsFor= or explicit Requires= plus After= and a preflight check.
• A Compose bind mount uses long syntax with bind.create_host_path: false.
• Reboot ordering and server-unavailable behavior are acceptance tests, not post-incident debugging.

Systemd automatically gives network mounts ordering around network-online.target; _netdev, nofail, and x-systemd.requires= change those dependencies. Read the actual semantics in the systemd.mount manual rather than adding options until a boot “works.”

If an NFS-backed container fails because the network mount was late, fix the declared dependency and readiness test. Do not respond by disabling root squashing, making directories world-writable, adding endless retry scripts, or giving the container host privilege.

Step 12 — Record the intended configuration in Git#

TL;DR: Use private Git as a reviewed text-change ledger for model-safe design and evidence, not as a backup for secrets, databases, or household data.

Work profile: operational size small; quota small; human effort small; agent effort medium; wait small; outage none; no service/network/storage/household disruption; clock duration unknown until target-specific evidence exists.

Git is the change ledger for text, not a backup system and not a configuration engine by itself. The owner needs four concepts:

• A repository is the directory and history for the design records.
• A commit is a named snapshot of reviewed text changes.
• A diff is the exact change between states.
• A remote is another copy of that repository; it is not automatically a backup of server data or secrets.

The agent can perform routine Git mechanics. The human should be shown the semantic diff and validation result, not a lecture about every Git command. The agent must never put passwords, private keys, recovery codes, raw tokens, or unredacted sensitive logs in the repository.

For this host, commit:

• the decision record;
• model-safe hardware and disk inventory using opaque IDs; the human-only identity map remains outside the repository;
• the selected OS release and verified ISO checksum;
• firmware settings and photographs' secure location;
• network intent, reservations, recovery addresses, and host-key fingerprints;
• package-source inventory;
• generated configuration source and validators;
• agent account and grant intent;
• build results and failure-drill records;
• the reinstall procedure.

Keep the repository private and mirrored off the Mac mini. Back up the credential store separately. Git history will help explain and recreate /etc; it will not restore a database, family share, or secret.

Step 13 — Give the agent one complete phase of work#

TL;DR: Authorize one bounded bootstrap so the agent batches discovery and staging while the human performs only physical, secret, destructive, or privileged boundary actions.

Work profile: operational size small; quota small; human effort small; agent effort medium; wait none; outage none; no direct disruption; defines future execution; clock duration unknown until target-specific evidence exists.

Once the owner approves this chapter's bounded bootstrap, the agent should continue through ordinary read-only checks and reversible unprivileged setup without asking “shall I do the next command?” after each success.

Use this operating contract:

Establish the first Linux host through the exit tests in this chapter. First perform and record all authorized read-only discovery. Prepare complete candidates and validators before requesting privilege. Never feed me commands one at a time: for the human boundary, provide one small reviewed script or equivalent with one invocation and the full dependency closure. Build in host/account/path preflight, Bash/Python syntax checks, representative regex fixtures, native configuration validation, postcondition tests, evidence capture, and safe stop behavior. Ask me only for a value judgment, a physical action, a secret entry, a newly privileged/destructive boundary, or any rollback. Never request general sudo, Docker-socket access, or my personal SSH identity. If observed state differs from the approved plan or validation fails, freeze mutation and report the discrepancy tersely. End every response with NEXT ACTION naming the actor, host or UI surface, account, path, exact invocation or wait event, expected result, and resume condition.

Human actions in this phase should converge toward:

1. physically disconnecting or reconnecting named hardware;
2. making installer choices from the recorded decision table;
3. entering a local recovery secret directly;
4. running one reviewed bootstrap script or equivalent transaction, through one invocation, with temporary privilege and built-in preflight, validators, outcome tests, evidence, and safe stop behavior;
5. confirming the firewall/network candidate after an alternate path exists;
6. acknowledging the final drill result.

The agent should not ask again for actions already inside that approved boundary. A compaction or new session does not expand it: reload the durable record, re-observe live state, and resume only when the current facts still match.

Common mistakes#

TL;DR: Reject premature tooling, broad agent privilege, unsafe access cutovers, hidden mount failures, and recovery designs that preserve undocumented drift.

Final test#

TL;DR: Prove cold boot, human and agent access boundaries, firewall denial, DNS-independent access, direct-cable recovery, update/log health, AC return, and off-host reproducibility before reconnecting data disks.

Work profile: operational size large; quota medium; human effort large; agent effort medium; wait large; outage planned new-server/network interruption; no valuable data connected; household network unaffected; clock duration unknown until target-specific evidence exists.

Run this before reconnecting data drives:

1. Cold boot from AC-off state.
2. Log in at the local console with the recovery credential.
3. Reach the host from the Mac mini over its reserved LAN IP.
4. Confirm a human key succeeds, a password SSH attempt fails, root SSH fails, and the agent key reaches only the agent account.
5. Confirm the agent's forbidden group, sudo, secret, and device tests fail.
6. Inspect firewall rules and prove an unauthorized LAN client cannot open SSH.
7. Stop DNS and prove both human and agent can still connect by IP.
8. Power off the relevant switch or disconnect the LAN, attach the labeled rescue cable, and connect by the recovery IP.
9. Reboot through the rescue path and reconnect.
10. Confirm time synchronization, automatic-update health, prior-boot logs, no failed units, expected listeners, and the OS filesystem's free-space floor.
11. Restore normal networking, remove AC after a clean shutdown, restore it, and observe the configured firmware behavior.
12. Using only the off-host records, have a second LLM produce a reinstall plan and identify every missing fact. Do not let it execute the plan.

Exit test: another competent person—or a fresh agent session with no conversational memory—can identify the machine, reach it by two independent local paths, explain every privileged identity, reproduce its base configuration, and state exactly what would be lost if the OS device died now.

Proceed next to Phase 3, which stabilizes remote entry, the household edge, and provisional power protection. Leave the still-unmodified data drives disconnected. Reconnect and inventory them only when Phase 4 begins and the storage erasure boundary, device-record template, and required UPS capacity are ready.