05 — Plan storage, accounts, and recovery

Chapter TL;DR: Inventory and protect every disk, decide what loss is acceptable, set stable user and group IDs, then build file sharing, backup, and restore before moving valuable data.

This chapter turns discovered disks into storage with defined behavior for failure, accounts, sharing, backup, and restore. The design covers disk loss, server replacement, mistakes, stolen credentials, ransomware, and later rebuilds.

The order matters:

1. Protect and inventory every existing disk.
2. Classify the data and decide acceptable loss and recovery time.
3. Choose the pool layout and its growth path.
4. Reserve one numeric identity namespace before creating shared files.
5. Create datasets and access boundaries, not one giant share.
6. Establish an independent backup and prove a restore.
7. Only then migrate valuable data and enable household access.

Recommended baseline:

• A small SSD or NVMe device for the operating system, separate from the data pool.
• OpenZFS on directly attached disks, normally as one two-disk mirror to begin and additional mirror pairs as capacity is needed.
• ZFS datasets that share free space, with quotas used only where one workload must be contained.
• Stable UID/GID assignments recorded before the first file is created.
• Standalone Samba with one account per person for Windows and macOS clients.
• NFS only for trusted Linux hosts and application service accounts, with root squashing preserved.
• Frequent local snapshots for quick mistakes, plus encrypted file-level backups to an independent, deletion-resistant destination.
• A backup writer that can add backups but cannot prune them, and an LLM operator that cannot repartition, format, destroy, detach, or weaken retention.

This baseline gives a technically capable beginner checksummed storage, useful snapshots, straightforward growth, mixed-client file sharing, and a credible recovery path.

The chapter contract#

TL;DR: Begin only with safe prerequisites, divide owner decisions from agent work, and finish only after recovery and denial tests.

Enter only when#

TL;DR: Stop at inventory unless Tier 0 access, safe shutdown, agent denial tests, protected disks, and intended users are established.

• Tier 0 local administration works: the owner can reach the server from the available keyboarded client without the internet, cloud SSO, or the file service itself.
• The storage host can shut down cleanly during a power failure or the owner has explicitly accepted the temporary risk while building it.
• The agent has noninteractive, read-only discovery access and has been tested to ensure that raw-disk writes and general sudo fail.
• No disk containing the only copy of wanted data has been authorized for reuse.
• The owner has named who will use the storage and which applications are likely to need it.

If any of these conditions is false, stop at inventory. Do not “temporarily” build a pool and promise to fix the foundation later.

The minimum Phase 4 path#

TL;DR: Complete one recoverable pool, stable identities, scoped SMB share, independent backup, and tested restore; optional features can wait.

Phase 4 is complete with five outcomes: the selected pool/datasets, the recorded numeric identity plan, one deliberately scoped SMB share for people, an independent backup, and a proved restore. The hot-remote-data analysis is a decision only if that need exists. NFS is just in time for a real Linux application consumer, normally after Docker arrives in Phase 5. Peer backup, directory service, snapshot browsing, and additional shares are later upgrades.

Do not let optional sections turn into blockers. The minimum drill uses Windows and/or macOS clients that the household actually owns, exercises allowed and denied SMB identities, and restores a file plus its ACL into staging. It does not require a container, NFS client, FreeIPA, or second household site.

The owner decides#

TL;DR: The owner sets data value, RPO/RTO, erasure and purchase choices, encryption availability, and bulk-media backup policy.

• Which data would be painful, expensive, embarrassing, or impossible to lose.
• The maximum acceptable data loss for each class—its recovery point objective, or RPO.
• The maximum tolerable outage for each class—its recovery time objective, or RTO.
• Whether silent unattended reboot is worth weaker protection against theft of the entire server.
• Which old disks may be erased, what new hardware may be purchased, and which cloud or peer may hold encrypted backups.
• Whether replaceable bulk media is worth backing up.

The agent does#

TL;DR: The agent inventories, models, prepares plans and one privileged packet, and verifies results without acquiring destructive authority.

• Collects the physical and logical inventory, without changing disks.
• Separates observation from inference and flags unknown or contradictory device identity.
• Calculates usable capacity, headroom, fault tolerance, growth cost, and recovery consequences for the credible layouts.
• Generates the identity registry, dataset plan, ACL matrix, backup policy, migration plan, and exact acceptance tests.
• Stages and validates the approved plan, then prepares one execution artifact whose single invocation performs the mutation, verifies live and client results, and records evidence before reporting success. A separately reviewed fixed action may replace only a repeated, exact step later.
• Monitors health and capacity without acquiring deletion or retention authority.

The human still labels, cables, inserts, and removes hardware. Any command that can erase a disk requires a maintenance-window approval tied to the disk’s durable identity—not merely a name such as /dev/sdb.

Leave only when#

TL;DR: Exit only after import, scrub, client access, restore, rollback preservation, and agent-denial evidence all pass.

• The pool imports after a cold reboot and reports the intended topology.
• A scrub completes without unaccounted errors.
• Windows and macOS clients pass the allow and deny tests for representative adult and child accounts.
• Every configured NFS client, if any, presents the intended service UID/GID; remote root remains squashed.
• A backup exists outside the pool and outside the storage host’s ordinary administrative authority.
• A representative restore succeeds to an empty staging location, and the restored files or application export are actually opened and checked.
• The source copy remains available until the migrated service passes its soak period.
• The agent’s standing credentials are demonstrably unable to destroy the pool, erase disks, overwrite live restores, delete snapshots, prune backups, or change retention.

Understand six different storage promises#

TL;DR: Redundancy, snapshots, replicas, synchronization, hot copies, and backups solve different failures and must not be conflated.

Most storage disasters begin with one mechanism being mistaken for another.

OpenZFS snapshots are read-only point-in-time copies, but they live in the pool whose blocks they preserve. They are excellent undo points and a useful replication source, not an independent backup. The OpenZFS documentation explains both their copy-on-write behavior and their growing space use as live data diverges from them. OpenZFS: snapshots, clones, and bookmarks

The minimum safe pattern is therefore:

live dataset
    ├── local snapshots                 fast undo
    └── encrypted backup repository     independent history
            └── retention authority     unavailable to the source and agent

Adding a mirror improves the first line. It does not remove the need for the second.

Stage 1 — Protect existing disks, then inventory them#

TL;DR: Treat every unknown disk as irreplaceable and prohibit writes until identity, contents, health, and preservation status are known.

Work profile: operational size large; quota large; human effort large; agent effort large; wait large; outage possible server/storage inspection outage; no writes; unknown disks treated as irreplaceable; clock duration unknown until target-specific evidence exists.

Do not install, initialize, partition, format, “repair,” or add any discovered disk to a pool during inventory. Treat every unlabeled drive as if it holds the only copy of something important.

Identify disks by durable hardware IDs#

TL;DR: Bind physical disks to durable identifiers and keep exact identities separate from the model-safe inventory.

Linux device letters depend on discovery order and can change after a reboot, cable move, or controller change. The durable record uses serial number, WWN/EUI, model, capacity, and the /dev/disk/by-id/ link. Record the chassis bay and cable label too; software identity cannot tell the owner which physical drive to pull unless the physical map exists.

The inventory for every HDD, SATA SSD, and NVMe device includes:

• Manufacturer, exact model, serial number, WWN/EUI, firmware, and warranty status.
• Advertised capacity, logical sector size, physical sector size, and rotational status.
• SATA/SAS/NVMe/USB transport, controller, port or bay, cable label, and power path.
• SSD endurance used, or HDD recording technology—CMR, SMR, or unknown.
• Power-on hours, temperatures, media/data-integrity errors, reallocated or pending sectors where applicable, self-test history, and NVMe critical warnings.
• Existing partition table, filesystem signatures, labels, UUIDs, mounts, encryption, RAID membership, and known contents.
• Whether this is the only copy, may be erased, should be quarantined, or is eligible for the proposed pool.

smartctl and smartd are the standard smartmontools programs for ATA, SCSI, and NVMe health data. Their current source documents --scan-open, full device reporting, JSON output, and background self-tests. smartmontools source documentation and smartctl option definitions

A read-only evidence capture on Linux will normally draw from commands like these:

lsblk --json --output NAME,PATH,SIZE,TYPE,MODEL,SERIAL,WWN,TRAN,ROTA,LOG-SEC,PHY-SEC,FSTYPE,FSVER,LABEL,UUID,MOUNTPOINTS
findmnt --json
sudo smartctl --scan-open
sudo smartctl -x --json=o /dev/disk/by-id/REVIEWED-DEVICE-ID
sudo wipefs --no-act /dev/disk/by-id/REVIEWED-DEVICE-ID

These are examples of evidence, not an instruction to grant arbitrary sudo. On day one, the human runs one reviewed read-only discovery packet and retains the full output locally. Keep two views: an authoritative local inventory with exact serial/WWN/path mappings, and a model-safe view that substitutes opaque IDs such as disk-a while preserving size, sector geometry, health, transport, and proposed role. The agent can design topology from the second view.

Before a destructive packet is rendered, choose one explicit boundary. A clean independent model may compare exact device identifiers only when the owner has consented to that selected provider after reviewing its data controls, or when an existing separately reviewed local protected mechanism performs the comparison without exposing arbitrary mappings. Otherwise the owner keeps the authoritative mapping private, resolves the opaque IDs into the protected erasure table, and verifies the exact physical device. An independent model may still review the sanitized table's structure, calculations, stop conditions, and opaque-ID consistency, but its verdict must say that it did not verify the exact physical identity. The model that authored the packet is not its independent reviewer. No second human is required. Do not claim an unimplemented local mechanism as protection, and never fall back to /dev/sdX.

SMART is evidence, not a warranty. A drive that passes can fail tomorrow, and a drive with unexplained errors is not rehabilitated by clearing its log. Before running extended self-tests on a drive that may contain the only wanted copy, copy the data elsewhere. Run long tests one drive at a time and outside peak service load.

Inspect the path to each disk#

TL;DR: Validate controller, bridge, cable, enclosure, cooling, and power paths because storage reliability includes everything between disk and OS.

A good filesystem cannot compensate for an unreliable USB bridge, marginal cable, overloaded power supply, hidden hardware RAID, or an enclosure that reuses serial numbers.

The default storage path is:

disk → SATA/SAS/NVMe → HBA or motherboard controller → operating system → ZFS

Pass disks through individually. Do not put hardware RAID underneath ZFS. OpenZFS recommends an HBA rather than a RAID controller because hidden redundancy and caching prevent ZFS from seeing device identity, sector behavior, and the individual copy needed for self-healing. OpenZFS hardware guidance

USB is acceptable for a disconnected backup disk and can be acceptable for a single noncritical device with a proven bridge. It is not the first choice for a multi-disk primary pool. If shallow, quiet hardware cannot hold the required disks, prefer a quiet tower or shallow 3U/4U chassis before buying an old, deep, fan-heavy enterprise disk shelf. Add an external SAS shelf only when bay count, independent replacement, or compute/storage separation justifies its depth, power, noise, cabling, and new failure points.

Classify the inventory result#

TL;DR: Assign every disk exactly one state—preserve, quarantine, backup-only, cold spare, or explicitly erasable pool candidate.

Every disk exits discovery in exactly one state:

• Preserve: contains wanted data or its status is uncertain. No writes.
• Quarantine: health, cabling, identity, or provenance is suspect. It may be tested after its data is safe, but it is not pool material.
• Backup-only: adequate for a secondary, normally disconnected copy whose failure is tolerable and monitored.
• Cold spare: healthy, tested, labeled, large enough, and stored ready for a specific vdev.
• Pool candidate: healthy, directly identifiable, compatible with the intended peers, and explicitly authorized for erasure.

“The agent found no mount point” is not erasure authorization.

Stage 1 exit test: the owner can point at each physical drive and match it to one unambiguous inventory record; every preservation decision is recorded; and the agent’s raw-disk write attempt is denied.

Stage 2 — Classify the data and set recovery targets#

TL;DR: Define data value, consistency, recovery, retention, and confidentiality requirements before designing capacity or redundancy.

Work profile: operational size medium; quota medium; human effort large; agent effort medium; wait small; outage none; no technical disruption; owner value decisions; clock duration unknown until target-specific evidence exists.

Start with outcomes, not terabytes. A useful initial classification is:

Secrets and encryption keys need their own recovery plan. They may protect Class A data, but placing the only decryption key inside that data’s backup is not a plan.

For each dataset, record:

• Owner and allowed users or services.
• Confidentiality and the consequence of disclosure.
• Current size, measured monthly growth, snapshot churn, and expected two-year size.
• RPO: how much recent work may be lost.
• RTO: how long restoration may take.
• Application consistency requirements.
• Local availability requirement during internet loss.
• Backup destinations, retention, and estimated restore bandwidth.
• The exact restoration test that proves the data is usable.

If the owner cannot yet choose an RPO, use these starting points: one day for Classes A and B, the application’s last successful consistent export for Class C, and no promise for D or E. Tighten the schedule only when the value justifies the storage, bandwidth, and alerting cost.

Allow for free space and growth#

TL;DR: Size for usable redundancy, growth, snapshots, restore workspace, and operating margin rather than label capacity.

For planning, use the smallest member of a mirror or RAIDZ vdev, subtract parity or mirror copies, allow for filesystem overhead and snapshots, and then preserve operating headroom. OpenZFS recommends keeping more than 10% free to avoid the allocator’s expensive low-free-space behavior. This guide is deliberately more conservative: warn at 70%, plan expansion at 80%, and treat 90% as an incident. OpenZFS workload tuning: free space

Plan enough space for:

current protected data
+ two years of measured or estimated growth
+ snapshot churn during the longest retention window
+ restore and migration workspace
+ at least 20% routine free-space margin

Put caches, downloads, transcodes, and backup staging behind quotas so they cannot consume the margin reserved for family data or application state.

Stage 2 exit test: every planned dataset has a class, owner, RPO, RTO, growth estimate, and restore test; the capacity model still has at least 20% routine headroom at the planning horizon.

Stage 3 — Choose a storage layout you can grow and recover#

TL;DR: Select a layout whose failures, expansion steps, irreversible choices, and physical constraints the owner can explain.

Work profile: operational size XL; quota XL; human effort large; agent effort XL; wait large; outage none before authorization; future storage blast radius; no disk mutation in design step; clock duration unknown until target-specific evidence exists.

Separate the operating system from the data#

TL;DR: Keep the rebuildable OS/runtime separate from durable state so replacing the host does not require touching data disks.

Use a small SSD or NVMe device for Linux and the container runtime. Put durable application state and shared files on the data pool. The host should be rebuildable without repartitioning or reformatting the data disks.

The boot device may be a single SSD if a same-day rebuild is acceptable. Mirror it only if its downtime justifies the extra device. In either case, back up desired-state files, package/container manifests, identity records, secret references, and restore instructions. A byte-for-byte operating-system image is optional; reproducibility is the primary recovery method.

Do not let convenience turn the boot disk into an undocumented second data pool. If an application writes durable data under /var/lib, map that state deliberately into its data dataset or document and back it up as an exception.

The topology decision tree#

TL;DR: Prefer mirrors for small mixed-use pools, consider RAIDZ2 for matched sequential workloads, and never stripe wanted data without redundancy.

Only one usable disk?
├─ Yes → no redundancy exists. Use ZFS single-disk for checksums/snapshots
│        or ext4 for maximum simplicity; establish independent backup first.
└─ No
   ├─ Two disks → one two-way ZFS mirror. This is the default starting pool.
   ├─ Three disks
   │  ├─ capacity is not binding → two-way mirror + tested cold spare/backup disk
   │  └─ capacity is binding → RAIDZ1 is acceptable only with proven backup
   │                         and explicit one-disk-failure risk acceptance
   ├─ Four disks → two mirror vdevs by default; RAIDZ2 when sequential capacity
   │               and tolerance of any two disk failures matter more than growth/IOPS
   └─ Five or more matched disks
      ├─ mostly sequential bulk data → RAIDZ2 is credible
      └─ VM/database/random IO or pairwise growth → mirrored vdevs

Do not use RAID0 or a ZFS pool containing a nonredundant top-level data vdev for wanted data. ZFS stripes across top-level vdevs; losing any one of them loses the pool. Adding one bare disk to an otherwise mirrored pool therefore removes the pool’s effective redundancy. OpenZFS: vdevs and pool failure behavior

For a small mixed-use homelab, mirrors are the default because they:

• Start with two disks.
• Deliver good random I/O for application state as well as sequential file service.
• Grow naturally by adding another matched mirror pair.
• Allow one side to be replaced and resilvered at a time.
• Make later replacement and recovery easier to reason about.

RAIDZ2 becomes attractive when there are enough similarly sized disks, the workload is mainly large sequential files, bay efficiency matters, and the owner accepts a wider failure and maintenance domain. RAIDZ1 is not forbidden, but it spends all redundancy on a single failure. It is a poor place for the only copy of irreplaceable data and a poor excuse for skipping backup.

Drive redundancy is not chassis, controller, power-supply, filesystem, or operator redundancy. A combined app-and-storage server can be the right first machine; its RTO must simply acknowledge that one motherboard failure takes both roles down.

Why ZFS is the default#

TL;DR: ZFS provides one coherent checksummed, repairable, snapshot-capable storage model without unnecessary day-one tuning layers.

ZFS combines the pool manager, filesystem, end-to-end data checksums, redundant repair, datasets, snapshots, quotas, encryption, scrub, and replication semantics in one coherent layer. It detects corruption with checksums stored separately from the blocks they describe and, when a valid redundant copy exists, can repair the damaged copy. OpenZFS: checksums and OpenZFS: scrub and resilver

That coherence is more valuable here than squeezing the last terabyte from mismatched leftovers. It also avoids guessing fixed logical-volume sizes: datasets share the pool and grow with their contents, while quotas contain only the workloads that need containment.

Keep the first pool simple:

• Use whole disks by durable ID and allow ZFS to see them directly.
• Use mirror vdevs unless the recorded data and bay constraints justify RAIDZ2.
• Match vdev redundancy. Never force past a topology-mismatch warning.
• Leave sync enabled. Do not trade correctness for benchmark numbers.
• Enable ordinary compression (compression=on) and leave deduplication off. OpenZFS itself warns not to enable dedup without a specific, sized use case. OpenZFS: deduplication
• Leave record size at the default until a documented workload and measurement justify changing it.
• Do not add SLOG, L2ARC, special, dedup, or spare vdevs on day one. In particular, a lost special vdev can lose the pool and a SLOG is not a general write cache. OpenZFS: vdev types
• Keep automatic replacement off. An agent or controller must not format an unexpected device merely because it appeared in a bay.
• Record the pool GUID, topology, feature flags, sector-size decision, disk IDs, and exact tested import procedure.

Decide ashift before creating each vdev#

TL;DR: Verify sector requirements and record ashift before creation because the choice is effectively permanent per vdev.

ZFS records an alignment value called ashift when a top-level vdev is created. That value is effectively permanent for that vdev: changing the pool default later does not rewrite existing vdevs. A disk or USB bridge may also report 512-byte logical sectors while the media really writes 4 KiB sectors. Trusting that report can produce badly misaligned writes for the life of the vdev. OpenZFS therefore treats alignment as a creation-time decision, not a tuning detail to revisit after data arrives. OpenZFS: alignment shift and OpenZFS: ashift pool property

Use this rule for a new homelab pool:

1. Record both logical and physical sector size for every candidate, plus the controller or bridge through which it was measured. Check the model documentation when the device path may conceal the truth.
2. Use ashift=12 for 4 KiB-native or 512e disks. Also prefer 12 for an old true-512-byte HDD vdev when future replacement disks are likely to use 4 KiB sectors; the small possible capacity cost is usually preferable to locking a new pool to 512-byte alignment.
3. Use a larger value only when the exact flash device, vendor evidence, or a controlled test establishes a larger minimum write unit—for example, ashift=13 for an 8 KiB requirement. Do not cargo-cult 13 onto ordinary disks.
4. For a mixed vdev, satisfy the largest verified requirement among its children. Record one proposed value and its evidence for every top-level vdev.
5. Put the proposed ashift in the pre-creation erasure table. Have a clean independent model check the model-safe geometry and calculation. The owner verifies the protected exact-device mapping unless a consented provider or reviewed local protected mechanism performs that comparison. Verify the resulting vdev value before copying data.

Do not choose ashift=9 merely because LOG-SEC says 512. Do not let automatic detection silently make this irreversible decision when the inventory or replacement plan points to 4 KiB media.

Pool creation is destructive and some layout choices are one-way. This guide does not provide a generic zpool create command. The implementation agent creates one digest-bound script after binding every argument to the approved WWN/serial inventory inside the selected privacy boundary. The script tests its Bash/Python and any ID-matching regex with safe fixtures, runs the supported dry run, repeats the live device-signature checks, creates only the approved pool, verifies the resulting pool GUID, vdev topology, devices, and ashift, records the evidence, and reports success only after those postconditions pass. A terse protected erasure table, owner verification, and independent review of every field the reviewer is allowed to see come before the owner runs its one exact command. A future fixed action may replace that human step only after independent review of device binding, expiry, replay protection, and failure handling.

Encryption is a recovery decision#

TL;DR: Choose dataset encryption and key-loading behavior by balancing theft protection against unattended recovery, then test an off-host recovery key.

Use separate encrypted datasets for private Classes A–C and unencrypted datasets only where the confidentiality decision explicitly permits it. ZFS encryption is selected when a dataset is created and cannot simply be turned on for its existing blocks later. Its key format and location also determine whether mounting will block for human input. OpenZFS dataset properties: encryption and key location

Choose one of these policies deliberately:

• Lights-out default: a root-only key on the separate boot device unlocks the private datasets at boot. This protects removed or separately stolen data disks and simplifies disposal, but it does not protect against theft of the running or complete server.
• Stronger physical-theft protection: a human supplies the key after reboot, or an independent device releases it under a narrow policy. This improves theft resistance but adds a Tier 2 dependency and may turn a routine outage into a manual recovery.
• Most-sensitive exception: a small manually unlocked dataset holds material whose confidentiality is worth delayed availability; normal household shares still unlock automatically.

Keep a recovery copy of each key outside the server and outside the pool, protected in the owner’s secret store. A sole recovery-bundle or key copy must not sit behind the failed host, inside the manually locked dataset it is needed to open, or in the backup it is needed to decrypt. An encrypted secondary copy is valid when its key follows an independent recovery path. Never place a key on a shell command line, in chat, in Git, or unprotected on a printed card. Test the key path with the server unavailable before migration.

Credible alternatives#

TL;DR: Choose ext4, MD/LVM, Btrfs, or an appliance only when its tested recovery path is simpler for this owner.

The Linux kernel documents ext4’s journal as protection against inconsistent metadata after a crash; that is useful but different from ZFS checksumming and redundant repair. Linux ext4 journal documentation Btrfs snapshots, scrub, RAID1, and repair are credible, but its project documentation says RAID56 should not be used for production. Btrfs feature status

Do not choose an alternative merely because an LLM knows a command for it. Choose it because the recovery procedure is simpler for this owner and has passed the same failure and restore tests.

Design growth before filling bays#

TL;DR: Record how capacity expands, which resources constrain it, and how failed disks are identified and replaced before pool creation.

For mirrors, grow by adding another redundant mirror vdev or by replacing every member of one vdev with larger devices, one at a time, waiting for each resilver. For RAIDZ, current OpenZFS supports widening a vdev on versions with RAIDZ expansion; fault tolerance remains the same, old and new blocks may retain different data-to-parity ratios, and the operation rewrites allocated data. Adding a top-level vdev does not rebalance old data automatically. OpenZFS: changing pool layout

Therefore record all of these before creation:

• Empty bays, HBA ports, PCIe lanes, power connectors, cooling, and maximum chassis depth.
• The smallest purchasable expansion unit: one disk, a mirror pair, or a RAIDZ-width change supported by the installed release.
• Whether a larger replacement drive yields no new space until every member of the vdev has been replaced.
• How a failed disk will be identified physically and which tested spare can replace it.
• The capacity threshold and lead time for ordering disks.
• Whether the next growth event should instead separate compute and storage.

Stage 3 exit test: the owner can explain what one disk failure does, what two disk failures can do, how the next capacity increment is added, and what operation cannot be casually reversed. The approved plan contains only pool-candidate disk IDs and still fits the physical power, cooling, controller, and bay inventory.

Stage 4 — Set stable user and group IDs before sharing files#

TL;DR: Allocate stable UID/GID meanings before data exists so shares, restores, and future hosts preserve ownership correctly.

Work profile: operational size large; quota large; human effort medium; agent effort large; wait medium; outage possible identity migration outage; file ownership and access affected; clock duration unknown until target-specific evidence exists.

Linux files store numeric owners and groups. Names are a local rendering. If UID 220005 means plex on one host and a child’s account on another, NFS and restored files do not know which human intention was meant; they see the same number.

This is cheap to prevent and expensive to repair. Establish one registry before the first shared dataset, even if the first implementation uses local accounts.

Reserve ranges, then allocate identities once#

TL;DR: Inventory every existing and future namespace, reserve nonoverlapping ranges, and never casually recycle numeric identities.

First inspect existing /etc/passwd, /etc/group, /etc/subuid, /etc/subgid, NAS accounts, container mappings, and any planned directory-service range. Choose nonoverlapping allocation blocks. If FreeIPA is a credible later destination, reserve its entire future primary range now; do not reserve a small leftover and assume migration will work itself out.

The default strategy below reserves one 200,000-ID FreeIPA primary range. People and shared groups receive their eventual directory IDs from named slices inside it even while local files provision those identities. Service and agent accounts that will remain local are outside it. The first half of the directory range remains free for the installer and automatic allocations. This is a synthetic new-build example, not a universal numbering standard:

schema: 1
numeric_namespace:
  directory_primary:            {start: 200000, end: 399999}
  directory_automatic_headroom: {start: 200000, end: 299999}
  directory_people:             {start: 300000, end: 309999}
  directory_shared_groups:      {start: 310000, end: 319999}
  directory_future:             {start: 320000, end: 399999}
  local_services:               {start: 400000, end: 419999}
  local_agents:                 {start: 420000, end: 420999}

people:
  - {name: alex, uid: 300001, primary_gid: 300001, role: adult, authority: local-declared, directory_target: freeipa}
  - {name: sam,  uid: 300002, primary_gid: 300002, role: adult, authority: local-declared, directory_target: freeipa}
  - {name: kid-a, uid: 300101, primary_gid: 300101, role: child, authority: local-declared, directory_target: freeipa}

groups:
  - {name: household-adults, gid: 310001, members: [alex, sam], directory_target: freeipa}
  - {name: household-kids,   gid: 310002, members: [kid-a], directory_target: freeipa}
  - {name: household-all,    gid: 310003, members: [alex, sam, kid-a], directory_target: freeipa}

services:
  - {name: svc-plex, uid: 400001, primary_gid: 400001, authority: local-declared}
  - {name: svc-backup, uid: 400002, primary_gid: 400002, authority: local-declared}

agents:
  - {name: agent-storage-observer, uid: 420001, primary_gid: 420001, authority: local-declared}

The discovery agent must prove these or any replacement blocks do not collide before they become desired state. The parent directory_primary describes the future FreeIPA installation range; its child slices partition that range and therefore intentionally sit inside it. Those child names are planning labels, not rules automatically enforced by FreeIPA. Once installed, FreeIPA owns the full primary range, so import every preallocated identity promptly and monitor its real allocator state. UID and GID are separate namespaces: a person's same-number private group is intentional, while unrelated allocation blocks within either namespace must be disjoint. Keep subordinate container-ID ranges separate and recorded too.

Rules for the registry:

• Never reuse a retired UID or GID for a different principal.
• Give each person an individual identity, including children.
• Use groups for access policy; do not encode household roles into hundreds of per-file exceptions.
• Give each application its own service identity unless two applications intentionally share write authority.
• Give agents machine identities distinct from service processes and people.
• Record who currently issues each identity, its intended future authority, its allocation block, and its migration state.
• Provision the same name, UID, primary GID, and required groups declaratively on every participating Linux host.
• Back up the registry and include it in every recovery bundle. Restore identity before mounting shared data read-write.

FreeIPA allocates POSIX identities from a deployment ID range. Its primary range is selected during installation, cannot be casually changed later, and defaults to 200,000 identities. Do not let the installer choose a random range if files already carry planned directory IDs. FreeIPA: identity mapping and ID ranges

When FreeIPA becomes justified, preserve ownership rather than renumbering files:

1. Re-run the collision scan across every host, NAS, container subordinate range, and restored-data sample.
2. Install FreeIPA with the exact recorded directory_primary start and size. Stop if the installed range differs.
3. Before creating ordinary directory users ad hoc, stage the registered people and shared groups with their exact existing UID/GID values. Verify that FreeIPA returns those numbers and that its automatic allocator has not assigned one elsewhere.
4. Enroll one non-storage test host. Confirm NSS and group resolution, login, file access, and intentionally denied access before changing a server that owns data.
5. Cut over one participating host at a time. Do not leave a same-named local account shadowing the directory identity; remove or disable it only after the directory identity resolves to the same numbers and a local recovery account remains available.
6. Compare sampled ownership before and after, then update identity_authority and directory_migration in the registry. No recursive chown should be necessary when the numbers were preserved.

If the future directory product or installer cannot accept the registered values, stop and design an explicit, rehearsed ownership migration. A reserved range reduces risk only when the directory is installed with that range and the identities are imported deliberately.

Start with local accounts#

TL;DR: Generate consistent local accounts first and add centralized identity only when measured multi-host drift justifies its recovery dependency.

Use generated local accounts while there is one Linux storage host, a few people, and no need for Kerberos. “Local” does not mean “hand-edited independently.” The versioned registry is the source; the agent renders host-specific desired state, checks for collisions and existing ownership, stages it, validates it, and a root-owned installer applies the bounded change.

Move to FreeIPA when one or more of these become persistent:

• Multiple Linux hosts need the same people, groups, SSH identities, or NFS ownership.
• UID/GID drift or account lifecycle has caused repeated manual work or near misses.
• NFS needs cryptographic user authentication rather than trusting client-supplied numeric IDs.
• Service principals, Kerberos keytabs, host enrollment, or centralized Linux access policy now solve a measured problem.
• The owner can provide reliable DNS, synchronized time, backups, and preferably a second identity replica rather than making one new VM a household-wide login dependency.

FreeIPA combines LDAP, Kerberos, DNS integration, certificate services, host identity, and policy. That is why it is a better later Linux-domain choice than adding bare OpenLDAP and then separately inventing Kerberos, enrollment, certificate, and lifecycle machinery. FreeIPA architecture

Do not deploy Active Directory merely to avoid typing an SMB password once per client. Choose AD—Microsoft AD DS or a deliberately operated Samba AD domain—when the household actually wants domain-joined Windows sign-in, Kerberos SSO, centralized Windows computer identity, or Group Policy. Microsoft describes AD DS as the directory, authentication, access-control, and replication system, and Group Policy as centralized user and computer configuration. Those are valuable capabilities and a real operational commitment. Microsoft: AD DS overview and Microsoft: Group Policy overview

Web SSO is a separate decision. Logging into a web dashboard through an identity proxy does not align POSIX IDs, authenticate NFS, or give a Windows workstation a Kerberos ticket for SMB.

Stage 4 exit test: on every Linux host that will touch shared files, id returns the same numeric identity and group membership for each registered person and service. A deliberately unregistered or conflicting allocation is rejected. The registry has a tested restore copy.

Stage 5 — Separate data by access and recovery needs#

TL;DR: Separate datasets wherever owners, permissions, quotas, snapshots, backups, or application-consistency rules differ.

Work profile: operational size large; quota large; human effort medium; agent effort large; wait large; outage planned storage setup; authorized pool/datasets affected; no valuable migration yet; clock duration unknown until target-specific evidence exists.

Do not create one dataset called data and export its root to everyone. Dataset boundaries should follow different owners, snapshot schedules, backup policies, quotas, or application-consistency rules.

A small starting hierarchy might be:

tank/
├── household/
│   ├── shared/          SMB; all household users; Class A/B
│   ├── adults/          SMB; adults only; Class A
│   ├── kids/            SMB; child owners + adults; quota; Class A/B
│   └── homes/           SMB; one child dataset per person when useful
├── media/
│   ├── library/         SMB read-mostly + Plex read; Class D
│   ├── incoming/        isolated, quota-limited staging; Class E
│   └── transcode/       local only; strict quota; no snapshots; Class E
├── apps/
│   ├── plex/            service-owned state; Class B/C
│   ├── automation/      service-owned state; Class B/C
│   └── exports/         application-consistent backup exports; Class C
├── infrastructure/      desired state and recovery bundles; Class B
└── restore-staging/     quota-limited; never exported by default

On Linux, enable POSIX ACL support and efficient extended attributes on datasets that need Samba or shared group ACLs. Current OpenZFS guidance pairs acltype=posix with xattr=sa on Linux. Set these at dataset creation where possible because property changes do not rewrite old files. OpenZFS: ACLs and extended attributes

Use the minimum property customization needed:

• Compression on.
• POSIX ACLs and xattr=sa on Linux share datasets.
• Dataset-specific snapshot policy.
• Quotas for children, incoming downloads, transcodes, caches, restore staging, and any application capable of runaway growth.
• Encryption roots aligned to confidentiality and key-handling boundaries.
• No execution permission on staging or user-upload datasets unless an application specifically needs it.
• No deduplication, unsafe sync setting, or speculative cache device.

ZFS quotas constrain a dataset and optionally its descendants; without them, every dataset can consume the pool’s shared free space. OpenZFS: quotas and reservations Avoid reservations until an outcome truly requires guaranteed space; unused reservations can create confusing apparent scarcity elsewhere.

Create the ACL matrix before the ACLs. The first version can be simple:

“Adults” means full authority over household data, not infrastructure administration. A compromised adult laptop must still be unable to administer the server, rewrite configuration history, read agent credentials, destroy snapshots, or delete backups.

Stage 5 exit test: every path has one owning dataset, one data class, one snapshot/backup policy, and an explicit access row. No household account can traverse the pool root or another application’s private state. Quota tests fail safely before the pool’s reserve is threatened.

Stage 6 — Serve people with SMB#

TL;DR: Use authenticated SMB for household Windows and macOS access while keeping storage administration separate.

Work profile: operational size large; quota large; human effort medium; agent effort large; wait medium; outage planned file-service interruption; household share access affected; test data first; clock duration unknown until target-specific evidence exists.

Use SMB for Windows and macOS clients. It is the common protocol, supports authenticated per-user access, and avoids asking household computers to understand Linux numeric identities.

The simple authentication model#

TL;DR: Start with standalone Samba, one named identity per person, modern SMB, and no guest or SMB1 fallback.

Run Samba as a standalone member of a workgroup, with one Samba identity per person mapped to the corresponding local Unix identity. Use user-level authentication, no guest fallback, and modern SMB. Samba’s current default standalone password backend is tdbsam, and current protocol negotiation starts at SMB2; Microsoft strongly recommends leaving SMB1 disabled because of its security weaknesses. Samba smb.conf reference and Microsoft: detect and disable SMB1

This creates a separate network password, but each client can store it in its native credential store. On macOS, connect in Finder to a stable name such as smb://files.home.arpa/shared as a registered user and save the credential in Keychain. On Windows, map \\files.home.arpa\shared, select reconnect at sign-in, and store that user’s network credential. Apple and Microsoft document these normal client paths. Apple: connect a Mac to an SMB server and Microsoft: SMB file sharing and mapped drives

That is convenient enough for a small household. Domain-wide passwordless SSO is the later AD/Kerberos decision, not a prerequisite for useful file service.

Design shares around who needs access#

TL;DR: Expose stable purpose-based shares with minimum access rather than mirroring the pool's physical organization.

Begin with a few stable share names:

• shared: household collaboration.
• adults: private adult material.
• kids: a parent-visible namespace with per-child ownership and quotas.
• media: read-only for ordinary users; write access limited to the curation workflow.
• Individual home shares only if they solve a clear need.

Do not export the ZFS pool root, application-state datasets, backup repository, snapshot administration path, or download/transcode staging to ordinary clients. Do not give adults an “admin” share that contains server configuration or credentials.

Samba can store Windows ACL information in protected extended attributes with vfs_acl_xattr. Its documentation emphasizes that the mapping from Windows SIDs to Unix users and groups must remain consistent. Leave system-ACL mapping enabled when the same files must have meaningful local or NFS permissions. Samba: vfs_acl_xattr

The UID/GID registry alone is not enough to preserve that mapping. A standalone server also has a local machine SID, a passdb containing Samba identities, and possible Windows-group-to-Unix-group mappings. Before ACL-bearing shares enter normal use:

• record the NetBIOS name, workgroup, passdb backend, net getlocalsid result, and Samba version;
• discover the installation's actual private, state, and lock directories rather than assuming distribution-specific paths;
• make a protected, quiesced or Samba-supported backup of the active passdb, local-SID/secrets state, group mappings, and other persistent databases required by the selected modules;
• record net groupmap list and each exact principal-to-SID mapping in the human-only Samba recovery record; keep only its opaque reference and mapping source in the model-safe identity registry; and
• keep password databases and password hashes out of Git, chat, and ordinary agent-readable evidence. Store only their protected backup reference and recovery procedure.

On a clean rebuild, restore or deliberately set the original local SID and restore the passdb/group mappings before serving existing security.NTACL data. Then compare a known file's Windows ACL, POSIX ACL, and effective access from an allowed and denied client. If a later move to AD, FreeIPA-backed Samba, or another SID authority changes those SIDs, treat it as an ACL migration: inventory the old and new mappings, translate with supported Samba tooling, and test a copy before touching authoritative data. Unchanged UID/GID values do not by themselves make SID-bearing ACLs portable. Samba's net tool exposes local SID and group-mapping operations. Samba: net

Prefer server-owned ACL policy at the roots. Users may control their own ordinary files, but they may not redefine who administers a share, expose a private root, or grant an application account broader traversal.

Test file sharing from household computers#

TL;DR: Prove real allow/deny, reboot, ACL, and identity-recovery behavior with disposable data before household migration.

Stage 6 uses an empty disposable share or copied test fixtures and test identities. Do not migrate family data, map the share for ordinary household use, or make it authoritative yet. The point is to prove authentication, ACL behavior, reboot, and Samba identity recovery before Stage 9 establishes independent backup and Stage 10 performs the controlled migration and client cutover.

Every Samba change uses one sealed installer and one invocation. The following stages happen inside it; the owner does not receive them as separate commands:

1. Render a complete candidate from structured desired state.
2. Check duplicate sections, conflicting options, paths, users, groups, and dataset properties.
3. Run testparm against the staged file. Samba cautions that a successful testparm proves internal configuration correctness, not that the service will work. Samba: testparm
4. Review the bounded diff and digests; through the consolidated human packet, copy the complete candidate to a new root-owned agent-non-writable release path, verify its digests, rerun testparm there, and install only those sealed bytes atomically. A separately reviewed fixed action may replace this step later only if it enforces the same binding.
5. Coordinate tests from one Windows and one macOS client as an adult, child, and intentionally denied identity.
6. Reboot the server, resume automatically, and reconnect clients.
7. Print success only after service health, client access, denial, reboot, and evidence checks pass. On failure, freeze further mutation, preserve evidence and the last-known-good path, explain the prepared rollback and its risks, and always ask the owner whether to invoke it or hold for a separately reviewed fix-forward.

Expose read-only snapshots as “Previous Versions” only after ordinary snapshots and backups work. Samba explicitly says its shadow_copy2 feature is an end-user convenience and not a backup or archive. Samba: vfs_shadow_copy2

Stage 6 exit test: each representative client reconnects by stable DNS name after sign-in and after a server reboot; allowed create/read/rename/delete operations succeed; cross-user and cross-role attempts fail; SMB1 and guest access fail; no client credential grants server administration or backup deletion. A disposable recovery test restores the original local SID, passdb/group mappings, one representative SID-bearing ACL, and effective allowed/denied client access.

Stage 7 — Use NFS only for trusted Linux applications#

TL;DR: Add NFS only for trusted Linux consumers needing native numeric ownership; keep ordinary household access on SMB.

Work profile: operational size large; quota large; human effort medium; agent effort large; wait medium; outage planned application/storage interruption; trusted Linux consumers only; clock duration unknown until target-specific evidence exists.

NFS complements SMB; it is not the “advanced version” of SMB. Use it when trusted Linux hosts or applications need native ownership, modes, ACLs, and server-to-server filesystem access. Keep human Windows/macOS file access on SMB.

Preserve root squashing#

TL;DR: Align service UID/GID values across trusted clients and retain root squashing unless an exceptional design proves otherwise.

With ordinary AUTH_SYS, an NFS client tells the server the UID and GID making a request. Direct mapping requires those numbers to mean the same principal on every system. The server normally maps remote UID 0 to an anonymous account: root squashing. Both the Linux exports(5) manual and Red Hat’s NFS guide describe that as the default. Linux exports(5) and Red Hat: deploying and securing NFS

Do not “fix” a container permission error with no_root_squash, chmod 777, or a shared UID. Fix the identity and path:

1. Give the application a registered service UID/GID.
2. Provision that identity on the NFS server and client host.
3. Run the container process as that identity, accounting for any rootless user-namespace mapping.
4. Give the service ACL access only to its dataset.
5. Export only that dataset to the named client address or VPN identity.
6. Keep root_squash; use read-only export where writing is unnecessary.
7. Test creation and ownership from inside the actual container. Prove that remote UID 0 maps to the anonymous identity and is denied, then explicitly demonstrate and record that root on an AUTH_SYS client can impersonate the permitted nonzero service UID.

Do not export NFS to the public internet. On a flat LAN, restrict exports to exact trusted hosts, bind/firewall NFS to the LAN and management overlay, and regard any general-purpose client as capable of impersonating AUTH_SYS identities if compromised.

When that trust model becomes unacceptable, the upgrade is NFSv4 with Kerberos—not disabling protections. With RPCSEC_GSS, the server cryptographically authenticates the user instead of trusting the client’s claimed UID; krb5i also protects integrity and krb5p adds traffic encryption. That is a strong trigger for FreeIPA and its Kerberos service lifecycle. Red Hat: NFS AUTH_GSS security

Do not edit the same files through SMB and NFS#

TL;DR: Do not make the same writable tree serve SMB and NFS unless identity, ACL, locking, and application semantics are proven compatible.

Do not export the same writable tree through SMB and NFS by default. The protocols can differ in identity mapping, ACL semantics, case behavior, locking, and application expectations. Prefer separate datasets:

• SMB datasets for people.
• NFS datasets for trusted Linux services.
• An explicit promotion, import, or application workflow between them when needed.

If a real workflow requires both protocols on one tree, central identity and a cross-protocol ACL/locking test matrix become prerequisites. The exception must be documented; “it mounted” is not an acceptance test.

NFS mounts are also dependencies. A container starting before its host mount is ready is an ordering/readiness problem, not evidence that permissions or NFS should be redesigned. The host must declare the remote mount dependency, fail closed when it is absent, and start the dependent service only after a read/write probe succeeds. Do not hide a race with retry loops and broader privilege.

Stage 7 exit test: the intended service can read/write its dataset and files appear with the registered numeric owner; unrelated ordinary service identities are denied; remote UID 0 is squashed and denied; the documented test proves client root can impersonate the permitted service UID under AUTH_SYS, so the client is explicitly trusted or the design stops for NFSv4/Kerberos; reboot ordering works; and taking the NFS server away prevents the dependent application from writing into an empty local mountpoint.

Stage 8 — Decide which remote files stay writable#

TL;DR: Treat continuous remote availability as a separate design from backup and name the authoritative writable copy.

Work profile: operational size large; quota XL; human effort large; agent effort XL; wait large; outage none during design; future network/storage/privacy impact; clock duration unknown until target-specific evidence exists.

Remote availability and backup are separate purchases.

Files used directly by people#

TL;DR: Prefer a mature sync service for user documents when its privacy, account-recovery, and conflict tradeoffs are acceptable.

A commercial file-sync service is usually the lowest-maintenance answer for documents that must be continuously available on laptops and phones. It brings mature clients, offline caching, sharing, and provider operations. The tradeoffs are subscription cost, provider access and policy, account-recovery dependence, metadata disclosure, and the possibility that deletion or ransomware damage synchronizes.

Use the homelab SMB service through the VPN when local control matters more and WAN latency is acceptable. Never expose TCP 445 publicly. For large or frequently used remote data, measure the residential uplink and recovery time; the existence of a domain name does not create bandwidth.

If applications need the data#

TL;DR: Avoid generic WAN filesystems for live application state unless the application explicitly supports and survives their failure semantics.

Do not pretend a WAN-mounted filesystem is local storage. Ask:

• What happens to reads and writes when the WAN disappears?
• Does the application require POSIX locking, rename atomicity, or low-latency fsync?
• Which side is authoritative after concurrent or offline changes?
• Can a cache serve stale data safely, and how is it invalidated?
• How are partial writes, retries, egress cost, and credential rotation handled?
• Can the application instead run near the data?

The default choices are, in order:

1. Keep the application and its live database on local storage; back it up remotely.
2. If remote clients need results, expose an authenticated application API rather than its filesystem.
3. For applications designed for it, use an object-store API with explicit consistency and caching behavior.
4. Use a WAN filesystem only when the application explicitly supports its failure semantics and the design has been tested under packet loss, latency, and complete disconnection.

Never put a live database inside a consumer sync folder, on a generic SMB/NFS WAN mount, or behind two competing bidirectional sync engines.

Record the source of truth#

TL;DR: Declare the authoritative writer, replica/cache roles, conflict policy, and offline-write behavior for every remotely available dataset.

For every remotely available dataset, write one sentence:

The authoritative writable copy is ___; ___ is a read cache/replica/sync target; conflicts are resolved by ___; offline writes are allowed/not allowed.

If that sentence cannot be completed, the system is not ready for writes.

Stage 8 exit test: disconnect the WAN during a representative read and write. The application or user experience fails or degrades exactly as documented, then reconciles without silent overwrite when connectivity returns.

Stage 9 — Prove backup and restore before migration#

TL;DR: Establish independent, deletion-resistant backup and prove usable restoration before valuable data moves.

Work profile: operational size XL; quota XL; human effort large; agent effort XL; wait XL; outage possible backup/restore test load; storage and network load; no source deletion; clock duration unknown until target-specific evidence exists.

A practical first backup design#

TL;DR: Combine bounded local snapshots, application-consistent exports, encrypted off-host history, separated retention authority, and scheduled restore tests.

For Classes A–C:

1. Take local ZFS snapshots for rapid rollback: hourly for 48 hours, daily for 30 days, and weekly for 12 weeks is a reasonable starting policy.
2. Produce application-consistent exports for databases and other state that a raw file snapshot may not restore safely.
3. Send a daily encrypted restic backup to an independent destination.
4. Give the routine backup credential append-only authority; keep delete/prune and retention control elsewhere.
5. Keep at least 30 days of off-site history and monthly recovery points for a year, then adjust from measured growth and the owner’s actual RPO.
6. Give each valuable class a maximum restore-test interval—quarterly is a reasonable first value for irreplaceable data—and also test after every material storage, identity, encryption, or backup-tool change. Automate nondestructive staged restores where practical; do not notify a human for routine success.

Before valuable data moves, record every recurring promise in the recurring protection record. Include every job required by policy and every policy-protected path; do not silently omit one. The canonical record carries the job owner, installed mechanism, schedule, model-safe protected-source references, limits, retention, evidence, alerts, and disable procedure. A sentence saying “daily” or “scheduled” is a plan, not proof. The record links evidence; it does not prove who installed anything.

The agent pre-fills observable installation, enabled-state, schedule, selection, and evidence fields. The owner decides what policy must protect and supplies retention decisions and protected references. This division keeps the record useful without recreating the worksheet as human paperwork.

As later target proof, use a controlled schedule invocation or observe the next run; do not wait a month merely to prove configuration. Also run unattended with the agent disconnected. Read evidence from the documented status path. Then cause a safe synthetic failure, such as an unreadable disposable fixture or a test-only alert input, and confirm that the job exits as failed, creates no success marker, preserves diagnostics, and reaches the recorded alert recipient and route. Do not damage a pool or unique data to test an alert. These runtime checks do not run as part of content validation.

For Class D, back up the catalog, playlists, custom metadata, and acquisition record even when the media payload is intentionally omitted. For Class E, back up nothing.

CISA recommends offline encrypted backups and regular availability and integrity tests because ransomware commonly seeks accessible backups to delete or encrypt. CISA #StopRansomware guide A permanently mounted writable “backup” under the same adult or agent credentials does not meet that threat.

Why this guide uses restic for file backups#

TL;DR: Use restic for portable encrypted file-level recovery while separating routine append authority from deletion and retention authority.

Restic provides encrypted, deduplicated file-level snapshots across local, SFTP, REST, object-storage, and other backends. It supports restoring into an empty target and checking repository structure or reading all stored data. Its own quickstart makes the essential loop explicit: back up, list snapshots, restore to a separate path, and check the repository. Restic introduction and restore/check workflow

Run scheduled backup as a dedicated system service, not inside an LLM session. A human provisions the repository password and backend credential directly into a root-owned service credential store. The local runner is root-owned and limited by its unit and configuration to the named sources. It must be able to read the ownership, POSIX ACLs, required extended attributes—including protected Samba ACL metadata—and protected identity/Samba recovery state selected by policy; its remote repository credential remains append-only. If any selected path or required metadata is unreadable, the run fails and emits no success marker. Restic documents that it saves and restores extended attributes such as ACLs, but the actual source privilege and a clean restore determine whether protected attributes were captured. Restic: backup metadata and Restic: restoring extended attributes

The agent reads an exported status and evidence reference but cannot retrieve the service credentials, change its sources, or start arbitrary commands. On day one, a human starts an exceptional on-demand run. Later, an independently reviewed no-argument fixed action may start the one named job if it enforces concurrency and cadence limits, source/destination identity, quota/cost bounds, audit, and a protected human grant. Keep a recovery copy of the repository password in the owner’s secret store—restic warns that losing it makes the repository unrecoverable. Restic: preparing a repository

Use restic’s REST server in append-only mode or a backend with equivalent non-delete credentials. Restic’s retention documentation is unusually clear about the boundary: forget and prune require full read/write/delete access, so maintenance must run from a separate, well-secured client; for append-only repositories it recommends time-window retention such as --keep-within to resist maliciously inserted snapshots. Restic: removal, pruning, and append-only security

Repository checks are not restores. Schedule metadata checks frequently and rotate full-data reads over time, but still restore files and application exports to empty staging. A correct checksum is not proof that the owner has the password, understands the restore command, preserved numeric identities, or can start the application.

Use ZFS replication only as an additional copy#

TL;DR: Add ZFS replication for faster compatible recovery, but retain independent file-level backup and protected history.

When both ends use ZFS, snapshot send/receive can efficiently preserve datasets, properties, and incremental changes. Raw send keeps encrypted dataset contents encrypted so a receiving site need not hold the decryption key. OpenZFS: send and receive

Use it when fast full-dataset recovery justifies a second ZFS system. Keep a file-level backup too for portability and ordinary restores. A writable ZFS replica whose snapshots the source can destroy is still not independent history. Receive replicated datasets unmounted at the backup destination and separate receive authority from snapshot-destruction authority.

Reciprocal peer backup, safely#

TL;DR: Isolate each household's encrypted repository, credentials, quota, retention authority, and failure domain before calling reciprocal storage a backup.

Two households can exchange encrypted backup capacity without giving either household administrative access to the other. The safe shape is:

Site A source service
    └── encrypted append-only repository A, stored at Site B
         ├── fixed quota
         ├── reachable only through VPN/approved backup endpoint
         ├── unreadable without a Site A restic password/key
         └── pruned only by Site A's separate retention client

Site B source service
    └── a completely separate repository B, stored at Site A

Required boundaries:

• Separate Unix accounts, datasets, quotas, credentials, repository keys, and logs in each direction.
• The peer cannot decrypt or browse the other household’s backups, but the peer's storage administrator can still delete or withhold the ciphertext.
• The source backup writer cannot delete, prune, change retention, escape its repository, or administer the peer host.
• The routine Site A writer necessarily holds a protected restic password/key and can decrypt/read repository contents while it runs; otherwise restic cannot open the repository. Its separate backend credential is append-only. Neither credential is available to ordinary agent sessions.
• Site A's separate retention client holds its own restic password/key plus a different backend credential with delete/prune authority. Site B holds no restic password/key and does not operate the retention client.
• Multiple restic passwords protect secret distribution but all unlock the same repository master key. The useful separation here is backend verbs and process isolation: compromise of the routine writer must not confer delete/prune authority. It is not cryptographic separation between the writer and retention client.
• The backend exposes read/write/delete only to that retention client's authenticated maintenance path; the routine writer's path remains append-only. If the selected REST service cannot enforce different client modes, use a separate maintenance listener or a human-controlled maintenance window rather than weakening the writer endpoint.
• The destination’s ordinary file-sharing and media-transfer accounts cannot access backup storage.
• Retention continues when the source writer alone is compromised or silent; compromise of the separate retention client is a different failure domain.
• Bandwidth is scheduled and capped so backups cannot starve interactive traffic or media streaming.
• The first large seed and a full recovery estimate are planned against the real uplink.
• Either party can end the arrangement with an agreed export/destruction procedure and no dependency on the other’s goodwill to recover its own key.

The destination enforces quota and availability; it does not enforce restic snapshot retention without receiving the decryption key and full repository credentials. Giving Site B a restic pruning service would therefore contradict the promise that Site B cannot read Site A's backup. Conversely, keeping the key from Site B cannot stop a malicious or compromised Site B administrator from erasing the stored ciphertext. Use another independently controlled backup or storage-layer object lock when protection from peer deletion is required.

Reciprocity does not create independence if both repositories are managed by the same credentials, both sites are within one likely disaster area, or each source writer can erase its remote history. Until append-only behavior, retention separation, restore, quota, peer-loss, and disconnection behavior are tested, call this a peer replica experiment—not a backup.

For a simpler first off-site target, use commercial object storage with versioning/immutability and a narrowly scoped application key. Object-lock systems can prevent deletion or overwrite until a retention date, but a mistaken retention period can also lock in cost. For example, S3 Object Lock uses a WORM model and protects particular object versions. Amazon S3 Object Lock Treat provider-account recovery, billing failure, region choice, egress cost, and retention-lock administration as parts of the backup design.

The restore ladder#

TL;DR: Prove recovery progressively from individual files and ACLs through applications, datasets, host loss, and credential loss.

Test recovery in increasing scope:

1. File: restore three representative files of different sizes and metadata; open or parse them and compare a recorded checksum where meaningful.
2. ACL: restore a directory tree into staging after the identity registry is loaded; verify adult, child, service, and denied access.
3. Application: restore the application export and configuration into a disposable instance, then perform an outcome-level test.
4. Dataset: restore enough data to prove capacity, paths, ownership, and performance on a clean target.
5. Host loss: from the recovery bundle and backup alone, rebuild a disposable host or VM far enough to import or restore one representative service.
6. Credential loss: prove the owner—not the routine agent—can retrieve the backup key and revocation/replacement procedure.

The restore target is always a new, quota-limited staging path. Never “test” by overwriting live data.

Stage 9 exit test: an off-site recovery point exists; the source and agent cannot delete its protected history; repository check succeeds; a clean staged restore from that off-site copy passes the file, ACL, and one application test; and the elapsed time is compatible with the recorded RTO.

Stage 10 — Migrate while keeping the original copy intact#

TL;DR: Copy rather than move, validate incrementally, switch consumers gradually, and preserve the unchanged source until the owner closes rollback.

Work profile: operational size XL; quota XL; human effort large; agent effort XL; wait XL; outage planned file/application cutover; unique-data, storage, network, and household access affected; clock duration unknown until target-specific evidence exists.

Migration is a controlled copy, not a move.

1. Freeze pool and identity desired state for the migration window.
2. Confirm the independent backup and restore evidence are current.
3. Mount or attach the source read-only where practical.
4. Perform an initial copy while the old service remains authoritative.
5. Preserve numeric ownership, ACLs, extended attributes, timestamps, hard links, and sparse-file behavior where the data class requires them.
6. Compare counts, sizes, errors, and hashes appropriate to the dataset. Do not rely only on a zero exit code.
7. Quiesce writers, make the application-consistent export, and perform the final incremental copy without deletion flags.
8. Switch one representative client or application by stable DNS name.
9. Run the full allow/deny, reboot, snapshot, backup, and restore tests.
10. Switch remaining clients and begin a defined soak period.
11. Keep the source unchanged and disconnected or read-only until the owner closes rollback.
12. Erase or repurpose the source only under a new, exact device authorization.

Never allow an agent to add --delete, reformat the source, or prune the pre-migration backup merely to make the old and new trees “match.” Extra old data during a migration is usually a capacity problem; deleted only copies are a recovery problem.

Stage 10 exit test: migrated data and access controls match the inventory; representative applications pass; backup and restore run from the new paths; and rollback remains possible throughout the recorded soak period.

What the agent may do to storage#

TL;DR: Let the agent observe and prepare bounded changes while hosts and backup systems enforce that it is not the storage administrator.

The LLM is an operator of bounded capabilities, not the storage administrator. Prompts and rules express intent; the host and backup destination enforce the boundary.

Standing identities#

TL;DR: Give each persistent identity only the exact observation, backup-write, or service capability it needs.

On day one, the agent prepares named actions such as “snapshot the household datasets,” “run backup policy A,” or “scrub pool GUID X” as one small human-run script or equivalent transaction with one invocation, preconditions, exact targets, syntax/fixture/native validation, outcome tests, a prepared rollback, and safe-stop behavior. It never receives arbitrary systemctl, shell, block-device, ZFS, backup-tool, or sudo authority. If a future fixed action replaces a repeated human step, that program—not agent prose—must enumerate and validate every parameter rather than interpolate it into a shell command. The exact sealed invocation may perform only its already-authorized transaction-local failure cleanup. Any different rollback or backtrack requires a fresh owner choice.

Actions that always need fresh human approval#

TL;DR: Require fresh reconciliation, explicit owner approval, and independent review for erasure, topology, retention, restore, and identity hazards.

These require a fresh plan, live device reconciliation, explicit owner approval, and—where data loss is possible—an independent review:

• Partition-table writes; wipefs; filesystem or pool creation.
• zpool create, destroy, add, remove, detach, labelclear, or forced import.
• Changing vdev geometry, replacing a disk, reducing a volume, or clearing an error without explaining it.
• ZFS rollback or recursive snapshot destruction.
• Samba or NFS changes that broaden clients, enable guest access, weaken protocol security, or disable root squashing.
• Restic forget, prune, repository repair, retention changes, bucket deletion, or object-lock bypass.
• Synchronization with deletion propagation.
• Restoring over a live dataset.
• Changing numeric ownership recursively.

A destructive plan names the target by serial/WWN and pool GUID, states existing signatures and contents, records the last successful backup and restore, provides the exact expected postcondition, and expires unused. If live state differs by one device, mount, process, or identifier, the human stops; any future fixed action must refuse automatically.

Automatic operations still have limits#

TL;DR: Bound snapshots, backups, scrubs, alerts, and remediation so automation cannot exhaust capacity or conceal repeated failure.

• Snapshot creation has an installed root-owned job, a named policy owner, a tested schedule, and a rate limit; uncontrolled snapshots can fill a pool.
• Scrubs have an installed root-owned job and are serialized with resilvers and other heavy maintenance.
• Backup retries have a tested budget; repeated failure creates one actionable incident rather than an endless loop.
• Restores always allocate a new staging directory and enforce a quota.
• Capacity alarms are stateful: warning at 70%, action plan at 80%, incident at 90%, with one resolution when cleared.
• A degraded pool's installed alert path triggers evidence collection and an owner action packet. It does not trigger automatic disk replacement.
• SMART attributes are retained as evidence, but alerts focus on a device-health conclusion and required action rather than dumping every vendor-specific counter.

Render and validate configuration before replacing it#

TL;DR: Render and validate complete desired state, seal reviewed bytes, install atomically, and verify from the consumer side.

Identity, Samba, NFS, dataset, snapshot, and backup policy begin as structured desired state. The agent writes a new staged artifact, checks duplicate identities and ranges, validates paths and referenced principals, runs the native validator, and renders a bounded diff plus digests. The human packet copies the complete candidate closure to a new root-owned agent-non-writable release path, verifies the digests, reruns validation against those sealed bytes, and asks the higher-trust installer to replace the live configuration atomically from that path.

This is especially important for /etc/exports: Red Hat documents that a single misplaced space can change a host-specific writable export into a much broader export. Red Hat: NFS export security and syntax Do not let an LLM repeatedly patch it with regular expressions.

After context compaction or a new model session, storage mutation remains disabled until the agent reloads the decision ledger, inventory, identity registry, active maintenance window, change journal, and last restore evidence; then reconciles them against live pool GUIDs, disk IDs, mounts, jobs, and health.

The storage record another session needs#

TL;DR: Preserve the minimal versioned inventory, decisions, identities, policies, evidence, and runbooks needed to rebuild without chat memory.

At chapter completion, the system of record contains these small, versioned artifacts:

Detailed command output belongs in evidence attachments, not in the decision record. The bootstrap packet must remain small enough for a new agent session to read before acting.

Final recovery test#

TL;DR: Declare storage complete only after cold boot, client permissions, failure handling, restore, migration rollback, and agent-denial tests succeed.

Work profile: operational size XL; quota large; human effort large; agent effort large; wait large; outage planned storage/file-service interruption; attended household disruption; rollback remains open; clock duration unknown until target-specific evidence exists.

Do not declare storage complete because a share appears in Finder or Explorer. Run this drill:

1. Reboot the storage host with no interactive LLM session running.
2. Confirm intended encrypted datasets unlock—or deliberately require the documented human step—and services wait for their storage dependencies.
3. Confirm pool topology and health by GUID and durable disk IDs.
4. From Windows and macOS, test adult, child, and denied access by stable DNS name.
5. If NFS has been enabled for a real application host, create a file as the registered service account and verify numeric ownership on both ends; prove UID 0 itself is squashed and denied; then demonstrate that client root can impersonate the permitted nonzero service UID under AUTH_SYS and record the client as trusted—or stop and require NFSv4/Kerberos. Otherwise defer this test to that Phase 5 integration.
6. Delete a disposable client file, recover it from a snapshot, then recover a different copy from the off-site backup into staging.
7. After the first container consumes a network mount in Phase 5, stop that mount and prove the container does not silently write into the underlying empty directory. Record this as pending—not failed—during the minimum Phase 4 drill.
8. Present the routine agent with a request to destroy a snapshot, format an unapproved spare, disable root squashing, prune the repository, and restore over live data. Every attempt must fail at the infrastructure boundary.
9. Disconnect the agent workstation. Observe the next installed snapshot and backup jobs through their recorded evidence paths, and confirm their failure alerts were tested. The data service must not depend on an LLM being awake.
10. Record elapsed restore time, deviations, and the exact human actions required. Fix the design if the result violates an RPO, RTO, or lights-out assumption.

When these tests pass, storage is no longer a pet computer with some shared folders. It is a recoverable service whose data, identities, authority, and history can move to the next machine without depending on the old machine—or the agent’s memory—remaining intact.