Skip to main content

Airgap Design

This is the most important Day 0 page. There is no homelab equivalent — airgap operation is the defining constraint of the entire enclave. Every architectural decision flows from it.

What the Airgap Boundary Is​

The airgap boundary is a physical and logical line that nuc-00 straddles. Before the boundary is crossed:

  • nuc-00 has internet access
  • Hauler runs on nuc-00 and pulls every required artifact from upstream registries and package servers
  • The collected artifacts are saved to a compressed tarball

After the boundary is crossed:

  • nuc-00 has no internet access
  • There is no route to upstream registries from any node in the enclave
  • The only software that exists is what was collected by Hauler before crossing
caution

Once you cross the airgap, nothing comes in that you didn't bring.

There is no "I'll just pull that image real quick." There is no "let me grab the latest chart." If an artifact wasn't in the Hauler manifest before sync, it doesn't exist on the other side of the boundary. Plan accordingly.

The Hauler Lifecycle​

Hauler operates in two distinct phases separated by the physical act of crossing the boundary.

┌─────────────────────────────────────────────────────────────────────┐
│  INTERNET SIDE (nuc-00, before airgap crossing)                     │
│                                                                     │
│  1. hauler store sync   — pull all artifacts per manifest           │
│       Images from: registry.rancher.com, docker.io, ghcr.io, etc.  │
│       Charts from: upstream Helm repos                              │
│       Files: RKE2 tarballs, Harvester ISO, step binaries            │
│                                                                     │
│  2. hauler store save   — write store to compressed tarball         │
│       → /srv/www/htdocs/hauler/carbide-enclave-hauler.tar.zst       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
                              │
                              │  ◄──  SNEAKERNET / USB / DATA DIODE
                              │
┌─────────────────────────────────────────────────────────────────────┐
│  AIRGAP SIDE (nuc-00, after airgap crossing)                        │
│                                                                     │
│  3. hauler store load   — extract tarball back into Hauler store    │
│                                                                     │
│  4. hauler store serve  — run ephemeral OCI registry on :5000       │
│       Used during bootstrap before Harbor exists                    │
│                                                                     │
│  5. hauler store push   — push all images/charts to Harbor          │
│       Harbor becomes the authoritative permanent registry           │
│       Hauler registry stops after this step                         │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

What Hauler Collects​

Hauler's manifest (generated at runtime by scripts/hauler.sh) enumerates every artifact the platform needs. Categories:

CategoryExamples
Container imagesRKE2 components, Rancher, Harbor, Keycloak, cert-manager, GPU Operator, vLLM
Helm chartsRancher, cert-manager, Harbor, Keycloak, GPU Operator
Binary filesRKE2 install tarballs (amd64 + arm64), Harvester ISO, step CLI binary

All versions are pinned — no latest tags anywhere. Version numbers live in scripts/env.d/carbide-enclave.sh and are interpolated into the Hauler manifest at generation time.

Two-Phase Registry Strategy​

This is one of the most important design decisions in the enclave. There are two registries, and each has a distinct role:

Phase 1: Hauler Ephemeral Registry (Bootstrap)​

After loading the Hauler tarball on the airgap side, you run:

hauler store serve registry  # starts on :5000

This is a temporary OCI registry used only during bootstrap — specifically, during the RKE2 cluster bring-up before Harbor exists. RKE2 is configured to pull images from 10.0.0.10:5000 during this window.

Phase 2: Harbor (Permanent)​

Once Harbor is installed and running, you run:

hauler store push --registry harbor.carbide-enclave.kubernerdes.com

This pushes everything from the Hauler store into Harbor. After this step:

  • Harbor is the authoritative registry for all cluster workloads
  • Rancher's system-default-registry is set to Harbor
  • The Hauler registry on :5000 is stopped and stays stopped
caution

The Hauler registry is temporary infrastructure, not a production service. It exists only to bootstrap Harbor. Once hauler store push completes and Harbor is healthy, shut down the Hauler registry. Leaving it running creates confusion about which registry is authoritative — and it has no access control.

The Five Hard Airgap Rules​

These rules have no exceptions in a production enclave. "Just this once" is how supply chains get compromised.

Rule 1: Every image, chart, and binary must flow through Hauler.

No helm repo add pointing at the internet. No docker pull from docker.io. No downloading binaries on a running node. If it isn't in the Hauler store, it doesn't exist in the enclave.

Rule 2: Hauler manifests are the source of truth for what enters the enclave.

When adding a new component, add its Hauler manifest entry first, then write the install procedure. The manifest is the bill of materials. If you install something without a manifest entry, you have an untracked dependency that will break on the next rebuild.

Rule 3: arm64 parity is required for any artifact touching the DGX Spark.

Add linux/arm64 platform entries alongside linux/amd64 in every image reference that runs on or is managed by the DGX Spark. This includes GPU Operator, vLLM, Ollama, ExMachina, and the RKE2 agent tarball itself.

Rule 4: TLS everywhere, internal CA only.

step-ca is the root. cert-manager issues all certificates. --tls-skip-verify and similar flags are acceptable only during the initial CA bootstrap window — never in a final config committed to the repo. No ad-hoc self-signed certificates.

Rule 5: No hardcoded secrets.

Credentials live in ~/.config/RGS/creds on each operator machine. This file is never committed — it exists outside any git working tree. Non-secret environment variables live in scripts/env.d/carbide-enclave.sh (committed). Never commit a real credential value.

Pre-Airgap Checklist​

Everything on this list must be done before nuc-00 loses internet access:

  • ~/.config/RGS/creds is populated on nuc-00 (all credentials present and correct)
  • Hauler manifest has been reviewed — all required components and versions are listed
  • bash scripts/hauler.sh sync completed successfully — no errors, all artifacts pulled
  • Hauler store verified: hauler store info shows expected image count and chart count
  • bash scripts/hauler.sh save completed — tarball written to /srv/www/htdocs/hauler/carbide-enclave-hauler.tar.zst
  • Tarball checksum recorded (SHA256) for verification on the other side
  • Transfer medium (USB drive, NAS share, etc.) has enough capacity for the tarball
  • Harvester ISO is in the Hauler store or otherwise available for iPXE boot
  • arm64 artifacts are confirmed present for all DGX-adjacent components
tip

Run hauler store info after sync to get a count of images and their total uncompressed size. The compressed tarball will be significantly smaller, but the uncompressed size tells you how much space the Hauler store needs on the airgap side.

After Crossing the Boundary​

Once the tarball is on the airgap side:

# On nuc-00, airgap side
hauler store load carbide-enclave-hauler.tar.zst

# Verify the store
hauler store info

# Start the bootstrap registry (temporary — Phase 1)
hauler store serve registry

From here, Day 1 proceeds. The bootstrap registry stays up through RKE2 cluster formation and Harbor install. Once Harbor is running and hauler store push completes, stop the registry and proceed with Harbor as the permanent artifact source.