Skip to main content

Updates

Updates in an airgapped environment follow the same path as the initial install: Hauler collects on the internet side, you transfer the tarball across the airgap boundary, and Harbor distributes to the cluster. There is no shortcut — and that is by design. The airgap boundary is the control plane for what software is allowed to run.

Version Bump Procedure​

All version pins live in a single file: scripts/env.d/carbide-enclave.sh. Every script sources this file. Bump the version there and every script automatically picks it up on the next run — you never need to hunt down hardcoded version strings.

Step 1 — Edit the version​

vi /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/env.d/carbide-enclave.sh
# Bump: RKE2_VERSION, RANCHER_VERSION, HARBOR_VERSION, etc.

Commit the change to git so the version history is auditable.

Step 2 — Re-sync with Hauler (internet side)​

This step requires internet access. Run it on nuc-00 before crossing the airgap boundary, or on any internet-connected machine that has the repo and Hauler installed.

bash /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/hauler.sh sync

Hauler reads the version variables, generates fresh manifests under /srv/www/htdocs/hauler/manifests/, and pulls all new artifacts into /var/lib/hauler/. This may take a significant amount of time depending on how many images changed.

Step 3 — Save to tarball​

bash /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/hauler.sh save
# Output: /srv/www/htdocs/hauler/carbide-enclave-<timestamp>.tar.zst

Step 4 — Transfer and load (airgap side)​

Copy the tarball across the airgap boundary via USB drive, removable media, or whatever physical transfer method the environment permits. Then load it on the airgap side:

sudo bash /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/hauler.sh load \
  /path/to/carbide-enclave-<timestamp>.tar.zst

Step 5 — Push updated images to Harbor​

bash /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/hauler.sh push

This pushes all updated images from the Hauler store into Harbor. Existing images with the same tag are overwritten. Harbor's garbage collection can be run afterward to reclaim space from old layers.

Step 6 — Rolling restart or Helm upgrade​

Once Harbor has the new images, upgrade the affected component:

# Example: RKE2 node upgrade (drain first — one node at a time)
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig drain <node> \
  --ignore-daemonsets --delete-emptydir-data
# ... upgrade node via 40_bootstrap-rke2.sh, then uncordon
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig uncordon <node>

# Example: Helm chart upgrade (Rancher)
helm upgrade rancher rancher-stable/rancher \
  --namespace cattle-system \
  --reuse-values \
  --set rancherImageTag=<new-version>
caution

Do not upgrade multiple components at the same time, and do not proceed to the next component until the current one is verified healthy. Run verify-enclave.sh between each upgrade. A failed Rancher upgrade while Harbor is also mid-upgrade produces a state that is very difficult to recover from.

Component-Specific Notes​

RKE2​

RKE2 nodes must be upgraded one at a time with a drain/uncordon cycle. The control plane nodes (rancher-01, rancher-02, rancher-03) must be upgraded before agent nodes (DGX Spark). The etcd cluster tolerates one node being down at a time — do not drain two control plane nodes simultaneously.

# Check cluster health before draining
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig get nodes
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig get pods -A --field-selector=status.phase!=Running

Rancher Manager​

After a Rancher Helm upgrade, wait for all cattle-system pods to be Running and for Rancher to report a clean ping before touching any downstream clusters or auth config.

kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  rollout status deployment/rancher -n cattle-system

Harbor​

Harbor stores all registry data on the NAS via NFS. Before a Harbor upgrade, confirm the NFS mount is healthy and that no Hauler push or image pull is in progress. Harbor's built-in health check endpoint:

curl -sk https://harbor.carbide-enclave.kubernerdes.com/api/v2.0/health | jq .status
# Expected: "healthy"