Skip to main content

Smoke Test — verify-enclave.sh

scripts/verify-enclave.sh runs a full-stack smoke test from nuc-00, checking each layer of the deployment in bootstrap order. Run it any time to confirm the environment is healthy or to see which components still need to be deployed.

Usage​

# Full check (run from nuc-00 as mansible)
bash /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/verify-enclave.sh

# Skip DGX Spark checks (if not yet joined)
bash scripts/verify-enclave.sh --skip-dgx

# Same flag via env var
SKIP_DGX=1 bash scripts/verify-enclave.sh

The script prints PASS, FAIL, or SKIP for each check and exits non-zero if any check fails. Components not yet deployed are reported as SKIP — safe to run at any point in the deployment sequence.

What it checks​

SectionChecks
nuc-00 servicessystemctl is-active for chronyd, named, dhcpd, apache2, tftp.socket, step-ca; Apache HTTP response
step-caRoot CA cert present + expiry date; ACME directory endpoint reachable
DNS9 FQDNs resolve to expected IPs: ca, hauler, rancher-01/02/03, rke2, rancher, harbor, keycloak
HaulerOCI registry :5000; file server :8080
HarvesterAPI reachable; all nodes Ready; step-ca in additional-ca; ≥3 LoadBalancers configured
RKE2 clusterAPI reachable; all nodes Ready; no pods outside Running/Completed
cert-managerPods Running; ClusterIssuers Ready
Harbor/api/v2.0/systeminfo responds (200 or 401 both count)
Keycloak/health/ready returns 200
Rancher/ping returns 200
DGX SparkPingable; node Ready in RKE2; nvidia.com/gpu allocatable

Sample output​

[enclave] carbide-enclave smoke test
[enclave] env:    carbide-enclave
[enclave] domain: carbide-enclave.kubernerdes.com
[enclave] date:   2026-06-16T12:00:00Z

── nuc-00 local services ──
  PASS  chronyd
  PASS  named
  PASS  dhcpd
  PASS  apache2
  PASS  tftp.socket
  PASS  step-ca
  PASS  apache2 serving http://localhost/

── step-ca ──
  PASS  root CA cert present  (expires: Jun 16 12:00:00 2036 GMT)
  PASS  ACME directory reachable  (https://ca.carbide-enclave.kubernerdes.com:8443/acme/acme/directory)

── DNS resolution ──
  PASS  ca.carbide-enclave.kubernerdes.com → 10.0.0.10
  PASS  harbor.carbide-enclave.kubernerdes.com → 10.0.0.99
  ...

── Harvester cluster ──
  PASS  Harvester API reachable
  PASS  all 3 Harvester node(s) Ready
  PASS  step-ca in Harvester additional-ca  (subject=CN=Carbide Enclave Root CA)
  PASS  3 Harvester LoadBalancer(s) configured

── RKE2 management cluster ──
  SKIP  no kubeconfig at ~/.kube/carbide-enclave-rke2.kubeconfig

...

════════════════════════════════════════════════
  PASS 18   FAIL 0   SKIP 9
════════════════════════════════════════════════
[enclave] verification PASSED

HTTPS trust​

The script uses the step-ca root cert (/etc/step-ca/certs/root_ca.crt) for all HTTPS checks if it is present. Before step-ca is bootstrapped, it falls back to -k (skip verify) so the script remains runnable at any stage of deployment.

Exit code​

Exit codeMeaning
0All checks passed (SKIPs do not count as failures)
1One or more checks FAILED