Monitoring

Smoke Testing with verify-enclave.sh

Start here — after any change, after any maintenance window, and when diagnosing an issue. verify-enclave.sh runs all health checks in bootstrap order, so the first FAIL you see is the layer that broke. Everything below it in the output may also be failing as a consequence, but the root cause is at the first failure.

Quick run

bash /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/verify-enclave.sh

Run this from nuc-00 as mansible (or any operator account with kubectl access and the kubeconfig files in place). The script does not require root.

Reading the output

Status	Meaning
PASS	Check succeeded — component is reachable and healthy
FAIL	Check failed — component is not responding or returned unexpected output
SKIP	Component is not yet deployed — this is not an error

SKIP is expected for components that have not been deployed yet. During Day 1, most checks will be SKIP early on and graduate to PASS as each component is deployed. A SKIP that was previously PASS indicates something was removed or the check's prerequisite disappeared.

Reference

For the full list of checks, flags, and output format, see the verify-enclave.sh reference.

---

Observability Stack

:::info Coming soon The full observability stack (Grafana, Prometheus, Alertmanager) will be documented here once it is deployed. :::

The planned observability deployment:

• VIP: 10.0.0.40 (the observability-ingress Harvester LoadBalancer)
• Namespace: vms-observability on Harvester (for VMs), and a corresponding namespace on the RKE2 cluster for the monitoring Helm release
• Components: Prometheus for metrics collection, Grafana for dashboards, Alertmanager for routing alerts

Rancher Manager's built-in Monitoring app (which deploys the rancher-monitoring Helm chart) is the planned delivery mechanism. All images are served from Harbor — no external metrics endpoints or remote read/write.

Key dashboards planned:

• Node resource utilization (CPU, memory, disk) across all RKE2 nodes and the DGX Spark
• GPU utilization and memory from DCGM Exporter (DGX Spark)
• Harbor registry health and storage consumption
• RKE2 control plane latency and etcd health
• Hauler registry availability (port 5000 on nuc-00)

Until the observability stack is deployed, verify-enclave.sh is the primary health signal.