Skip to main content

Troubleshooting

Start with verify-enclave.sh — it runs all checks in bootstrap order and tells you which layer failed. The first FAIL in the output is the root cause. Everything below it may also be failing as a consequence, but fix the first failure first.

bash /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/verify-enclave.sh

The sections below give per-layer diagnostic commands for when verify-enclave.sh identifies a failing layer.

nuc-00 Services​

The bastion runs DNS, DHCP, NTP, web, and TFTP. If any of these are down, nearly every other component will fail — nodes cannot boot, DNS queries fail silently, and Hauler's registry may be inaccessible.

Service not active

systemctl status named
systemctl status dhcpd
systemctl status chronyd
systemctl status apache2
systemctl status tftp.socket
# If a service is failed:
journalctl -u named -n 50 --no-pager

DNS not resolving

# Test authoritative resolution for a known record
dig @10.0.0.10 rancher.carbide-enclave.kubernerdes.com
dig @10.0.0.10 harbor.carbide-enclave.kubernerdes.com
dig @10.0.0.10 keycloak.carbide-enclave.kubernerdes.com

# Check named status and zone file errors
systemctl status named
journalctl -u named -n 50 --no-pager

DHCP not handing out leases

# Check active leases
cat /var/lib/dhcp/db/dhcpd.leases

# Verify config parses cleanly
dhcpd -t -cf /etc/dhcpd.conf

journalctl -u dhcpd -n 50 --no-pager

step-ca​

step-ca is the internal root CA and ACME server. cert-manager's StepIssuer talks to it directly. If step-ca is down, no new certificates can be issued.

ACME endpoint not responding

systemctl status step-ca
journalctl -u step-ca -n 50 --no-pager

# Health check (the cert is self-signed at this layer, so -k is acceptable here)
curl -k https://ca.carbide-enclave.kubernerdes.com:8443/health
# Expected: {"status":"ok"}

CA cert not in trust store

openssl verify \
  -CAfile /etc/step-ca/certs/root_ca.crt \
  /etc/step-ca/certs/intermediate_ca.crt
# Expected: intermediate_ca.crt: OK

Root cert not reachable from the cluster

# Should return the PEM-encoded root cert
curl -s http://10.0.0.10/step/carbide-enclave-root-ca.crt | openssl x509 -noout -subject -dates

Hauler​

Hauler's registry runs on nuc-00 port 5000 and serves images during Harvester install and initial RKE2 bootstrapping. After Harbor is up, it becomes a staging area for the Hauler → Harbor push.

Registry not responding on port 5000

# Is the hauler serve process running?
ps aux | grep hauler

# Is port 5000 listening?
ss -tlnp | grep 5000

# Test a pull (substitute a known image ref)
curl -s http://10.0.0.10:5000/v2/_catalog | jq .

Image not found in Hauler store

The store may be incomplete from a partial sync. Re-run the sync to pull missing artifacts:

bash /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/hauler.sh sync

If the sync requires internet and you are on the airgap side, you need to collect on the internet side first, save to a tarball, transfer, and load. See Updates.

Harvester​

Node not joining

Check the node's console via KVM or physical access. Common causes:

  • DHCP not assigning the correct IP (check dhcpd.leases and MAC entries in dhcpd.conf)
  • iPXE failed to fetch the node config (check Apache access log: tail -f /var/log/apache2/access_log)
  • The cluster VIP 10.0.0.100 was not reachable when nuc-02/03 tried to join

VIP not reachable

kubectl --kubeconfig ~/.kube/carbide-enclave-harvester.kubeconfig \
  get loadbalancer -A
# Look for the rancher-api LB showing ALLOCATED-ADDRESS = 10.0.0.30

If the VIP is not allocated, check whether the IPPool resource exists and has the correct CIDR, and whether the Harvester LB controller pod is running:

kubectl --kubeconfig ~/.kube/carbide-enclave-harvester.kubeconfig \
  get pods -n harvester-system -l app=harvester-load-balancer

CA not injected

kubectl --kubeconfig ~/.kube/carbide-enclave-harvester.kubeconfig \
  get settings.harvesterhci.io additional-ca \
  -o jsonpath='{.value}' | openssl x509 -noout -subject -dates

If this returns an empty string, re-run 30_bootstrap-harvester.sh. If it returns a cert but Harbor TLS is still rejected, verify the cert being served at http://10.0.0.10/step/carbide-enclave-root-ca.crt matches the CA that signed Harbor's certificate.

RKE2​

Node stuck NotReady

# Check the RKE2 service log on the affected node
ssh mansible@10.0.0.31 "sudo journalctl -u rke2-server -n 100 --no-pager"
# For agent nodes:
ssh mansible@10.0.0.251 "sudo journalctl -u rke2-agent -n 100 --no-pager"

Image pull failure

RKE2 nodes pull images from the Hauler registry (during bootstrap) or from Harbor (after Harbor is up). Image pull failures usually mean one of:

  • The registry endpoint is not reachable from the node
  • The /etc/rancher/rke2/registries.yaml on the node has the wrong mirror URL
  • The Harbor certificate is not trusted (CA not injected into the node trust store)
# Check the mirror config on a node
ssh mansible@10.0.0.31 "cat /etc/rancher/rke2/registries.yaml"

# Confirm the Hauler registry is reachable from the node
ssh mansible@10.0.0.31 "curl -s http://10.0.0.10:5000/v2/_catalog | head -c 200"

RKE2 VIP not reachable

The rancher-api Harvester LoadBalancer must be allocated and pointing at healthy backend VMs.

kubectl --kubeconfig ~/.kube/carbide-enclave-harvester.kubeconfig \
  get loadbalancer rancher-api -n vms-rancher

Confirm the RKE2 VMs are running and carry the label tag.harvesterhci.io/role: rke2-server.

cert-manager​

Certificate not issuing

kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  describe certificate -n <namespace> <cert-name>

# Check the StepIssuer is ready
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  get stepissuer -A
# READY should be True

# Check cert-manager logs for ACME errors
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  logs -n cert-manager -l app=cert-manager --tail=50

Common causes: step-ca not reachable from inside the cluster, StepIssuer provisioner credentials missing or expired, or the ACME challenge endpoint not resolving via cluster DNS.

Harbor​

Not reachable

# Is the ingress responding?
curl -sk https://harbor.carbide-enclave.kubernerdes.com/api/v2.0/health | jq .

# Check the Harbor workloads
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  get pods -n harbor

# Check the ingress object
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  get ingress -n harbor

# Check cert-manager issued the TLS cert
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  get certificate -n harbor

Login failing

If using local Harbor accounts: reset the admin password via the Harbor web UI (System → Users → admin → Set Password) or via the Harbor API.

If using Keycloak OIDC auth: check the OIDC provider config in Harbor's admin panel and confirm the Keycloak realm/client is active:

# Keycloak health
curl -sk https://keycloak.carbide-enclave.kubernerdes.com/health/ready

Keycloak​

Health check failing

kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  logs -n keycloak -l app.kubernetes.io/name=keycloak --tail=50

# Keycloak readiness probe endpoint
curl -sk https://keycloak.carbide-enclave.kubernerdes.com/health/ready
# Expected: {"status":"UP"}

Common causes: the PostgreSQL backend is not reachable, the TLS certificate is not trusted, or a Keycloak pod has not yet completed its startup sequence (it is slow on first boot).

Rancher Manager​

Ping failing

# Check Rancher pods
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  get pods -n cattle-system

# Rancher deployment rollout status
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  rollout status deployment/rancher -n cattle-system

# Check ingress
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  get ingress -n cattle-system

# Check the TLS cert
kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  get certificate -n cattle-system

# Rancher API health (from nuc-00)
curl -sk https://rancher.carbide-enclave.kubernerdes.com/ping
# Expected: pong

If Rancher pods are CrashLooping, check whether system-default-registry (Harbor) is reachable and whether the Harbor certificate is trusted by the node. A common cause is Harbor's TLS certificate changing (e.g., after a cert-manager renewal) before the new cert is propagated to node trust stores.

tip

If you are locked out of Rancher due to a broken OIDC config, you can reset the auth provider by deleting the AuthConfig resource. This restores local authentication and lets you log in with the bootstrap admin password.

kubectl --kubeconfig ~/.kube/carbide-enclave-rke2.kubeconfig \
  delete authconfig keycloakoidc

Rancher will fall back to local authentication immediately — no restart needed.