Skip to main content

RKE2 Management Cluster

Three SL-Micro VMs running on Harvester (rancher-01, rancher-02, rancher-03) form an HA RKE2 server cluster. All cluster traffic reaches them via the 10.0.0.30 Harvester LB. The cluster hosts cert-manager, Harbor, Keycloak, and Rancher Manager.

Overview​

VMIPRole
rancher-0110.0.0.31RKE2 cluster-init
rancher-0210.0.0.32RKE2 server join
rancher-0310.0.0.33RKE2 server join
  • RKE2 VIP: 10.0.0.30 — Kubernetes API :6443, supervisor :9345, Rancher HTTPS :443
  • Kubeconfig: ~/.kube/carbide-enclave-rke2.kubeconfig (retrieved by bootstrap script)
  • Guest OS: SL-Micro 6.2 (immutable root; /etc is a writable overlay)

Prerequisites​

  • Harvester cluster up and post-install complete (Harvester Post-Install)
  • Hauler OCI registry running on nuc-00 port 5000 (sudo bash scripts/hauler.sh serve)
  • Hauler file server running on nuc-00 port 8080 — or 40_bootstrap-rke2.sh will start both
  • RKE2_TOKEN set in ~/.config/RGS/creds
  • SL-Micro QCOW2 image uploaded to Harvester (see VM Image)
  • Three VMs provisioned and reachable at .31/.32/.33 (see Provision VMs)

VM image​

The RKE2 VMs run SL-Micro 6.2. The QCOW2 image must be uploaded to Harvester before OpenTofu can use it.

Upload via Harvester UI: Images → Create → URL

FieldValue
Namesl-micro-6.2
URL(SL-Micro QCOW2 download URL, or local upload from nuc-00)
Namespacedefault

Alternatively, if a Packer-built image with RKE2 pre-baked is available from infra/packer/, upload that image instead — it skips the RKE2 artifact download step.

Credentials​

Add to ~/.config/RGS/creds before running the bootstrap script:

export RKE2_TOKEN="<a strong random token — same value on all three nodes>"

Generate a token:

openssl rand -hex 32

Provision VMs​

VMs are provisioned via OpenTofu using the Harvester provider.

cd infra/tofu/rke2-cluster
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars — set harvester_kubeconfig, vm_image, ssh key, etc.
tofu init
tofu plan
tofu apply

After tofu apply, verify the VMs are up and SSH is reachable:

ssh mansible@10.0.0.31 hostname
ssh mansible@10.0.0.32 hostname
ssh mansible@10.0.0.33 hostname

:::tip Cloud-init The VMs are created with platform/rke2/cloud-init-rancher.yaml as their user-data. This installs the step-ca root CA at first boot and writes registries.yaml pointing at Harbor.

During the initial bootstrap, 40_bootstrap-rke2.sh overwrites registries.yaml on each node to point at the Hauler registry (:5000) since Harbor is not yet running. After Harbor is up and hauler.sh push completes, update registries.yaml on each node to point back at Harbor. :::

Bootstrap RKE2​

Run from nuc-00:

bash /srv/www/htdocs/carbide-enclave.kubernerdes.com/scripts/40_bootstrap-rke2.sh

The script is idempotent — safe to re-run. It checks state before each step and skips anything already done.

What the script does​

StepAction
1Starts Hauler OCI registry (:5000) and file server (:8080) if not already running
2Installs step-ca root cert on each VM (/etc/pki/trust/anchors/)
3Downloads RKE2 artifacts from Hauler file server to each VM
4Writes registries.yaml (Hauler as registry) and config.yaml on each VM
5Installs RKE2 on rancher-01 with cluster-init: true
6Waits for rancher-01 to reach Ready
7Installs RKE2 on rancher-02 and rancher-03 (join via VIP)
8Waits for all three nodes to reach Ready
9Retrieves kubeconfig → ~/.kube/carbide-enclave-rke2.kubeconfig

Expected runtime​

  • rancher-01 Ready: ~5–8 minutes after install starts
  • Full cluster (all three nodes Ready): ~10–15 minutes total

Verify​

export KUBECONFIG=~/.kube/carbide-enclave-rke2.kubeconfig

# All three nodes Ready
kubectl get nodes

# Expected:
# NAME          STATUS   ROLES                       AGE   VERSION
# rancher-01    Ready    control-plane,etcd,master   Xm    v1.32.x+rke2r2
# rancher-02    Ready    control-plane,etcd,master   Xm    v1.32.x+rke2r2
# rancher-03    Ready    control-plane,etcd,master   Xm    v1.32.x+rke2r2

# System pods running
kubectl get pods -n kube-system

# Hauler registry reachable from a node
ssh mansible@10.0.0.31 "curl -sf http://hauler.carbide-enclave.kubernerdes.com:5000/v2/"

RKE2 config reference​

The config written to each node by the bootstrap script:

/etc/rancher/rke2/config.yaml (rancher-01):

cluster-init: true
token: <RKE2_TOKEN>
tls-san:
  - 10.0.0.30           # Harvester LB VIP
  - rke2.carbide-enclave.kubernerdes.com
  - rancher.carbide-enclave.kubernerdes.com
  - rancher-01.carbide-enclave.kubernerdes.com
  - ...
system-default-registry: hauler.carbide-enclave.kubernerdes.com:5000

/etc/rancher/rke2/registries.yaml (all nodes, bootstrap phase):

mirrors:
  "hauler.carbide-enclave.kubernerdes.com:5000":
    endpoint:
      - "http://hauler.carbide-enclave.kubernerdes.com:5000"

After Harbor is up and hauler.sh push completes, this is replaced with a Harbor config — see Harbor Registry.

Key paths on VMs​

PathPurpose
/etc/rancher/rke2/config.yamlRKE2 server config
/etc/rancher/rke2/registries.yamlContainer registry mirror config
/etc/rancher/rke2/rke2.yamlCluster kubeconfig (root-readable)
/opt/rke2/bin/kubectlkubectl (SL-Micro immutable path)
/var/lib/rancher/rke2/agent/images/Airgap image bundles
/var/log/journal/RKE2 service logs (journalctl -u rke2-server)

Troubleshooting​

Node stuck in NotReady:

ssh mansible@10.0.0.3X "sudo journalctl -u rke2-server -n 100 --no-pager"

Image pull failure:

# Confirm Hauler registry is serving the image
curl -s http://10.0.0.10:5000/v2/<image>/tags/list

# Confirm registries.yaml is correct on the node
ssh mansible@10.0.0.3X "sudo cat /etc/rancher/rke2/registries.yaml"

VIP not reachable:

KUBECONFIG=~/.kube/carbide-enclave-harvester.kubeconfig \
  kubectl get loadbalancer rancher-api -n vms-rancher

Next step​

Deploy cert-manager and the StepIssuer: cert-manager + StepIssuer