Skip to main content

cert-manager + TLS Issuers

Status: Complete ✅ — cert-manager running, two ClusterIssuers active, wildcard cert issued.

cert-manager is the Kubernetes-native certificate lifecycle controller. In the enclave it integrates with step-ca (the internal root CA on nuc-00) via ACME to automatically issue and renew TLS certificates for all cluster services.

References​

ResourceURL
cert-manager docshttps://cert-manager.io/docs
step-ca ACME guidehttps://smallstep.com/docs/step-ca/acme-basics
RFC 2136 DNS updatehttps://datatracker.ietf.org/doc/html/rfc2136

Architecture​

  cert-manager (in RKE2 cluster)
       │
       ├─ ClusterIssuer: step-ca-acme  (HTTP01)
       │       └─ issues: per-service certs (Harbor, Keycloak, …)
       │          solver: nginx ingress HTTP01 challenge
       │          CA: step-ca ACME @ ca.carbide-enclave.kubernerdes.com:8443
       │
       └─ ClusterIssuer: step-ca-dns01 (DNS01/RFC2136)
               └─ issues: wildcard *.carbide-enclave.kubernerdes.com
                  solver: dynamic DNS TXT update via TSIG → BIND on nuc-00
                  CA: step-ca ACME @ ca.carbide-enclave.kubernerdes.com:8443

Both issuers use the same step-ca ACME server. The difference is the challenge type:

  • HTTP01 — cert-manager creates a temporary pod; step-ca validates via HTTP over the LAN. Simple, but cannot issue wildcard SANs.
  • DNS01/RFC2136 — cert-manager writes a _acme-challenge TXT record directly to BIND using a TSIG key, then tells step-ca to validate via DNS lookup. Required for wildcard certs.

Prerequisites​

  • RKE2 cluster healthy — RKE2 Management Cluster
  • step-ca running on nuc-00 with ACME provisioner — step-ca
  • BIND on nuc-00 configured with allow-update { key cert-manager-dns01; } (see TSIG setup below)

What was deployed​

cert-manager was installed automatically by 50_bootstrap-rancher.sh as a prerequisite for the Rancher TLS cert. The ClusterIssuers and wildcard cert were created by a separate script:

# Run from nuc-00 as mansible, after 10_bootstrap-nuc-00.sh has generated the TSIG key
bash scripts/60_bootstrap-cert-issuers.sh

ClusterIssuer: step-ca-acme (HTTP01)​

Used for all individual per-service certs. Reference manifest at platform/cert-manager/clusterissuer-acme.yaml.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: step-ca-acme
spec:
  acme:
    server: https://ca.carbide-enclave.kubernerdes.com:8443/acme/acme/directory
    caBundle: <base64 of /etc/step-ca/certs/root_ca.crt>
    privateKeySecretRef:
      name: step-ca-acme-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

ClusterIssuer: step-ca-dns01 (DNS01/RFC2136)​

Used for the wildcard cert. Reference manifest at platform/cert-manager/clusterissuer-dns01.yaml.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: step-ca-dns01
spec:
  acme:
    server: https://ca.carbide-enclave.kubernerdes.com:8443/acme/acme/directory
    caBundle: <base64 of /etc/step-ca/certs/root_ca.crt>
    privateKeySecretRef:
      name: step-ca-dns01-account-key
    solvers:
      - dns01:
          rfc2136:
            nameserver: 10.0.0.10:53
            tsigAlgorithm: HMACSHA256
            tsigKeyName: cert-manager-dns01
            tsigSecretSecretRef:
              name: cert-manager-dns01-tsig
              key: tsig-secret-key

Wildcard Certificate​

*.carbide-enclave.kubernerdes.com + apex carbide-enclave.kubernerdes.com, issued via DNS01. Secret lives in the cert-manager namespace. Reference manifest at platform/cert-manager/certificate-wildcard.yaml.

TSIG setup for DNS01​

cert-manager writes _acme-challenge TXT records directly to BIND using RFC 2136 dynamic DNS updates authenticated with a TSIG key. This is what enables wildcard cert issuance without a public DNS provider.

How it works​

  1. 10_bootstrap-nuc-00.sh generates the TSIG key at /etc/named.d/tsig-cert-manager.key using tsig-keygen -a hmac-sha256 cert-manager-dns01 before named-checkconf runs.
  2. BIND loads the key via include "/etc/named.d/tsig-cert-manager.key" in named.conf.
  3. The forward zone is configured to accept updates from that key: 
    allow-update { key cert-manager-dns01; };
  4. 60_bootstrap-cert-issuers.sh reads the key's secret value and creates a Kubernetes Secret (cert-manager/cert-manager-dns01-tsig) that cert-manager uses when it calls BIND.

:::note TSIG key is not committed The key file at /etc/named.d/tsig-cert-manager.key is runtime-generated and never committed to the repo. If nuc-00 is rebuilt, re-run 10_bootstrap-nuc-00.sh (which regenerates the key), then delete and recreate the cert-manager-dns01-tsig Kubernetes Secret, and delete the existing ClusterIssuer step-ca-dns01 so 60_bootstrap-cert-issuers.sh recreates it with the new key. :::

Issuing a certificate for a new service​

Individual service cert (HTTP01 — recommended)​

Add a Certificate object in the service's namespace. cert-manager issues and renews it automatically:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: harbor-tls
  namespace: harbor
spec:
  secretName: harbor-tls
  issuerRef:
    name: step-ca-acme
    kind: ClusterIssuer
  dnsNames:
    - harbor.carbide-enclave.kubernerdes.com
  duration: 2160h
  renewBefore: 360h

Using the wildcard cert in another namespace​

The wildcard secret lives in cert-manager. Copy it to the target namespace:

kubectl get secret wildcard-carbide-enclave-tls -n cert-manager -o yaml \
  | sed 's/namespace: cert-manager/namespace: TARGET_NS/' \
  | kubectl apply -f -

:::caution Wildcard copies are not auto-renewed Copying the secret is a one-time operation. When cert-manager renews the wildcard cert (90 days before expiry by default), the copy in the other namespace does not update automatically. For production services use an individual Certificate object with HTTP01 instead. :::

Verify​

# Check both issuers are Ready
kubectl get clusterissuer

# Check wildcard cert
kubectl get certificate wildcard-carbide-enclave -n cert-manager

# Check a service cert (example)
kubectl get certificate -A

Next step​

Harbor Registry