Highly Available Intermediate Certificate Authority Using Kubernetes

In our own private infrastructure environment, we often need to use our own self-signed TLS certificates to serve our sites over HTTPS.

Step CA can help you generate TLS certificates for your sites using the ACME protocol, and automate the TLS certificate renewal process as well.

In this post we will walk though the process of deploying a PostgreSQL cluster on Kubernetes, then deploying a Step CA Intermediate Certificate Authority that will use the PostgreSQL cluster as the database.

Here is how the final architecture will look like:

Architecture

Preparation

To start the entire deployment, let's create the stepca Kubernetes namespace where everything will be deployed to:

kubectl create ns stepca

Sign a leaf TLS cert for the domain where you will be hosting your Step CA. We can inject the TLS secret as follows:

kubectl create secret tls stepca-tls -n stepca --key private.key --cert public.crt

Download the step binary here and place the binary in /usr/bin

Then generate an intermediate certificate signing request:

step certificate create "Intermediate CA Name" intermediate.csr intermediate_ca_key --csr

Transfer the certificate signing request to your existing root CA and get it signed. You should have the root_ca.crt from your existing root CA, intermediate_ca.crt from signing the certificate signing request, and intermediate_ca.key that was created when you generated the intermediate certificate signing request from the previous step.

Deploying a PostgreSQL Cluster

There are many available PostgreSQL Operators that can help you lifecycle a highly available PostgreSQL cluster. In this example, we will be using CloudNativePG. I like CloudNativePG because the operator creates the database instances using Pods instead of StatefulSets, which avoids all the limitations that comes with StatefulSets.

To install the CloudNativePG Operator, run the following command:

kubectl apply -f \ https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.22/releases/cnpg-1.22.0.yaml

You can verify the Operator is installed with:

kubectl get deployment -n cnpg-system cnpg-controller-manager

Here is the YAML deployment file to create the Postgres Cluster. In this example we will be storing our postgres backups to our own MinIO instance hosted at minio.domain.org

# Inject MinIO Credentials as Secret
apiVersion: v1
kind: Secret
metadata:
  name: minio
  namespace: stepca
type: Opaque
stringData:
  ACCESS_KEY_ID: minio # S3 username
  ACCESS_SECRET_KEY: minio123 # S3 password
---
# Step CA Deployment YAML
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: stepca-postgres
  namespace: stepca
spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:15.0
  instances: 3
  primaryUpdateStrategy: unsupervised # Rolling update process to be automated and managed by Kubernetes
  monitoring:
    enablePodMonitor: true # Expose prometheus metrics
  storage:
    storageClass: vsan-default-storage-policy # Define the storage class you use in your Kubernetes cluster
    size: 1Gi
  backup:
    # Configure to use S3 to store backup resources
    barmanObjectStore:
      destinationPath: s3://stepca/ # S3 bucket location
      endpointURL: https://minio.domain.org # S3 endpoint
      s3Credentials:
        accessKeyId:
          name: minio
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: minio
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
        encryption: AES256
      retentionPolicy: "7d"

You should see the following Kubernetes resources once your PostgreSQL cluster is created:

Services:

stepca-postgres-r
- applications to connect to any of the instances for read-only workloads
stepca-postgres-ro
- applications to connect to any of the hot standby, non-primary replicas for read-only workloads
stepca-postgres-rw
- applications to connect to the primary instance for read-write workloads

Secrets:

stepca-postgres-app
- database credentials for the default user called app, corresponds to the user owning the database
stepca-postgres-ca
- self-signed CA generated and used to support TLS within the postgres cluster
stepca-postgres-replication
- streaming replication client certificate generated by the client CA
stepca-postgres-server
- server TLS certificate signed by the server CA
stepca-postgres-superuser
- superuser credentials to be used only for administrative purposes, corresponds to the postgres user
stepca-postgres-token
- kubernetes service account created for the database operator

Monitoring:

When enablePodMonitor is set to true, CloudNativePG will automatically expose prometheus metrics relating to CloudNativePG clusters, and create a PodMonitor resource for your prometheus to scrape the endpoint
- The pre-requisite is that you must have prometheus already installed

PostgreSQL Backups

For on-demand backups, apply the following YAML:

apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: backup-example
spec:
  cluster:
    name: stepca-postgres

To schedule backups, apply the following YAML:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: backup-daily-midnight
  namespace: stepca
spec:
  schedule: "0 0 16 * * *" # 0000 SGT in UTC time
  backupOwnerReference: self
  cluster:
    name: stepca-postgres

To restore backup from S3 object store, apply the following YAML:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: stepca-postgres
  namespace: stepca
spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:15.0
  instances: 3
  primaryUpdateStrategy: unsupervised # Rolling update process to be automated and managed by Kubernetes
  monitoring:
    enablePodMonitor: true # Expose prometheus metrics
  storage:
    storageClass: vsan-default-storage-policy # Define the storage class you use in your Kubernetes cluster
    size: 1Gi
  backup:
    # Configure to use S3 to store backup resources
    barmanObjectStore:
      destinationPath: s3://stepca/ # S3 bucket location
      endpointURL: https://minio.domain.org # S3 endpoint
      s3Credentials:
        accessKeyId:
          name: minio
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: minio
          key: ACCESS_SECRET_KEY
      wal:
        compression: gzip
        encryption: AES256
      retentionPolicy: "7d"
  bootstrap:
    recovery:
      source: stepca-postgres
      recoveryTarget:
        # Use backupID and targetImmediate to backup to that instant and stop immediately, or use targetTime to do point-in-time recovery where the database will run through the WAL to the timestamp specified after restoring from the nearest base backup
        backupID: 20240102T160000
        targetTime: "2024-01-02T09:00:25"
  externalClusters:
    - name: stepca-postgres # Name has to be same as the previous cluster since barman will be searching for backups based on the name
      barmanObjectStore:
        destinationPath: s3://stepca/
        endpointURL: https://minio.domain.org # S3 endpoint
        s3Credentials:
          accessKeyId:
            name: minio
            key: ACCESS_KEY_ID
          secretAccessKey:
            name: minio
            key: ACCESS_SECRET_KEY
          wal:
            maxParallel: 8 # Take advantage of the parallel WAL restore feature to dedicate up to 8 concurrent jobs to fetch required WAL files from the archive

Deploying Step CA

Step CA includes a helm chart to deploy on Kubernetes, but there is a disclaimer that they can only support 1 replica instance. Therefore we will not be using their helm chart, and instead we take the following deployment path:

Run a Step CA on Docker
Retrieve the configuration template to use in our actual deployment
Deploy our Step CA with our specified configuration injected as a Kubernetes secret

To run Step CA on Docker, we can use the following docker-compose.yml file:

version: '3.3'
services:
  ca:
    image: smallstep/step-ca:0.24.1
    networks:
      - default
    ports:
      - "9000:9000"
    environment:
      - DOCKER_STEPCA_INIT_NAME=${DOCKER_STEPCA_INIT_NAME} # Name of your CA - this will be the issuer of your CA certificates
      - DOCKER_STEPCA_INIT_DNS_NAMES=${DOCKER_STEPCA_INIT_DNS_NAMES} # Hostname(s) or IPs that the CA will accept requests on
      - DOCKER_STEPCA_INIT_PROVISIONER_NAME=${DOCKER_STEPCA_INIT_PROVISIONER_NAME} # Label for the initial admin (JWK) provisioner. Default: "admin"
      - DOCKER_STEPCA_INIT_PASSWORD=${DOCKER_STEPCA_INIT_PASSWORD} # Password for the encrypted CA keys and the default CA provisioner
    volumes:
      - ./data/home/step:/home/step
    restart: always
networks:
  default:
    ipam:
      driver: default
      config:
        - subnet: "172.97.0.0/16"
volumes:
  step_home:

From here, we can retrieve the configuration files from /home/step/, then edit the template to use the PostgreSQL cluster that we deployed earlier. A reference of the configuration options can be found here

A customised config/ca.json is as follows:

{
  "root": "/home/step/certs/root_ca.crt",
  "federatedRoots": null,
  "crt": "/home/step/certs/intermediate_ca.crt",
  "key": "/home/step/certs/intermediate_ca.key",
  "address": "9000",
  "insecureAddress": "true",
  "dnsNames": [
    "localhost",
    "ca.domain.org"
  ],
  "logger": {
    "format": "text"
  },
  "db":{
    "type": "postgresql",
    "dataSource": "postgresql://app:password@stepca-postgres-rw.stepca.svc.cluster.local:5432",
    "database": "app"
  },
  "authority": {
    "provisioners": [
      {
        "type": "JWK",
        "name": "admin",
        "key": {
          "use": "sig",
          "kty": "EC",
          "kid": "YYNxZ0rq0WsT2MlqLCWvgme3jszkmt99KjoGEJJwAKs",
          "crv": "P-256",
          "alg": "ES256",
          "x": "LsI8nHBflc-mrCbRqhl8d3hSl5sYuSM1AbXBmRfznyg",
          "y": "F99LoOvi7z-ZkumsgoHIhodP8q9brXe4bhF3szK-c_w"
        },
        "encryptedKey": "eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJjdHkiOiJqd2sranNvbiIsImVuYyI6IkEyNTZHQ00iLCJwMmMiOjEwMDAwMCwicDJzIjoiVERQS2dzcEItTUR4ZDJxTGo0VlpwdyJ9.2_j0cZgTm2eFkZ-hrtr1hBIvLxN0w3TZhbX0Jrrq7vBMaywhgFcGTA.mCasZCbZJ-JT7vjA.bW052WDKSf_ueEXq1dyxLq0n3qXWRO-LXr7OzBLdUKWKSBGQrzqS5KJWqdUCPoMIHTqpwYvm-iD6uFlcxKBYxnsAG_hoq_V3icvvwNQQSd_q7Thxr2_KtPIDJWNuX1t5qXp11hkgb-8d5HO93CmN7xNDG89pzSUepT6RYXOZ483mP5fre9qzkfnrjx3oPROCnf3SnIVUvqk7fwfXuniNsg3NrNqncHYUQNReiq3e9I1R60w0ZQTvIReY7-zfiq7iPgVqmu5I7XGgFK4iBv0L7UOEora65b4hRWeLxg5t7OCfUqrS9yxAk8FdjFb9sEfjopWViPRepB0dYPH8dVI.fb6-7XWqp0j6CR9Li0NI-Q",
        "claims": {
          "enableSSHCA": false,
          "disableRenewal": false,
          "allowRenewalAfterExpiry": false
        },
        "options": {
          "x509": {},
          "ssh": {}
        }
      },
      {
        "type": "ACME",
        "name": "acme",
        "forceCN": true,
        "claims": {
          "maxTLSCertDuration": "2160h0m0s",
          "defaultTLSCertDuration": "2160h0m0s",
          "policy": {
            "x509": {
              "allow": ["*.domain.org"]
            }
          }
        },
        "options": {
          "x509": {
            "templateFile": "templates/certs/x509/leaf.tpl"
          }
        }
      }
    ],
    "template": {},
    "backdate": "1m0s"
  },
  "tls": {
    "cipherSuites": [
      "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
      "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
    ],
    "minVersion": 1.2,
    "maxVersion": 1.3,
    "renegotiation": false
  },
  "commonName": "Step Online CA"
}

This ca.json references a custom leaf cert template leaf.tpl to set Subject Alternative Name (SAN) in the provisioned TLS certificate. Check here on how to configure your own Step CA templates.

The custom leaf.tpl as follows:

{
  "subject": {{ toJson .subject }},
{{- if .Insecure.User.dnsName }}
  "dnsNames": {{ toJSON .Insecure.User.dnsName }},
{{- else }}
  "sans": {{ toJson .SANs }},
{{- end }}
{{-if typeIs "*rsa.PublicKey" .Insecure.CR.PublicKey }}
  "keyUsage": ["keyEncipherment", "digitalSignature"],
{{- else }}
  "keyUsage": ["digitalSignature"],
{{- end }}
  "extKeyUsage": ["serverAuth", "clientAuth]
}

We can then create the entire deployment YAML file including the injecting of configuration as secrets, and the referencing of the secrets from the deployment resource.

Our example deployment assumes an NGINX ingress controller is already deployed in the cluster.
The deployment.yaml is as follows:

apiVersion: v1
kind: Secret
metadata:
  name: stepca-config
  namespace: stepca
type: Opaque
stringData:
  intermediate_ca_crt: |
    <YOUR INTERMEDIATE CA CRT>
  root_ca.crt: |
    <YOUR ROOT CA CRT>
  ca.json: |
    {
      "root": "/home/step/certs/root_ca.crt",
      "federatedRoots": null,
      "crt": "/home/step/certs/intermediate_ca.crt",
      "key": "/home/step/certs/intermediate_ca.key",
      "address": "9000",
      "insecureAddress": "true",
      "dnsNames": [
        "localhost",
        "ca.domain.org"
      ],
      "logger": {
        "format": "text"
      },
      "db":{
        "type": "postgresql",
        "dataSource": "postgresql://app:password@stepca-postgres-rw.stepca.svc.cluster.local:5432",
        "database": "app"
      },
      "authority": {
        "provisioners": [
          {
            "type": "JWK",
            "name": "admin",
            "key": {
              "use": "sig",
              "kty": "EC",
              "kid": "YYNxZ0rq0WsT2MlqLCWvgme3jszkmt99KjoGEJJwAKs",
              "crv": "P-256",
              "alg": "ES256",
              "x": "LsI8nHBflc-mrCbRqhl8d3hSl5sYuSM1AbXBmRfznyg",
              "y": "F99LoOvi7z-ZkumsgoHIhodP8q9brXe4bhF3szK-c_w"
            },
            "encryptedKey": "eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJjdHkiOiJqd2sranNvbiIsImVuYyI6IkEyNTZHQ00iLCJwMmMiOjEwMDAwMCwicDJzIjoiVERQS2dzcEItTUR4ZDJxTGo0VlpwdyJ9.2_j0cZgTm2eFkZ-hrtr1hBIvLxN0w3TZhbX0Jrrq7vBMaywhgFcGTA.mCasZCbZJ-JT7vjA.bW052WDKSf_ueEXq1dyxLq0n3qXWRO-LXr7OzBLdUKWKSBGQrzqS5KJWqdUCPoMIHTqpwYvm-iD6uFlcxKBYxnsAG_hoq_V3icvvwNQQSd_q7Thxr2_KtPIDJWNuX1t5qXp11hkgb-8d5HO93CmN7xNDG89pzSUepT6RYXOZ483mP5fre9qzkfnrjx3oPROCnf3SnIVUvqk7fwfXuniNsg3NrNqncHYUQNReiq3e9I1R60w0ZQTvIReY7-zfiq7iPgVqmu5I7XGgFK4iBv0L7UOEora65b4hRWeLxg5t7OCfUqrS9yxAk8FdjFb9sEfjopWViPRepB0dYPH8dVI.fb6-7XWqp0j6CR9Li0NI-Q",
            "claims": {
              "enableSSHCA": false,
              "disableRenewal": false,
              "allowRenewalAfterExpiry": false
            },
            "options": {
              "x509": {},
              "ssh": {}
            }
          },
          {
            "type": "ACME",
            "name": "acme",
            "forceCN": true,
            "claims": {
              "maxTLSCertDuration": "2160h0m0s",
              "defaultTLSCertDuration": "2160h0m0s",
              "policy": {
                "x509": {
                  "allow": ["*.domain.org"]
                }
              }
            },
            "options": {
              "x509": {
                "templateFile": "templates/certs/x509/leaf.tpl"
              }
            }
          }
        ],
        "template": {},
        "backdate": "1m0s"
      },
      "tls": {
        "cipherSuites": [
          "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
          "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
        ],
        "minVersion": 1.2,
        "maxVersion": 1.3,
        "renegotiation": false
      },
      "commonName": "Step Online CA"
    }
  defaults.json: |
    {
    "ca-url": "https://localhost:9000",
    "ca-config": "/home/step/config/ca.json",
    "fingerprint": "93cff06dc36251fb0c4985d0b5ed7265a368cd70697fba90355c93cc4aabff0d",
    "root": "/home/step/certs/root_ca.crt"
    }
  intermediate_ca_key: |
    <YOUR INTERMEDIATE CA KEY>
  password: |
    <YOUR STEP CA ADMIN PASSWORD>
  leaf.tpl: |
    {
    "subject": {{ toJson .subject }},
    {{- if .Insecure.User.dnsName }}
    "dnsNames": {{ toJSON .Insecure.User.dnsName }},
    {{- else }}
    "sans": {{ toJson .SANs }},
    {{- end }}
    {{-if typeIs "*rsa.PublicKey" .Insecure.CR.PublicKey }}
    "keyUsage": ["keyEncipherment", "digitalSignature"],
    {{- else }}
    "keyUsage": ["digitalSignature"],
    {{- end }}
    "extKeyUsage": ["serverAuth", "clientAuth]
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stepca
  namespace: stepca
spec:
  replicas: 1
  selector:
    matchLabels:
      app: stepca
  template:
    metadata:
      labels:
        app: stepca
    spec:
      containers:
        - name: stepca
          image: smallstep/step-ca:0.23.0
          ports:
            - containerPort: 9000
          env:
            - name: DOCKER_STEPCA_INIT_NAME
        value: stepca
      - name: DOCKER_STEPCA_INIT_DNS_NAMES
        value: localhost, ca.domain.org
      - name: DOCKER_STEPCA_INIT_PROVISIONER_NAME
        value: admin
      - name: DOCKER_STEP_CA_INIT_PASSWORD
        valueFrom:
          secretKeyRef:
            name: stepca-config
            key: password
      volumeMounts:
        - mountPath: /home/step
          name: stepca-config
          readOnly: false
      volumes:
        - name: stepca-config
          secret:
            secretName: stepca-config
            items:
              - key: intermediate_ca.crt
                path: certs/intermediate_ca.crt
              - key: root_ca.crt
                path: certs/root_ca.crt
              - key: ca.json
                path: config/ca.json
              - key: defaults.json
                path: config/defaults.json
              - key: intermediate_ca_key
                path: secrets/intermediate_ca_key
              - key: password
                path: secrets/password
              - key: leaf.tpl
                path: templates/certs/x509/leaf.tpl
              defaultMode: 0755
---
apiVersion: v1
kind: Service
metadata:
  name: stepca
  namespace: stepca
  labels:
    app: stepca
spec:
  type: ClusterIP
  ports:
  - port: 9000
  selector:
    app: stepca
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: stepca-ingress
  namespace: stepca
  annotations:
    kubernetes.io/ingress.class: 'nginx'
    nginx.ingress.kubernetes.io/backend-protocol: 'HTTPS'
spec:
  tls:
    - hosts:
        - ca.domain.org
        secretName: stepca-tls
  rules:
    - host: ca.domain.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: stepca
                port:
                  number: 9000
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: stepca
  namespace: stepca
  labels:
    app: stepca
    resource: horizontalpodautoscaler
spec:
  maxReplicas: 6
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: stepca
  targetCPUUtilizationPercentage: 80

You check the health of your Intermediate Certificate Authority:

curl https://ca.domain.org/health

# {"status":"ok"}

curl https://ca.domain.org/acme/acme/directory

# {"newNonce":"https://ca.domain.org/acme/acme/new-nonce","newAccount":"https://ca.domain.org/acme/acme/new-account","newOrder":"https://ca.domain.org/acme/acme/new-order","revokeCert":"https://ca.domain.org/acme/acme/revoke-cert","keyChange":"https://ca.domain.org/acme/acme/key-change"}