Vault HA using Raft

Pratik Kale

10 min readJul 23, 2020

This article will help you,

If you were looking for a way to store your secrets in a secure place and you like what Hashicorp Vault offers.
Want to create a H.A. secret store infrastructure/vault setup in < 30 mins without adding a maintenance overhead.
Have basic knowledge of Kubernetes and have access to a Kubernetes cluster.

Approach

Vault key value engine as a secret store
Raft as backend storage
kubernetes for managing deployment

Why Vault

Vault is one of the industry leading solutions for storing and accessing secrets
Applications can integrate with vault using well documented REST-API
Vault provides easy to use UI where we can set policies and access-control

Why Raft

Raft storage in vault is based on raft consensus algorithm, which replicates data amongst raft nodes making vault highly available (HA)
As raft is file system based, vault does not have to make any network calls to fetch data
Raft storage system is officially supported by Hashicorp

Why Kubernetes

Kubernetes provides an easy to use platform for deployment and maintainence of our vault cluster
If Vault pod goes down kubernetes will bring it up and maintain the given replica set
Raft uses file-system to backup secrets. These file-system can be mounted as a persistent volume claim

Final Architecture we will build using kube yaml config files

vault.yaml stateful set in kubernetes

Note: The yaml comments marked in bold are discussed in more detail

vault-cluster.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  # STATEFUL-SET-NAME
  name: vault-cluster
  namespace: ns
spec:
  serviceName: vaultraft
  replicas: 3
  selector:
    matchLabels:
      app: vaultraft
  template:
    metadata:
      labels:
        app: vaultraft
    spec:
      volumes:
        # WE WILL USE THIS CONFIG MAP TO LOAD VAULT CONFIGS 
        - name: config
          configMap:
            name: raft-config
            items:
              - key: config.hcl
                path: config.hcl 
        # WE WILL USE THIS TLS SECRET FOR HTTPS
        - name: tls
          secret:
            secretName: vault-cert
            items:
            - key: tls.crt
              path: vault.crt
            - key: tls.key
              path: vault.key
      containers:
        - name: vault
          image: docker.io/library/vault:1.4.1
          ports:
            - containerPort: 8200
              name: rest
              protocol: TCP
            - containerPort: 8201
              name: cluster
              protocol: TCP
          env:
            # UNIQUE ID THAT IDENTIFIES VAULT-CLUSTER NODE
            # VAULT NODES WILL COMMUNICATE USING THIS 
            # ID WITH EACH OTHER
            - name: VAULT_RAFT_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: VAULT_ADDR
              value: 127.0.0.1:8200
            - name: VAULT_API_ADDR
              value: 127.0.0.1:8200
            - name: VAULT_CLUSTER_ADDR
              value: 127.0.0.1:8201 
            # TRANSIT VAULT_TOKEN(AUTORENEWS) FOR AUTO UNSEAL
            - name: VAULT_TOKEN
              value: "transit_vault_token"
            - name: SKIP_SETCAP
              value: "true"
            - name: VAULT_SKIP_VERIFY
              value: "true"
          volumeMounts:
            # PVC VOLUME MOUNT WHERE WE WILL STORE VAULT DATA
            - name: data
              mountPath: /vault-data 
            # VOLUME MOUNT WHERE WE WILL LOAD VAULT CONFIG
            - name: config
              mountPath: /vault/config 
            # VOLUME MOUNT WHERE WE WILL LOAD TLS CERT FOR HTTPS
            - name: tls
              mountPath: /opt/ca 
            # LOAD TLS CERTS HERE FOR LINUX TO TRUST THEM
            - name: tls
              mountPath: /etc/ssl/certs
      initContainers:
        - name: fix-permissions
          image: busybox
          resources:
            limits:
              memory: "64Mi"
              cpu: 50m
            requests:
              memory: "64Mi"
              cpu: 50m 
          # INIT CONTAINER TO GIVE VAULT USER PERMISSION TO 
          # STORE DATA IN PVC
          command: ["sh", "-c", "chown -R 100:100 /vault-data"]
          volumeMounts:
            - name: data
              mountPath: /vault-data
            - name: config
              mountPath: /vault/config 
  # PVC FOR VAULT DATA -> SO DATA IS NEVER LOST
  volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          app: vault
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: cinder-standard
        resources:
          requests:
            storage: 15Gi

Lets discuss the above setup one by one

Step1: Create TLS secrets for https to work in vault

kubectl create secret tls vault-cert --cert ~/Downloads/vault.cert \--key ~/Downloads/vault.key -n {yourNamespace}

First step is to create TLS certs. These will be injected in vault-config and in the vault.yaml using Kubernetes secrets

How to create vault.cert (certificate) and (vault.key) private key?

From your companies Certificate Authority(CA) or an external CA create a certificate with following DNS entries in the SAN section. These DNS correspond to pods which we are going to create soon
From your companies Certificate Authority(CA) or an external CA create a certificate with following DNS entries in the SAN section. These DNS correspond to pods which we are going to create soon

https://vault-cluster-0.ns.local
https://vault-cluster-1.ns.local
https://vault-cluster-2.ns.local

If you are creating more than 3 pods please add all the other DNS entries in the SAN section.
The first part of the DNS entry is the stateful set name , followed by the index , then followed by your namespace and cluster name.
Once you obtain the public cert and private key from CA with above DNS entries, create kubernetes secrets as follows. I am assuming you save the cert and key in ~/Downloads as vault.cert and vault.key

kubectl create secret tls vault-cert --cert ~/Downloads/vault.cert \--key ~/Downloads/vault.key -n {yourNamespace}

After which you will volume mount these secrets in vault-cluster.yaml

#-----------------snippet from stateful set vault-cluster.yaml------    volumes:
        # WE WILL USE THIS CONFIG MAP TO LOAD VAULT CONFIGS 
        - name: config
          configMap:
            name: raft-config
            items:
              - key: config.hcl
                path: config.hcl 
#-----------------snippet from stateful set vault-cluster.yaml------          volumeMounts:
            # VOLUME MOUNT WHERE WE WILL LOAD TLS CERT FOR HTTPS
            - name: tls
              mountPath: /opt/ca 
            # LOAD TLS CERTS HERE FOR LINUX TO TRUST THEM
            - name: tls
              mountPath: /etc/ssl/certs

Step2: Create Vault config

apiVersion: v1
kind: ConfigMap
metadata:
  # THIS CONFIG IS USED IN VAULT STATEFUL-SET YAML
  name: raft-config
  namespace: ns
  labels:
    name: raft-config
data:
  config.hcl: |
    storage "raft" { 
      # PVC VOLUME FOR VAULT TO KEEP ALL VAULT DATA
      path  = "/vault-data"
      tls_skip_verify = "true" 
      # CLUSTER STATEFUL-SET DNS ALSO ADDED TO TLS CERTS IN STEP1
      retry_join {
        leader_api_addr = "https://vault-cluster-0.ns.local:8200"
      }
      retry_join {
        leader_api_addr = "https://vault-cluster-1.ns.local:8200"
      }
      retry_join {
        leader_api_addr = "https://vault-cluster-2.ns.local:8200"
      }
    } 
    # TRANSIT VAULT DNS TO AUTOUNSEAL VAULT CLUSTER
    seal "transit" {
     address = "https://vaulttransit.ns.local:8200"
     disable_renewal = "false"
     key_name = "autounseal"
     mount_path = "transit/"
     tls_skip_verify = "true"
    } 
    # LOCATION OF TLS CERT IN VAULT CLUSTER FOR HTTPS HANDSHAKE
    listener "tcp" {
     address = "0.0.0.0:8200"
     tls_cert_file = "/opt/ca/vault.crt"
     tls_key_file = "/opt/ca/vault.key"
     tls_skip_verify = "true"
    }
    ui=true
    disable_mlock = true

After which you will volume mount this config in vault-cluster.yaml

#-----------------snippet from stateful set vault-cluster.yaml------   volumes:

        # WE WILL USE THIS TLS SECRET FOR HTTPS
        - name: tls
          secret:
            secretName: vault-cert
            items:
            - key: tls.crt
              path: vault.crt
            - key: tls.key
              path: vault.key#-----------------snippet from stateful set vault-cluster.yaml------    volumeMounts:
 
             # VOLUME MOUNT WHERE WE WILL LOAD VAULT CONFIG
            - name: config
              mountPath: /vault/config

Step3: Create Transit Vault and Transit Token

Use Transit Vault to autounseal

We will also use transit vault to autounseal the vault cluster. You can also see this video on how to setup a transit vault

#-----------------snippet from configMap----------------------seal "transit" {
     address = "https://vaulttransit.ns.local:8200"
     disable_renewal = "false"
     key_name = "autounseal"
     mount_path = "transit/"
     tls_skip_verify = "true"
    }#-----------------snippet from stateful set vault-cluster.yaml------
          env:
            - name: VAULT_RAFT_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: VAULT_ADDR
              value: 127.0.0.1:8200
            - name: VAULT_API_ADDR
              value: 127.0.0.1:8200
            - name: VAULT_CLUSTER_ADDR
              value: 127.0.0.1:8201
            - name: VAULT_TOKEN
              value: "transit_vault_token"

VAULT_TOKEN will be used to communicate between main vault cluster and transit vault

Why do we need the Transit vault

Vault is by default sealed.
When vault pod in our cluster comes up it will come up in sealed state. Vault will reject all operations in sealed state.
With transit vault, if lets say any of the vault node goes down (pods can go down anytime in a cloud environment) , it will automatically be recreated by kubernetes and will autounseal itself using transit vault.

transitVault.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  # STATEFUL-SET-NAME
  name: transit-cluster
  namespace: ns
  serviceName: vaulttransit
  replicas: 1
  selector:
    matchLabels:
      app: vaulttransit
  template:
    metadata:
      labels:
        app: vaulttransit
    spec:
      volumes:
        # CONFIG FOR TRANSIT VAULT
        - name: config
          configMap:
            name: transit-config
            items:
              - key: config.hcl
                path: config.hcl 
        # TLS VAULT CERT AND KEY FOR HTTPS HANDSHAKE
        - name: tls
          secret:
            secretName: vault-cert
            items:
            - key: tls.crt
              path: vault.crt
            - key: tls.key
              path: vault.key
      containers:
        - name: vault
          image: docker.io/library/vault:1.4.1
          ports:
            - containerPort: 8200
              name: rest
              protocol: TCP
            - containerPort: 8201
              name: cluster
              protocol: TCP
          args:
            - "server"
          lifecycle: 
            # POST EXEC SCRIPT TO UNSEAL TRANSIT VAULT IF 
            # IT GOES DOWN. 
            postStart:
              exec:
                command: ["/bin/sh", "-c", "sh /vault-data/autounseal.sh"]
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: VAULT_ADDR
              value: "https://127.0.0.1:8200"
            - name: VAULT_API_ADDR
              value: "https://127.0.0.1:8200"
            - name: VAULT_CLUSTER_ADDR
              value: "https://$(POD_IP):8201"
            - name: SKIP_SETCAP
              value: "true"
            - name: VAULT_SKIP_VERIFY
              value: "true"
          volumeMounts:
            # PVC FOR TRANSIT VAULT WHICH HOLDS THE DATA
            - name: data
              mountPath: /vault-data 
            # VOLUME MOUNT CONFIG
            - name: config
              mountPath: /vault/config 
            # VOLUME MOUNT TLS CERTS
            - name: tls
              mountPath: /opt/ca
      initContainers:
        - name: fix-permissions
          image: busybox
          resources:
            limits:
              memory: "64Mi"
              cpu: 50m
            requests:
              memory: "64Mi"
              cpu: 50m 
          # PERMISSIONS TO READ TRANSIT VAULT PVC VOLUME
          command: ["sh", "-c", "chown -R 100:100 /vault-data"]
          volumeMounts:
            - name: data
              mountPath: /vault-data
            - name: config
              mountPath: /vault/config 
  # TRANSIT VAULT PVC SO DATA IS NEVER LOST
  volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          app: vault
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: cinder-standard
        resources:
          requests:
            storage: 1Gi

transit-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  # CONFIG NAME FOR TRANSIT VAULT
  name: transit-config
  namespace: pulsevault
  labels:
    name: transit-config
data:
  config.hcl: |
    storage "file" {
      path = "/vault-data"
    }
    listener "tcp" {
      address = "0.0.0.0:8200" 
      # VAULT CERTS FOR HTTPS
      tls_cert_file = "/opt/ca/vault.crt"
      tls_key_file = "/opt/ca/vault.key"
    }
    ui=true
    disable_mlock = true

Once you create a transit vault, log into it and generate the transit token which will be used to communicate with main vault cluster using VAULT_TOKEN environment variable in vault-cluster.yaml
Please follow this video to generate token for vault using transit vault and inject this token as an env variable VAULT_TOKEN in your vault-cluster stateful-set. Also when you generate token use below command to make it a periodic token with expiry of more than 1 day (I have it as 1 week)

vault token create -policy autounseal -period=168h

Step4: Auto-renew Transit Token

To autorenew transit token so that it never expires , we can use a kube cron job to achieve this.

autorenewcron.yaml (vault token renew $TRANSIT_TOKEN)

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: tokenrenew
  namespace: {namespace}
spec: 
  # EVERYDAY RUN AT 1AM
  schedule: "0 1 * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            run: tokenrenew-instance
        spec:
          volumes:
          # LOAD VAULT CERTS AND KEY
          - name: tls
            secret:
              secretName: vault-cert
              items:
              - key: tls.crt
                path: vault.crt
              - key: tls.key
                path: vault.key
          - name: transitsecrets
          containers:
          - name: vault
            image: docker.io/library/vault:1.4.1
            command: ["/bin/sh","-c"] 
            # LOGIN AND RENEW TRANSIT TOKEN
            args: ["vault login $ROOT_TOKEN > login.txt; vault token renew $TRANSIT_TOKEN > renew.txt"]
            env:
              # VAULT ADDRESS TO LOGIN WHICH WILL BE TRANSIT VAULT
              - name: VAULT_ADDR
                value: "https://vaulttransit.ns.local:8200" 
              # ROOT TOKEN OF TRANSIT VAULT LOADED AS SECRET
              - name: ROOT_TOKEN
                valueFrom:
                  secretKeyRef:
                    name: transit-secret
                    key: root-token 
              # PERIODIC TOKEN GENERATED USING VAULT TOKEN COMMAND
              - name: TRANSIT_TOKEN
                valueFrom:
                  secretKeyRef:
                    name: transit-secret
                    key: transit-token
            volumeMounts:
                - name: tls
                  mountPath: /etc/ssl/certs
                  readOnly: false
          restartPolicy: Never
      backoffLimit: 4

How does VAULT_TOKEN autounseal vault nodes??

When the vault cluster is initialized for first time using vault operator init, it will use the transit token or VAULT_TOKEN env variable to push all the unseal keys to transit vault using an api call with VAULT_TOKEN as authorization.

#-----------------snippet from stateful set----------------------
          env:
            - name: VAULT_RAFT_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: VAULT_ADDR
              value: 127.0.0.1:8200
            - name: VAULT_API_ADDR
              value: 127.0.0.1:8200
            - name: VAULT_CLUSTER_ADDR
              value: 127.0.0.1:8201
            - name: VAULT_TOKEN
              value: "put_your_transit_token_here"

Transit vault will then encrypt these unseal keys and send it back to the nodes in the cluster.
Now lets say a node in cluster goes down and it requires to unseal when it comes up, it just passes its encrypted unseal keys to transit vault and transit vault unseals this node
Thus with this the entire unsealing process is offloaded to transit vault. And once we initialize the cluster manually with vault operator init for first time, we will never have to worry about unsealing it manually again.
The communication between cluster and transit vault happens using VAULT_TOKEN
Hence to have no maintainence overhead, it is important for this token to autorenew itself.

Step5: Create Vault Headless Service and a Service

Headless svc ( this is optional but a good idea if you want your vault nodes to be discovered via a headless service)

kind: Service
apiVersion: v1
metadata:
  name: vaultraft
  labels:
    app: vaultraft
spec:
  selector:
    app: vaultraft
  clusterIP: None
  ports:
    - port: 8200
      name: rest
    - port: 8201
      name: inter-node

Service

apiVersion: v1
kind: Service
metadata:
  name: vaultsvc
  namespace: pulseraft
  labels:
    app: vaultsvc
spec:
  ports:
    - name: https
      port: 443
      targetPort: 8200
      protocol: TCP
  selector:
    app: vaultraft
  type: LoadBalancer

You will be able to access vault server at
https://vaultsvc.pulseraft.svc.local

Step6: Vault Operator Init

Once all above steps are done we will need to login to any of the vault nodes and initialize vault using

vault operator init

This is just one time process and will initialize the vault with credentials and store them in transit vault for subsequent unsealing or initializations
Note: This one time init process is needed, as the transit vault does not have the credentials needed to unseal our vault-cluster the very first time it comes up
But once we do this one time init process unseal secrets are stored in transit vault and all subsequent unsealing becomes automatic

Transit vault is just 1 node , So what if transit vault goes down??

We will manually write unseal keys to unseal.txt in our PVC /vault-data of transit vault
This is one time manual step. After which we will add a script to run as a post exec hook to unseal transit vault.
Put all unseal keys one per each line in /vault-data/unseal.txt
Put below script autounseal.sh in PVC in /vault-data

while IFS= read -r line || [ -n "$line" ]do   echo 're-creating username and password secret in namespace' $line   vault operator unseal $linedone < /vault-data/unseal.txt

The post exec command can then be put as a lifecycle hook in transit vault to run this autounseal.sh whenever the pod is initialized

#-----------------snippet from stateful set for transit vault-------containers:
        - name: vault
          image: docker.io/library/vault:1.4.1
          ports:
            - containerPort: 8200
              name: rest
              protocol: TCP
            - containerPort: 8201
              name: cluster
              protocol: TCP
          args:
            - "server"
          lifecycle:
            postStart:
              exec:
                command: ["/bin/sh", "-c", "sh /vault-data/autounseal.sh"]

After this one time manual setup, if transit vault goes down, it will autounseal itself using autounseal.sh

With this we have a 3 node vault HA cluster and a self healing system with no maintenance overhead !!