How to recover a destroyed deployment

Overview

Let’s consider a situation where we had an ArangoDeployment in Cluster mode (3 DbServers, 3 Coordinators, 3 Agents) with Local storage attached (only one K8s Node in the K8s cluster).

Due to some reason the ArangoDeployment was deleted (e.g. ETCD storage has been wiped out) and we want to recover it. Fortunately, we have a backup of the data on the disk.

To recover the ArangoDeployment we need to:

1. Create PV and PVC for each member with persistent storage (agent, dbservers, single)
2. Create a new ArangoDeployment with the same members IDs

Local storage data

We have a members (Agents & DbServers) data in the following directories:

> ls -1 /var/data/
f9rs2htwc9e0bzme
fepwdnnbf0keylgx
gqnkahucthoaityt
vka6ic19qcl1y3ec
rhlf8vixbsbewefo
rlzl467vfgsdpofu

To find out the name of the members to which data should be attached, we need to check the UUID file content in each directory:

> cat /var/data/*/UUID
AGNT-pntg5yc8
AGNT-kfyuj8ow
AGNT-bv5rofcz
PRMR-9xztmg4t
PRMR-l1pp19yl
PRMR-31akmzrp

Initial ArangoDeployment

Here is an example of the initial ArangoDeployment before deletion:

cat <<EOF | kubectl apply -f -
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  externalAccess:
    type: NodePort
  mode: Cluster
  agents:
    volumeClaimTemplate:
      spec:
        storageClassName: my-local-ssd
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  dbservers:
    volumeClaimTemplate:
      spec:
        storageClassName: my-local-ssd
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
EOF

Create PV and PVC

1. We need to create ArangoLocalStorage first:

    cat <<EOF | kubectl apply -f -
    apiVersion: "storage.arangodb.com/v1alpha"
    kind: "ArangoLocalStorage"
    metadata:
      name: "local-storage"
    spec:
      storageClass:
        name: my-local-ssd
        isDefault: true
      localPath:
      - /mnt/data
    EOF
   

2. Now create PV and PVC for every directory listed above

   • Agents - here is an example for AGNT-pntg5yc8(f9rs2htwc9e0bzme directory)

     • PV

        cat <<EOF | kubectl apply -f -
        apiVersion: "v1"
        kind: PersistentVolume
        metadata:
          labels:
            arango_deployment: cluster
            role: agent
          name: agent-pntg5yc8-f9rs2htwc9e0bzme
        spec:
          accessModes:
          - ReadWriteOnce
          capacity:
            storage: 1Gi
          local:
            path: /mnt/data/f9rs2htwc9e0bzme
          persistentVolumeReclaimPolicy: Retain
          storageClassName: my-local-ssd
          volumeMode: Filesystem
          nodeAffinity:
            required:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - minikube
        EOF
       

     • PVC

         cat <<EOF | kubectl apply -f -
         apiVersion: v1
         kind: PersistentVolumeClaim
         metadata:
           labels:
             app: arangodb
             arango_deployment: cluster
             role: agent
           name: agent-pntg5yc8
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 1Gi
           storageClassName: my-local-ssd
           volumeMode: Filesystem
           volumeName: agent-pntg5yc8-f9rs2htwc9e0bzme
         EOF
       

   • DbServers - here is an example for PRMR-9xztmg4t (vka6ic19qcl1y3ec directory)

     • PV

        cat <<EOF | kubectl apply -f -
        apiVersion: "v1"
        kind: PersistentVolume
        metadata:
          labels:
            arango_deployment: cluster
            role: dbserver
          name: dbserver-9xztmg4t-vka6ic19qcl1y3ec
        spec:
          accessModes:
          - ReadWriteOnce
          capacity:
            storage: 1Gi
          local:
            path: /mnt/data/vka6ic19qcl1y3ec
          persistentVolumeReclaimPolicy: Retain
          storageClassName: my-local-ssd
          volumeMode: Filesystem
          nodeAffinity:
            required:
              nodeSelectorTerms:
              - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - minikube
        EOF
       

     • PVC

         cat <<EOF | kubectl apply -f -
         apiVersion: v1
         kind: PersistentVolumeClaim
         metadata:
           labels:
             app: arangodb
             arango_deployment: cluster
             role: dbserver
           name: dbserver-9xztmg4t
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 1Gi
           storageClassName: my-local-ssd
           volumeMode: Filesystem
           volumeName: dbserver-9xztmg4t-vka6ic19qcl1y3ec
         EOF
       

Create ArangoDeployment with previously created PVC

Now we can create ArangoDeployment with previously created PVCs:

cat <<EOF | kubectl apply -f -
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "cluster"
spec:
  externalAccess:
    type: NodePort
  mode: Cluster
  agents:
    volumeClaimTemplate:
      spec:
        storageClassName: my-local-ssd
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
  dbservers:
    volumeClaimTemplate:
      spec:
        storageClassName: my-local-ssd
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem
status:
  agency:
    ids:
      - AGNT-pntg5yc8
      - AGNT-kfyuj8ow
      - AGNT-bv5rofcz
    size: 3
  members:
    agents:
      - id: AGNT-pntg5yc8
        persistentVolumeClaim:
          name: agent-pntg5yc8
        persistentVolumeClaimName: agent-pntg5yc8
      - id: AGNT-kfyuj8ow
        persistentVolumeClaim:
          name: agent-kfyuj8ow
        persistentVolumeClaimName: agent-kfyuj8ow
      - id: AGNT-bv5rofcz
        persistentVolumeClaim:
          name: agent-bv5rofcz
        persistentVolumeClaimName: agent-bv5rofcz
    dbservers:
      - id: PRMR-9xztmg4t
        persistentVolumeClaim:
          name: cluster-dbserver-9xztmg4t
        persistentVolumeClaimName: cluster-dbserver-9xztmg4t
      - id: PRMR-l1pp19yl
        persistentVolumeClaim:
          name: cluster-dbserver-l1pp19yl
        persistentVolumeClaimName: cluster-dbserver-l1pp19yl
      - id: PRMR-31akmzrp
        persistentVolumeClaim:
          name: cluster-dbserver-31akmzrp
        persistentVolumeClaimName: cluster-dbserver-31akmzrp
  EOF

That’s it! Now you can use ArangoDB with restored data.