Disaster-Recovery

This page describes several disaster recovery topics such as getting access to the hosts as backup administrator or retrieving a backup.

With push access to the git repository you may gain access to the hosts via ssh. This is especially targeted for backup administrators to recovery access if I’m not available. The deployment bot will take new commits and deploy them on my behalf.

Note
If the machine is currently inside a LUKS decryption shell you’ll need the LUKS decryption passphrase, since this is not recoverable by automation.

If the machine is booted and accepts deployments the process is as follows:

# override the file `group_vars/all` and set your ssh key
cat > group_vars/all<< EOF
sys_users:
  - name: recovery
    password: some-secret
    groups:
      - wheel
ssh_authorized_keys:
  - user: recovery
    key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEr0AItSOuH99cZn9MA2J8OiH4z6YmSdSYgtbzoJtPyt
    owner: recovery
    comment: recovery
EOF
# push the commit with a playbook option
git push -o ci.variable=PLAYBOOK=recovery.yml

You’ll now be able to login via ssh recovery@bench... and recover the sh** out of this infrastructure. Thanks in advance.

On the server in question use Podman and the duplicity container to restore a backup.

Note
You’ll require backup source authentication.

docker run --rm --security-opt "label=disable" -it -v "/root/.gnupg/:/root/.gnupg/" -v "/root/.ssh/:/root/.ssh/" -v "/var/srv:/var/srv" quay.io/sheogorath/duplicity:0.8 duplicity restore --encrypt-sign-key "<key-id>" -t "<time for restore>" "<backup source>/<hostname>" /var/srv/.restore

The .gnupg mount option ensures that the GPG key is available inside the container. The .ssh should contain the SSH host key of the backup source. The /var/srv directory is in this case the target on the host with enough storage for a restoration.