Contious-Integration and Deployment

You could say at the heart of the infrastructure is the CI/CD bot / mechanism that pumps the configuration through the machines.

Note
Only CGroupsv2 supports unprivileged runner containers

While you may find the role that deploys the docker-compose file inside the repository, the minified version looks as follows:

version: '2.2'
services:
  runner:
    image: "quay.io/shivering-isles/gitlab-ansible-runner:latest"
    security-opt:
      - "label=disable"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "gitlab-runner-data:/etc/gitlab-runner"
      - "gitlab-runner-ssh:/home/gitlab-runner/.ssh"
      - "gitlab-runner-machine:/root/.docker/machine"
    restart: always

Beware
This is a minimal, rather insecure version with full access to the docker socket similar to root access.

The corresponding gitlab-runner configuration for a docker executor looks as follows:

[[runners]]
  name = "infrastructure-runner"
  url = "https://git.shivering-isles.com"
  token = "secret-token"
  limit = 1
  environment = ["CI_VAULT_PASS=some-secret","CI_PRIVATE_KEY_FILE=/path/to/private.key"]
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker.io/library/fedora:35"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache","/path/to/host/.ssh:/root/.ssh"]
    shm_size = 0
  [runners.cache]

The important bits here are the environment variables CI_VAULT_PASS and CI_PRIVATE_KEY_FILE which are made available in the container executing the Ansible playbook run. The rollout uses those variables to decrypt the Ansible vault and to locate the key file for ssh access. The public key to this private key is already deployed on all hosts for the user ansible. The .ssh directory contains a known_hosts file with all SSH public keys of the hosts the rollout has to connect to.

Note
An aspect of security is that the ansible-vault containing the actual secrets is stored at the GitLab instance while the key to decrypt said vault is kept on a different machine. Both parts only come together when a rollout is under way.