DTR backups and recovery

DTR requires that a majority (n/2 + 1) of its replicas are healthy at all times for it to work. So if a majority of replicas is unhealthy or lost, the only way to restore DTR to a working state, is by recovering from a backup. This is why it’s important to ensure replicas are healthy and perform frequent backups.

Data managed by DTR

Docker Trusted Registry maintains data about:

This data is persisted on the host running DTR, using named volumes. Learn more about DTR named volumes.

To perform a backup of a DTR node, run the docker/dtr backup command. This command backups up the following data:

Backup DTR data

To create a backup of DTR you need to:

1. Backup image content
2. Backup DTR metadata

You should always create backups from the same DTR replica, to ensure a smoother restore.

Backup image content

Since you can configure the storage backend that DTR uses to store images, the way you backup images depends on the storage backend you’re using.

If you’ve configured DTR to store images on the local filesystem or NFS mount, you can backup the images by using ssh to log into a node where DTR is running, and creating a tar archive of the dtr-registry volume:

sudo tar -cf backup-images.tar \
  $(dirname $(docker volume inspect --format '{{.Mountpoint}}' dtr-registry-<replica-id>))

If you’re using a different storage backend, follow the best practices recommended for that system.

Backup DTR metadata

To create a DTR backup, load your UCP client bundle, and run the following command, replacing the placeholders for the real values:

read -sp 'ucp password: ' UCP_PASSWORD; \
docker run --log-driver none -i --rm \
  --env UCP_PASSWORD=$UCP_PASSWORD \
  docker/dtr:2.3.11 backup \
  --ucp-url <ucp-url> \
  --ucp-insecure-tls \
  --ucp-username <ucp-username> \
  --existing-replica-id <replica-id> > backup-metadata.tar

Where:

• <ucp-url> is the url you use to access UCP
• <ucp-username> is the username of a UCP administrator
• <replica-id> is the id of the DTR replica to backup

This prompts you for the UCP password, backups up the DTR metadata and saves the result into a tar archive. You can learn more about the supported flags in the reference documentation.

The backup command doesn’t stop DTR, so that you can take frequent backups without affecting your users. Also, the backup contains sensitive information like private keys, so you can encrypt the backup by running:

gpg --symmetric 

This prompts you for a password to encrypt the backup, copies the backup file and encrypts it.

Test your backups

To validate that the backup was correctly performed, you can print the contents of the tar file created. The backup of the images should look like:

tar -tf backup-images.tar

dtr-backup-v2.3.11/
dtr-backup-v2.3.11/rethink/
dtr-backup-v2.3.11/rethink/layers/

And the backup of the DTR metadata should look like:

tar -tf 

# The archive should look like this
dtr-backup-v2.3.11/
dtr-backup-v2.3.11/rethink/
dtr-backup-v2.3.11/rethink/properties/
dtr-backup-v2.3.11/rethink/properties/0

If you’ve encrypted the metadata backup, you can use:

gpg -d /tmp/backup.tar.gpg | tar -t

You can also create a backup of a UCP cluster and restore it into a new cluster. Then restore DTR on that new cluster to confirm that everything is working as expected.

Restore DTR data

If your DTR has a majority of unhealthy replicas, the one way to restore it to a working state is by restoring from an existing backup.

To restore DTR, you need to:

1. Stop any DTR containers that might be running
2. Restore the images from a backup
3. Restore DTR metadata from a backup
4. Re-fetch the vulnerability database

You need to restore DTR on the same UCP cluster where you’ve created the backup. If you restore on a different UCP cluster, all DTR resources will be owned by users that don’t exist, so you can’t manage the resources, even though they’re stored in the DTR data store.

When restoring, you need to use the same version of the docker/dtr image that you’ve used when creating the update. Other versions are not guaranteed to work.

Stop DTR containers

Start by removing any DTR container that is still running:

docker run -it --rm \
  docker/dtr:2.3.11 destroy \
  --ucp-insecure-tls

Restore images

If you had DTR configured to store images on the local filesystem, you can extract your backup:

sudo tar -xzf backup-images.tar -C /var/lib/docker/volumes

If you’re using a different storage backend, follow the best practices recommended for that system. When restoring the DTR metadata, DTR will be deployed with the same configurations it had when creating the backup.

Restore DTR metadata

You can restore the DTR metadata with the docker/dtr restore command. This performs a fresh installation of DTR, and reconfigures it with the configuration created during a backup.

Load your UCP client bundle, and run the following command, replacing the placeholders for the real values:

read -sp 'ucp password: ' UCP_PASSWORD; \
docker run -i --rm \
  --env UCP_PASSWORD=$UCP_PASSWORD \
  docker/dtr:2.3.11 restore \
  --ucp-url <ucp-url> \
  --ucp-insecure-tls \
  --ucp-username <ucp-username> \
  --ucp-node <hostname> \
  --replica-id <replica-id> \
  --dtr-external-url <dtr-external-url> < backup-metadata.tar

Where:

• <ucp-url> is the url you use to access UCP
• <ucp-username> is the username of a UCP administrator
• <hostname> is the hostname of the node where you’ve restored the images
• <replica-id> the id of the replica you backed up
• <dtr-external-url>the url that clients use to access DTR

Re-fetch the vulnerability database

If you’re scanning images, you now need to download the vulnerability database.

After you successfully restore DTR, you can join new replicas the same way you would after a fresh installation. Learn more.

Where to go next

• Set up high availability
• DTR architecture