# Backups and syncs

This mostly has to do with `hydractl` commands such as
`sync-{backups,media,tpc}`.

## Servers

* Backups can be automatically generated and sent to other servers, including
  NAS instances.
* Automated backup checking can be done.
* But all this does not exempt a team from manually testing backups and to make
  their own offline copies in external drives.

## Parameters

Start by running the following commands in your terminal, adjusting to your
case:

    export HYDRA="hydra-name"
    export DISK="disk-name"
    export DEVICE="/dev/sdc"
    export USER="`whoami`"
    export DOMAIN="`facter domain`"

    if [ "$USER" != "root" ]; then
      export SUDO="sudo"
    fi

## New external drive

To add a new external drive into the pool, proceed as follows.

First make sure to create a passphrase for the new disk into your hydra's keyring:

    keyringer $HYDRA pwgen disks/$DISK.$DOMAIN/luks/root

If you're using a USB drive, run this before plugging it on the computer:

    hydractl usb-enable

With the disk plugged in, make a layout with single `${DEVICE}1` partition
(example: a disk with only one partition like `/dev/sdc1`):

    $SUDO cfdisk ${DEVICE}

Now create the LUKS encryption header in this new disk, using the passphrase
generated above by keyringer:

    $SUDO cryptsetup luksFormat ${DEVICE}1

If you want to make this volume to have a label (helpful for detecting the disk
etc; downside: third parties can easily get the name), run this command:

    $SUDO cryptsetup config ${DEVICE}1 --label ${DISK}

Open the encrypted volume and create the basic filesystem structure needed
to sync content:

    $SUDO cryptsetup luksOpen ${DEVICE}1 $DISK
    $SUDO mkfs.ext4 /dev/mapper/$DISK
    $SUDO mkdir /media/$DISK
    $SUDO mount /dev/mapper/$DISK /media/$DISK
    $SUDO mkdir /media/$DISK/media.$DOMAIN
    $SUDO chown ${USER}: /media/$DISK/media.$DOMAIN

Finally, umount the drive:

    hydractl umount-media $DISK

## Regular sync

These commands should be enough to sync all media archives:

    hydractl usb-enable  # run this for USB drives, then connect the disk
    hydractl mount-media $DISK
    hydractl sync-media  $DISK

As this should handle syncing all backups:

    hydractl sync-backups $DISK

## TPC

Chek the [specific](tpc.md) documentation for more information about TPCs.

When regularly syncing a TPC, use a procedure that guarantees minimal
changes in the running TPC, to make sure syncing data won't create
inconsistencies in the destination, by doing something like:

1. Reboot the machine
2. Turn off networking.
3. Log in through console.

Then do the following:

    hydractl usb-enable  # run this for USB drives, then connect the disk
    hydractl mount-media $TPC
    hydractl sync-tpc    $TPC

If you don't want to do a full TPC sync, but just want to sync the home folders
(which is faster), use the following instead of the last command above:

    hydractl sync-home $TPC

To sync archives and remote backups, proceed with as explained in the NAS
section.

You may also want to consider using [borger][] ([mirror][]) to have encrypted
homedir backups in the external archive/backup volume.

[borger]: https://git.fluxo.info/borger/about/
[mirror]: https://0xacab.org/rhatto/borger

## Appliances

You might want to backup the whole SSD, M-SATA or microSD from your appliances.
If so, proceed as follows with the appliance device connected in your TPC:

    export APPLIANCE="appliance-name"
    export DEST="/var/backups/remote/${APPLIANCE}.${DOMAIN}/image/`date +%Y%m%d`"
    $SUDO mkdir -p $dest
    dcfldd if=/dev/sdb | bzip2 > $DEST/${APPLIANCE}.img.bz2

## Smartphone

Smartphones usually have their own way to be backed up. This is an example
based on the [android-backup][] utility:

    android-backup <device-name>

[android-backup]: https://git.fluxo.info/scripts/tree/android-backup

## Hardware rotation

It's recommended to rotate the current hardware in use in all
places:

* Backup disks.
* Laptops, so they're not kept out of use (and/or especially the batteries).

## Backup Kit

A Backup Kit is a box with the following items:

* External encrypted archive/backup disk.
* Case for SSD transportation.
* Laptop power adapter and cables.
* Dockstation SATA/USB/M.2/microSD/etc (with power adapter -- usually 12V).
* Power adapter for external hard drives (usually 12V, and might be compatible
  with the dockstationadapter, so you might carry just one).
* USB power adapter and cable (including USB 2, USB 3 and USB C).
* USB cables (USB 2, USB 3 and USB C) for the Dockstation and the external drive.
* TPC laptop with battery and TPS (SSD, M.2 etc), optionally with a UltraBase/Dockstation.
* Philips screwdriver and other tools.
* FCR-MG2 adaptor for microSD to USB.
* Anything else your need (like eyeglasses).
* Everything in a sealed box.

This may be the ultimate disaster recovery kit for your Hydra!

## Restore

Having backup data leaked is a serious security issue, and that's why we
encrypt backups.
But losing access to the encrypted material is data loss, so it's important
to make sure in advance that we can get back the material.

Procedures vary, and the following examples are sorted according to the
software used to make the backup.

### Duplicity

For [duplicity][]:

    duplicity collection-status file:///var/backups/duplicity
    duplicity restore --file-to-restore home/$USER --time 2018-03-25 file:///var/backups/duplicity/ /home/$USER

[duplicity]: https://duplicity.gitlab.io/

### Borg

For [Borg][]:

    export PATH_TO_RESTORE="path/to/be/restored"
    mkdir ~/temp/misc/restore
    cd    ~/temp/misc/restore
    borg list    ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg
    borg extract ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg::$USER-2018-06-11T17:07:39 mnt/crypt/home/$USER/$PATH_TO_RESTORE

Make sure to cleanup `~/temp/misc/restore` after recovering what you need.

Note on backup keys:

* In the past (before 2024), the Hydra Suite and it's companion [Puppet][]
  modules used pre-generated [Borg][] repository keys for the sake of automation.
  This is [not possible anymore][].
* As it's [important to keep copies of the borg repository key safely
  elsewhere][], the managed configuration supports OpenPGP-encrypting the
  repository key and uploading it to the remote repository.
* This OpenPGP-encrypted key file is named as `keyfile.asc` and is uploaded
  in the root folder of the remote repository.
* This OpenPGP-encrypted key file is encrypted and signed with a provided
  OpenPGP keypair and passphrase (convention is to use the machines's OpenPGP
  general purpose key, or the machine's role key).
* This allows the operators to fetch this encrypted keyfile and use their copy
  of the machine's OpenPGP key to extract the passphrase _on their
  encrypted-storage workstations_ (recommendation is to not do this on the remote
  repository).

Just to be sure, let's emphasize Borg's own recommendation:

> IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
>
> If you used a repokey mode, the key is stored in the repo, but you should
> back it up separately.
> Use "borg key export" to export the key, optionally in printable format.
> Write down the passphrase. Store both at safe place(s).

[Borg]: https://www.borgbackup.org/
[Puppet]: https://www.puppet.com/
[not possible anymore]: https://github.com/borgbackup/borg/issues/7047
[important to keep copies of the borg repository key safely elsewhere]: https://borgbackup.readthedocs.io/en/latest/faq.html#how-important-is-the-home-config-borg-directory

### eCryptfs

For [eCryptfs][]:

    $SUDO ecryptfs-recover-private /media/$DISK/home/.ecryptfs/$USER/.Private

[eCryptfs]: https://www.ecryptfs.org/