# Backups and syncs

This mostly has to do with `hydractl` commands such as
`sync-{backups,media,tpc}`.

## Servers

* Backups can be automatically generated and sent to other servers, including
  NAS instances.
* Automated backup checking can be done.
* But all this does not exempt a team from manually testing backups and to make
  their own offline copies in external drives.

## Parameters

Start by running the following commands in your terminal, adjusting to your
case:

    export HYDRA="hydra-name"
    export DISK="disk-name"
    export DEVICE="/dev/sdc"
    export USER="`whoami`"
    export DOMAIN="`facter domain`"

    if [ "$USER" != "root" ]; then
      export SUDO="sudo"
    fi

## New external drive

To add a new external drive into the pool, proceed as follows.

First make sure to create a passphrase for the new disk into your hydra's keyring:

    keyringer $HYDRA pwgen disks/$DISK.$DOMAIN/luks/root

If you're using a USB drive, run this before plugging it on the computer:

    hydractl usb-enable

With the disk plugged in, make a layout with single `${DEVICE}1` partition
(example: a disk with only one partition like `/dev/sdc1`):

    $SUDO cfdisk ${DEVICE}

Now create the LUKS encryption header in this new disk, using the passphrase
generated above by keyringer:

    $SUDO cryptsetup luksFormat ${DEVICE}1

Open the encrypted volume and create the basic filesystem structure needed
to sync content:

    $SUDO cryptsetup luksOpen ${DEVICE}1 $DISK
    $SUDO mkfs.ext4 /dev/mapper/$DISK
    $SUDO mkdir /media/$DISK
    $SUDO mount /dev/mapper/$DISK /media/$DISK
    $SUDO mkdir /media/$DISK/media.$DOMAIN
    $SUDO chown ${USER}: /media/$DISK/media.$DOMAIN

Finally, umount the drive:

    hydractl umount-media $DISK

## Regular sync

These commands should be enough to sync all media archives:

    hydractl usb-enable  # run this for USB drives, then connect the disk
    hydractl mount-media $DISK
    hydractl sync-media  $DISK

As this should handle syncing all backups:

    hydractl sync-backups $DISK

## TPC

Chek the [specific](tpc.md) documentation for more information about TPCs.

When regularly syncing a TPC, use a procedure that guarantees minimal
changes in the running TPC, to make sure syncing data won't create
inconsistencies in the destination, by doing something like:

1. Reboot the machine
2. Turn off networking.
3. Log in through console.

Then do the following:

    hydractl usb-enable  # run this for USB drives, then connect the disk
    hydractl mount-media $TPC
    hydractl sync-tpc    $TPC

If you don't want to do a full TPC sync, but just want to sync the home folders
(which is faster), use the following instead of the last command above:

    hydractl sync-home $TPC

To sync archives and remote backups, proceed with as explained in the NAS
section.

You may also want to consider using [borger][] ([mirror][]) to have encrypted
homedir backups in the external archive/backup volume.

[borger]: https://git.fluxo.info/borger/about/
[mirror]: https://0xacab.org/rhatto/borger

## Appliances

You might want to backup the whole SSD, M-SATA or microSD from your appliances.
If so, proceed as follows with the appliance device connected in your TPC:

    export APPLIANCE="appliance-name"
    export DEST="/var/backups/remote/${APPLIANCE}.${DOMAIN}/image/`date +%Y%m%d`"
    $SUDO mkdir -p $dest
    dcfldd if=/dev/sdb | bzip2 > $DEST/${APPLIANCE}.img.bz2

## Smartphone

Smartphones usually have their own way to be backed up. This is an example
based on the [android-backup][] utility:

    android-backup <device-name>

[android-backup]: https://git.fluxo.info/scripts/tree/android-backup

## Hardware rotation

It's recommended to rotate the current hardware in use in all
places:

* Backup disks.
* Laptops, so they're not kept out of use (and/or especially the batteries).

## Backup Kit

A Backup Kit is a box with the following items:

* External encrypted archive/backup disk.
* Case for SSD transportation.
* Laptop power adapter and cables.
* Dockstation SATA/USB/M.2/microSD/etc (with power adapter -- usually 12V).
* Power adapter for external hard drives (usually 12V, and might be compatible
  with the dockstationadapter, so you might carry just one).
* USB power adapter and cable (including USB 2, USB 3 and USB C).
* USB cables (USB 2, USB 3 and USB C) for the Dockstation and the external drive.
* TPC laptop with battery and TPS (SSD, M.2 etc), optionally with a UltraBase/Dockstation.
* Philips screwdriver and other tools.
* FCR-MG2 adaptor for microSD to USB.
* Anything else your need (like eyeglasses).
* Everything in a sealed box.

This may be the ultimate disaster recovery kit for your Hydra!

## Restore

Examples according to the software used to make the backup.

### Duplicity

For [duplicity][]:

    duplicity collection-status file:///var/backups/duplicity
    duplicity restore --file-to-restore home/$USER --time 2018-03-25 file:///var/backups/duplicity/ /home/$USER

[duplicity]: https://duplicity.gitlab.io/

### Borg

For [Borg][]:

    export PATH_TO_RESTORE="path/to/be/restored"
    mkdir ~/temp/misc/restore
    cd    ~/temp/misc/restore
    borg list    ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg
    borg extract ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg::$USER-2018-06-11T17:07:39 mnt/crypt/home/$USER/$PATH_TO_RESTORE

Make sure to cleanup `~/temp/misc/restore` after recovering what you need.

Note on backup keys:

* In the past (before 2024), the Hydra Suite and it's companion [Puppet][]
  modules used pre-generated [Borg][] repository keys for the sake of automation.
  This is [not possible anymore][].
* As it's [important to keep copies of the borg repository key safely
  elsewhere][], the managed configuration supports OpenPGP-encrypting the
  repository key and uploading it to the remote repository.
* This OpenPGP-encrypted key file is named as `keyfile.asc` and is uploaded
  in the root folder of the remote repository.
* This OpenPGP-encrypted key file is encrypted and signed with a provided
  OpenPGP keypair and passphrase (convention is to use the machines's OpenPGP
  general purpose key, or the machine's role key).
* This allows the operators to fetch this encrypted keyfile and use their copy
  of the machine's OpenPGP key to extract the passphrase _on their
  encrypted-storage workstations_ (recommendation is to not do this on the remote
  repository).

[Borg]: https://www.borgbackup.org/
[Puppet]: https://www.puppet.com/
[not possible anymore]: https://github.com/borgbackup/borg/issues/7047
[important to keep copies of the borg repository key safely elsewhere]: https://borgbackup.readthedocs.io/en/latest/faq.html#how-important-is-the-home-config-borg-directory

### eCryptfs

For [eCryptfs][]:

    $SUDO ecryptfs-recover-private /media/$DISK/home/.ecryptfs/$USER/.Private

[eCryptfs]: https://www.ecryptfs.org/