1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
|
# Backups and syncs
This mostly has to do with `hydractl` commands such as
`sync-{backups,media,tpc}`.
## Servers
* Backups can be automatically generated and sent to other servers, including
NAS instances.
* Automated backup checking can be done.
* But all this does not exempt a team from manually testing backups and to make
their own offline copies in external drives.
## New external drive
Proceed as follows
disk=new-disk-name
device=/dev/sdc
hydractl usb-enable # run this for USB drives, then connect the disk
sudo cfdisk ${device} # layout with single ${device}1 partition
sudo cryptsetup luksFormat ${device}1
sudo cryptsetup luksOpen ${device}1 $disk
sudo mkfs.ext4 /dev/mapper/$disk
sudo mkdir /media/$disk
sudo mount /dev/mapper/$disk /media/$disk
sudo mkdir /media/$disk/media.`facter domain`
sudo chown `whoami`: /media/$disk/media.`facter domain`
## NAS
These commands should be enough to sync all media archives:
hydractl usb-enable # run this for USB drives, then connect the disk
hydractl mount-media $MEDIA
hydractl sync-media $MEDIA
As this should handle syncing all backups:
hydractl sync-backups $MEDIA
## TPC
Chek the [specific](tpc.md) documentation for more information about TPCs.
When regularly syncing a TPC, use a procedure that guarantees minimal
changes in the running TPC, to make sure syncing data won't create
inconsistencies in the destination, by doing something like:
1. Reboot the machine
2. Turn off networking.
3. Log in through console.
Then do the following:
hydractl usb-enable # run this for USB drives, then connect the disk
hydractl mount-media $TPC
hydractl sync-tpc $TPC
To sync archives and remote backups, proceed with as explained in the NAS
section.
You may also want to consider using [borger][] ([mirror][]) to have encrypted
homedir backups in the external archive/backup volume.
[borger]: https://git.fluxo.info/borger/about/
[mirror]: https://0xacab.org/rhatto/borger
## Appliances
You might want to backup the whole SSD, M-SATA or microSD from your appliances.
If so, proceed as follows with the appliance device connected in your TPC:
export appliance=appliance-name
export dest="/var/backups/remote/$appliance.`facter domain`/image/`date +%Y%m%d`"
sudo mkdir -p $dest
dcfldd if=/dev/sdb | bzip2 > $dest/$appliance.img.bz2
## Smartphone
Smartphones usually have their own way to be backed up. This is an example
based on the [android-backup][] utility:
android-backup <device-name>
[android-backup]: https://git.fluxo.info/scripts/tree/android-backup
## Hardware rotation
It's recommended to rotate the current hardware in use in all
places:
* Backup disks.
* Laptops, so they're not kept out of use (and/or especially the batteries).
## Backup Kit
A Backup Kit is a box with the following items:
* External encrypted archive/backup disk.
* Case for SSD transportation.
* Laptop power adapter and cables.
* Dockstation SATA/USB/M.2/microSD/etc (with power adapter).
* USB power adapter and cable (including USB 2, USB 3 and USB C).
* USB cables (USB 2, USB 3 and USB C) for the Dockstation and the external drive.
* TPC laptop with battery and TPS (SSD, M.2 etc), optionally with a UltraBase/Dockstation.
* Philips screwdriver and other tools.
* FCR-MG2 adaptor for microSD to USB.
* Anything else your need (like eyeglasses).
* Everything in a sealed box.
This may be the ultimate disaster recovery kit for your Hydra!
## Restore
Examples according to the software used to make the backup.
### Duplicity
For [duplicity][]:
duplicity collection-status file:///var/backups/duplicity
duplicity restore --file-to-restore home/$USER --time 2018-03-25 file:///var/backups/duplicity/ /home/$USER
[duplicity]: https://duplicity.gitlab.io/
### Borg
For [Borg][]:
mkdir ~/temp/misc/restore
cd ~/temp/misc/restore
borg list ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg
borg extract ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg::$USER-2018-06-11T17:07:39 mnt/crypt/home/$USER/$FILE_OR_FOLDER
Make sure to cleanup `~/temp/misc/restore` after recovering what you need.
Note on backup keys:
* In the past (before 2024), the Hydra Suite and it's companion [Puppet][]
modules used pre-generated [Borg][] repository keys for the sake of automation.
This is [not possible anymore][].
* As it's [important to keep copies of the borg repository key safely
elsewhere][], the managed configuration supports OpenPGP-encrypting the
repository key and uploading it to the remote repository.
* This OpenPGP-encrypted key file is named as `keyfile.asc` and is uploaded
in the root folder of the remote repository.
* This OpenPGP-encrypted key file is encrypted and signed with a provided
OpenPGP keypair and passphrase (convention is to use the machines's OpenPGP
general purpose key, or the machine's role key).
* This allows the operators to fetch this encrypted keyfile and use their copy
of the machine's OpenPGP key to extract the passphrase _on their
encrypted-storage workstations_ (recommendation is to not do this on the remote
repository).
[Borg]: https://www.borgbackup.org/
[Puppet]: https://www.puppet.com/
[not possible anymore]: https://github.com/borgbackup/borg/issues/7047
[important to keep copies of the borg repository key safely elsewhere]: https://borgbackup.readthedocs.io/en/latest/faq.html#how-important-is-the-home-config-borg-directory
### eCryptfs
For [eCryptfs][]:
sudo ecryptfs-recover-private /media/$MEDIA/home/.ecryptfs/$USER/.Private
[eCryptfs]: https://www.ecryptfs.org/
|
