1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
|
# Backups and syncs
This mostly has to do with `hydractl` commands such as
`sync-{backups,media,tpc}`.
## Servers
* Backups can be automatically generated and sent to other servers, including
NAS instances.
* Automated backup checking can be done.
* But all this does not exempt a team from manually testing backups and to make
their own offline copies in external drives.
## Parameters
Start by running the following commands in your terminal, adjusting to your
case:
export HYDRA="hydra-name"
export DISK="disk-name"
export DEVICE="/dev/sdc"
export USER="`whoami`"
export DOMAIN="`facter domain`"
if [ "$USER" != "root" ]; then
export SUDO="sudo"
fi
## New external drive
To add a new external drive into the pool, proceed as follows.
First make sure to create a passphrase for the new disk into your hydra's keyring:
keyringer $HYDRA pwgen disks/$DISK.$DOMAIN/luks/root
If you're using a USB drive, run this before plugging it on the computer:
hydractl usb-enable
With the disk plugged in, make a layout with single `${DEVICE}1` partition
(example: a disk with only one partition like `/dev/sdc1`):
$SUDO cfdisk ${DEVICE}
Now create the LUKS encryption header in this new disk, using the passphrase
generated above by keyringer:
$SUDO cryptsetup luksFormat ${DEVICE}1
Open the encrypted volume and create the basic filesystem structure needed
to sync content:
$SUDO cryptsetup luksOpen ${DEVICE}1 $DISK
$SUDO mkfs.ext4 /dev/mapper/$DISK
$SUDO mkdir /media/$DISK
$SUDO mount /dev/mapper/$DISK /media/$DISK
$SUDO mkdir /media/$DISK/media.$DOMAIN
$SUDO chown ${USER}: /media/$DISK/media.$DOMAIN
Finally, umount the drive:
hydractl umount-media $DISK
## Regular sync
These commands should be enough to sync all media archives:
hydractl usb-enable # run this for USB drives, then connect the disk
hydractl mount-media $DISK
hydractl sync-media $DISK
As this should handle syncing all backups:
hydractl sync-backups $DISK
## TPC
Chek the [specific](tpc.md) documentation for more information about TPCs.
When regularly syncing a TPC, use a procedure that guarantees minimal
changes in the running TPC, to make sure syncing data won't create
inconsistencies in the destination, by doing something like:
1. Reboot the machine
2. Turn off networking.
3. Log in through console.
Then do the following:
hydractl usb-enable # run this for USB drives, then connect the disk
hydractl mount-media $TPC
hydractl sync-tpc $TPC
If you don't want to do a full TPC sync, but just want to sync the home folders
(which is faster), use the following instead of the last command above:
hydractl sync-home $TPC
To sync archives and remote backups, proceed with as explained in the NAS
section.
You may also want to consider using [borger][] ([mirror][]) to have encrypted
homedir backups in the external archive/backup volume.
[borger]: https://git.fluxo.info/borger/about/
[mirror]: https://0xacab.org/rhatto/borger
## Appliances
You might want to backup the whole SSD, M-SATA or microSD from your appliances.
If so, proceed as follows with the appliance device connected in your TPC:
export APPLIANCE="appliance-name"
export DEST="/var/backups/remote/${APPLIANCE}.${DOMAIN}/image/`date +%Y%m%d`"
$SUDO mkdir -p $dest
dcfldd if=/dev/sdb | bzip2 > $DEST/${APPLIANCE}.img.bz2
## Smartphone
Smartphones usually have their own way to be backed up. This is an example
based on the [android-backup][] utility:
android-backup <device-name>
[android-backup]: https://git.fluxo.info/scripts/tree/android-backup
## Hardware rotation
It's recommended to rotate the current hardware in use in all
places:
* Backup disks.
* Laptops, so they're not kept out of use (and/or especially the batteries).
## Backup Kit
A Backup Kit is a box with the following items:
* External encrypted archive/backup disk.
* Case for SSD transportation.
* Laptop power adapter and cables.
* Dockstation SATA/USB/M.2/microSD/etc (with power adapter -- usually 12V).
* Power adapter for external hard drives (usually 12V, and might be compatible
with the dockstationadapter, so you might carry just one).
* USB power adapter and cable (including USB 2, USB 3 and USB C).
* USB cables (USB 2, USB 3 and USB C) for the Dockstation and the external drive.
* TPC laptop with battery and TPS (SSD, M.2 etc), optionally with a UltraBase/Dockstation.
* Philips screwdriver and other tools.
* FCR-MG2 adaptor for microSD to USB.
* Anything else your need (like eyeglasses).
* Everything in a sealed box.
This may be the ultimate disaster recovery kit for your Hydra!
## Restore
Having backup data leaked is a serious security issue, and that's why we
encrypt backups.
But losing access to the encrypted material is data loss, so it's important
to make sure in advance that we can get back the material.
Procedures vary, and the following examples are sorted according to the
software used to make the backup.
### Duplicity
For [duplicity][]:
duplicity collection-status file:///var/backups/duplicity
duplicity restore --file-to-restore home/$USER --time 2018-03-25 file:///var/backups/duplicity/ /home/$USER
[duplicity]: https://duplicity.gitlab.io/
### Borg
For [Borg][]:
export PATH_TO_RESTORE="path/to/be/restored"
mkdir ~/temp/misc/restore
cd ~/temp/misc/restore
borg list ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg
borg extract ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg::$USER-2018-06-11T17:07:39 mnt/crypt/home/$USER/$PATH_TO_RESTORE
Make sure to cleanup `~/temp/misc/restore` after recovering what you need.
Note on backup keys:
* In the past (before 2024), the Hydra Suite and it's companion [Puppet][]
modules used pre-generated [Borg][] repository keys for the sake of automation.
This is [not possible anymore][].
* As it's [important to keep copies of the borg repository key safely
elsewhere][], the managed configuration supports OpenPGP-encrypting the
repository key and uploading it to the remote repository.
* This OpenPGP-encrypted key file is named as `keyfile.asc` and is uploaded
in the root folder of the remote repository.
* This OpenPGP-encrypted key file is encrypted and signed with a provided
OpenPGP keypair and passphrase (convention is to use the machines's OpenPGP
general purpose key, or the machine's role key).
* This allows the operators to fetch this encrypted keyfile and use their copy
of the machine's OpenPGP key to extract the passphrase _on their
encrypted-storage workstations_ (recommendation is to not do this on the remote
repository).
Just to be sure, let's emphasize Borg's own recommendation:
> IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
>
> If you used a repokey mode, the key is stored in the repo, but you should
> back it up separately.
> Use "borg key export" to export the key, optionally in printable format.
> Write down the passphrase. Store both at safe place(s).
[Borg]: https://www.borgbackup.org/
[Puppet]: https://www.puppet.com/
[not possible anymore]: https://github.com/borgbackup/borg/issues/7047
[important to keep copies of the borg repository key safely elsewhere]: https://borgbackup.readthedocs.io/en/latest/faq.html#how-important-is-the-home-config-borg-directory
### eCryptfs
For [eCryptfs][]:
$SUDO ecryptfs-recover-private /media/$DISK/home/.ecryptfs/$USER/.Private
[eCryptfs]: https://www.ecryptfs.org/
|
