diff options
Diffstat (limited to 'docs')
-rw-r--r-- | docs/backups.md | 111 | ||||
-rw-r--r-- | docs/changelog.md | 21 | ||||
-rw-r--r-- | docs/index.md | 4 | ||||
-rw-r--r-- | docs/todo.md | 33 | ||||
-rw-r--r-- | docs/tpc.md | 85 |
5 files changed, 179 insertions, 75 deletions
diff --git a/docs/backups.md b/docs/backups.md index c612116..d39a85c 100644 --- a/docs/backups.md +++ b/docs/backups.md @@ -11,33 +11,73 @@ This mostly has to do with `hydractl` commands such as * But all this does not exempt a team from manually testing backups and to make their own offline copies in external drives. +## Parameters + +Start by running the following commands in your terminal, adjusting to your +case: + + export HYDRA="hydra-name" + export DISK="disk-name" + export DEVICE="/dev/sdc" + export USER="`whoami`" + export DOMAIN="`facter domain`" + + if [ "$USER" != "root" ]; then + export SUDO="sudo" + fi + ## New external drive -Proceed as follows +To add a new external drive into the pool, proceed as follows. + +First make sure to create a passphrase for the new disk into your hydra's keyring: + + keyringer $HYDRA pwgen disks/$DISK.$DOMAIN/luks/root + +If you're using a USB drive, run this before plugging it on the computer: + + hydractl usb-enable + +With the disk plugged in, make a layout with single `${DEVICE}1` partition +(example: a disk with only one partition like `/dev/sdc1`): + + $SUDO cfdisk ${DEVICE} + +Now create the LUKS encryption header in this new disk, using the passphrase +generated above by keyringer: - disk=new-disk-name - device=/dev/sdc - hydractl usb-enable # run this for USB drives, then connect the disk - sudo cfdisk ${device} # layout with single ${device}1 partition - sudo cryptsetup luksFormat ${device}1 - sudo cryptsetup luksOpen ${device}1 $disk - sudo mkfs.ext4 /dev/mapper/$disk - sudo mkdir /media/$disk - sudo mount /dev/mapper/$disk /media/$disk - sudo mkdir /media/$disk/media.`facter domain` - sudo chown `whoami`: /media/$disk/media.`facter domain` + $SUDO cryptsetup luksFormat ${DEVICE}1 -## NAS +If you want to make this volume to have a label (helpful for detecting the disk +etc; downside: third parties can easily get the name), run this command: + + $SUDO cryptsetup config ${DEVICE}1 --label ${DISK} + +Open the encrypted volume and create the basic filesystem structure needed +to sync content: + + $SUDO cryptsetup luksOpen ${DEVICE}1 $DISK + $SUDO mkfs.ext4 /dev/mapper/$DISK + $SUDO mkdir /media/$DISK + $SUDO mount /dev/mapper/$DISK /media/$DISK + $SUDO mkdir /media/$DISK/media.$DOMAIN + $SUDO chown ${USER}: /media/$DISK/media.$DOMAIN + +Finally, umount the drive: + + hydractl umount-media $DISK + +## Regular sync These commands should be enough to sync all media archives: hydractl usb-enable # run this for USB drives, then connect the disk - hydractl mount-media $MEDIA - hydractl sync-media $MEDIA + hydractl mount-media $DISK + hydractl sync-media $DISK As this should handle syncing all backups: - hydractl sync-backups $MEDIA + hydractl sync-backups $DISK ## TPC @@ -57,6 +97,11 @@ Then do the following: hydractl mount-media $TPC hydractl sync-tpc $TPC +If you don't want to do a full TPC sync, but just want to sync the home folders +(which is faster), use the following instead of the last command above: + + hydractl sync-home $TPC + To sync archives and remote backups, proceed with as explained in the NAS section. @@ -71,10 +116,10 @@ homedir backups in the external archive/backup volume. You might want to backup the whole SSD, M-SATA or microSD from your appliances. If so, proceed as follows with the appliance device connected in your TPC: - export appliance=appliance-name - export dest="/var/backups/remote/$appliance.`facter domain`/image/`date +%Y%m%d`" - sudo mkdir -p $dest - dcfldd if=/dev/sdb | bzip2 > $dest/$appliance.img.bz2 + export APPLIANCE="appliance-name" + export DEST="/var/backups/remote/${APPLIANCE}.${DOMAIN}/image/`date +%Y%m%d`" + $SUDO mkdir -p $dest + dcfldd if=/dev/sdb | bzip2 > $DEST/${APPLIANCE}.img.bz2 ## Smartphone @@ -100,7 +145,9 @@ A Backup Kit is a box with the following items: * External encrypted archive/backup disk. * Case for SSD transportation. * Laptop power adapter and cables. -* Dockstation SATA/USB/M.2/microSD/etc (with power adapter). +* Dockstation SATA/USB/M.2/microSD/etc (with power adapter -- usually 12V). +* Power adapter for external hard drives (usually 12V, and might be compatible + with the dockstationadapter, so you might carry just one). * USB power adapter and cable (including USB 2, USB 3 and USB C). * USB cables (USB 2, USB 3 and USB C) for the Dockstation and the external drive. * TPC laptop with battery and TPS (SSD, M.2 etc), optionally with a UltraBase/Dockstation. @@ -113,7 +160,13 @@ This may be the ultimate disaster recovery kit for your Hydra! ## Restore -Examples according to the software used to make the backup. +Having backup data leaked is a serious security issue, and that's why we +encrypt backups. +But losing access to the encrypted material is data loss, so it's important +to make sure in advance that we can get back the material. + +Procedures vary, and the following examples are sorted according to the +software used to make the backup. ### Duplicity @@ -128,10 +181,11 @@ For [duplicity][]: For [Borg][]: + export PATH_TO_RESTORE="path/to/be/restored" mkdir ~/temp/misc/restore cd ~/temp/misc/restore borg list ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg - borg extract ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg::$USER-2018-06-11T17:07:39 mnt/crypt/home/$USER/$FILE_OR_FOLDER + borg extract ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg::$USER-2018-06-11T17:07:39 mnt/crypt/home/$USER/$PATH_TO_RESTORE Make sure to cleanup `~/temp/misc/restore` after recovering what you need. @@ -153,6 +207,15 @@ Note on backup keys: encrypted-storage workstations_ (recommendation is to not do this on the remote repository). +Just to be sure, let's emphasize Borg's own recommendation: + +> IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo! +> +> If you used a repokey mode, the key is stored in the repo, but you should +> back it up separately. +> Use "borg key export" to export the key, optionally in printable format. +> Write down the passphrase. Store both at safe place(s). + [Borg]: https://www.borgbackup.org/ [Puppet]: https://www.puppet.com/ [not possible anymore]: https://github.com/borgbackup/borg/issues/7047 @@ -162,6 +225,6 @@ Note on backup keys: For [eCryptfs][]: - sudo ecryptfs-recover-private /media/$MEDIA/home/.ecryptfs/$USER/.Private + $SUDO ecryptfs-recover-private /media/$DISK/home/.ecryptfs/$USER/.Private [eCryptfs]: https://www.ecryptfs.org/ diff --git a/docs/changelog.md b/docs/changelog.md index 3620de4..155459b 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -2,9 +2,20 @@ ## 0.3.0 - Unrelased +### hydra + +* [x] Deploy: + * [x] Automatically fix permission of (or delete) puppet's `devices` + folder: https://github.com/puppetlabs/puppet-specifications/blob/master/file_paths.md * [x] Keys: - * [x] Deprecate generating and deploying/import borg keys, since - pre-generation is not a supported behavior right now: - https://github.com/borgbackup/borg/issues/7047 - * [x] Document about how keys are encrypted and backed up in the server: - https://borgbackup.readthedocs.io/en/latest/faq.html#how-important-is-the-home-config-borg-directory + * [x] Deprecate generating and deploying/import borg keys, since + pre-generation is not a supported behavior right now: + https://github.com/borgbackup/borg/issues/7047 + * [x] Document about how keys are encrypted and backed up in the server: + https://borgbackup.readthedocs.io/en/latest/faq.html#how-important-is-the-home-config-borg-directory + +### hydractl + +* [x] Provision: increase default partition sizes from 20G to 40G, as nowadays + distro size increase a lot and a 20G system partition can be filled in + easily. diff --git a/docs/index.md b/docs/index.md index 58aead8..66915c2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,5 +1,7 @@ # Hydra Suite - Command and Control +![Hydra](assets/logo.png){ align=left } + Hydra is a suite for orchestration and management of machines composed of **lots of conventions and assumptions**, and the following basic commands: @@ -10,7 +12,7 @@ Hydra is a suite for orchestration and management of machines composed of An Hydra is not a "cloud computing" platform, it's something else: a set of systems that operate together, where any node can be used to spawn new nodes. -This is an ongoing experiment is how a person or a collective can manage +This is an ongoing experiment in how a person or a collective can manage many computers in an unified way. It's not production ready, and it always struggles to pass the test of time. diff --git a/docs/todo.md b/docs/todo.md index 1abe831..48cbf94 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -2,17 +2,44 @@ ## hydra -* [ ] Support for [cumin][] ([Debian package](https://tracker.debian.org/pkg/cumin)). +* [ ] Deploy: + * [ ] Collect basic hardware information along with facts: cpuid, + dmidecode, hwinfo, lsblk, lscpu, lshw, lspci, lsusb etc. + Store somewhere under the `config/hardware` folder? + * [ ] Command line is broken for ansible when multiple nodes are provided. + * [ ] Use console-based GnuPG agent when calling `keyringer`. +* [ ] Mass: + * [ ] Support for [cumin][] ([Debian package](https://tracker.debian.org/pkg/cumin)). [cumin]: https://github.com/wikimedia/cumin ## hydractl +* [ ] External volume script (replacing new drive procedure from + `docs/backups.md`). +* [ ] Mount/umount media: + * [ ] Fix STANDBY handling on external drives: + > Disabling STANDBY on drive... + > couldn't find field acronym: STANDBY + > [perhaps a '--transport=<tn>' or '--vendor=<vn>' option is needed] + * [ ] Mount/umount system volume supporting split partiton scheme (`root`, + `var`, `home` etc). + * [ ] Try to detected the device partition (`/dev/sdb1` etc) based + on the LUKS2 label. +* [ ] Syncing: + * [ ] Syncing packages: a frontend to `apt-offline` that uses `git-annex` + repositories: getting, installing, cleaning. One node can request + packages through an external drive, and another can fetch then. + * [ ] Integration with [Baobáxia](https://baobaxia.mocambos.net)? + Maybe that will already happen if `git-annex` can run directly + on a mucua's repository. + * [ ] Merge `sync-home` into `sync-tpc`, or make `sync-tpc` call + `sync-home`, to avoid code duplication. * [ ] Provision: * [ ] Fix booting issues detailed [here](tpc.md#booting). -* [ ] Deploy: - * [ ] Use console-based GnuPG agent when calling `keyringer`. * [ ] Upgrade: + * [ ] Support for [fwupd](https://fwupd.org), at least for showing + available updates. * [ ] Raspbian does not have the `non-free-firmware` component? Maybe this is not an issue, as we may move away from Raspbian/Raspberry Pi OS. diff --git a/docs/tpc.md b/docs/tpc.md index d3e7e06..c2a6954 100644 --- a/docs/tpc.md +++ b/docs/tpc.md @@ -6,7 +6,10 @@ The Hydra Suite was made with the [Trusted Personal Computer (TPC)][] This guide documents the lifecycle of a TPC storage device, or a "TPS Cartridge": - Cartridge = TPS = Trusted Personal Storage (like and SSD, microSD or M.2) + Cartridge = TPS = Trusted Personal Storage + +A TPS can usually be whatever removable block device like an SSD, microSD, M.2 +etc. The idea is that a number of TPCs can have interchangeable cartridges, as long as each cartridge has an unique name. @@ -19,12 +22,12 @@ as each cartridge has an unique name. Start by running the following commands in your terminal, adjusting to your case: - export HYDRA="nome-da-hydra" - export VOLNAME="nome-do-disco" - export DOMAIN="dominio.example.org" + export HYDRA="hydra-name" + export DISK="disk-name" export DEVICE="/dev/sdb" - export USER=`whoami` - export DEST=/media/$VOLNAME + export DOMAIN="`facter domain`" + export USER="`whoami`" + export DEST="/media/$DISK" export CONFIG_FOLDER="`hydra $HYDRA folder`" if [ "$USER" != "root" ]; then @@ -46,52 +49,52 @@ procedures from this document can be done from another TPC. If non-existing, create entries and keys: - hydra $HYDRA newnode $VOLNAME tpc + hydra $HYDRA newnode $DISK tpc Customize: - vim $CONFIG_FOLDER/config/provision/$VOLNAME.conf - vim $CONFIG_FOLDER/puppet/config/node/$VOLNAME.$DOMAIN.yaml - vim $CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN.yaml + vim $CONFIG_FOLDER/config/provision/$DISK.conf + vim $CONFIG_FOLDER/puppet/config/node/$DISK.$DOMAIN.yaml + vim $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml ### Secrets Proceed adding other passwords and secrets at -`$CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN`, make sure to +`$CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN`, make sure to encode them with [hiera-yaml][]: - hydra $HYDRA eyaml $VOLNAME encrypt -q -o block -p -l some::password + hydra $HYDRA eyaml $DISK encrypt -q -o block -p -l some::password If you already have the secret somewhere, you can use a construction like this which already adds the eyaml block into the hiera config file: keyringer $HYDA decrypt /path/to/some/secret | \ tr -d '\n' | \ - hydra $HYDRA eyaml $VOLNAME encrypt -q -o block --stdin -l some::password >> \ - $CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN.yaml + hydra $HYDRA eyaml $DISK encrypt -q -o block --stdin -l some::password >> \ + $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml You can also ensure a fresh random passphrase is used, using your favourite generator like this: head -c ${1:-20} /dev/urandom | base64 | \ tr -d '\n' | \ - hydra $HYDRA eyaml $VOLNAME encrypt -q -o block --stdin -l some::random:password >> \ - $CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN.yaml + hydra $HYDRA eyaml $DISK encrypt -q -o block --stdin -l some::random::password >> \ + $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml For `passwd(5)` and `shadow(5)` hashed passphrases, use something like this: mkpasswd -m sha-512 | \ tr -d '\n' | \ - hydra $HYDRA eyaml $VOLNAME encrypt -q -o block --stdin -l some::random:password >> \ - $CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN.yaml + hydra $HYDRA eyaml $DISK encrypt -q -o block --stdin -l some::random::password >> \ + $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml Or this, for a random passphrase: head -c ${1:-20} /dev/urandom | base64 | \ mkpasswd -m sha-512 --stdin | \ tr -d '\n' | \ - hydra $HYDRA eyaml $VOLNAME encrypt -q -o block --stdin -l some::random:password >> \ - $CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN.yaml + hydra $HYDRA eyaml $DISK encrypt -q -o block --stdin -l some::random::password >> \ + $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml [hiera-yaml]: https://github.com/voxpupuli/hiera-eyaml @@ -100,7 +103,7 @@ Or this, for a random passphrase: The proceed with the system installation, which should create all volumes in the TPS and setup the base Operating System: - hydra $HYDRA provision $VOLNAME + hydra $HYDRA provision $DISK ### Metadata @@ -112,23 +115,23 @@ can be useful during recovery procedures. 1. Write down UUIDS, models and serials from the new disk. Examples: - lsblk $DEVICE -n -o +UUID | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/lsblk - udevadm info --query=all --name=$DEVICE | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/udevadm + lsblk $DEVICE -n -o +UUID | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/lsblk + udevadm info --query=all --name=$DEVICE | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/udevadm # For non-USB and non-microSD disks only - $SUDO hdparm -I $DEVICE | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/hdparm - $SUDO smartctl -i $DEVICE | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/smartctl + $SUDO hdparm -I $DEVICE | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/hdparm + $SUDO smartctl -i $DEVICE | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/smartctl 2. Save partition tables, the LUKS header and other relevant information: - $SUDO LC_ALL=C sfdisk -d $DEVICE | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/sfdisk + $SUDO LC_ALL=C sfdisk -d $DEVICE | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/sfdisk - $SUDO cryptsetup luksDump /dev/mapper/$VOLNAME-root | keyringer $HYDRA encrypt nodes/houdini.fluxo.info/luksDump-root - $SUDO cryptsetup luksDump /dev/mapper/$VOLNAME-swap | keyringer $HYDRA encrypt nodes/houdini.fluxo.info/luksDump-swap + $SUDO cryptsetup luksDump /dev/mapper/$DISK-root | keyringer $HYDRA encrypt nodes/houdini.fluxo.info/luksDump-root + $SUDO cryptsetup luksDump /dev/mapper/$DISK-swap | keyringer $HYDRA encrypt nodes/houdini.fluxo.info/luksDump-swap - $SUDO cryptsetup luksHeaderBackup /dev/mapper/$VOLNAME-root --header-backup-file luksHeaderBackup-root + $SUDO cryptsetup luksHeaderBackup /dev/mapper/$DISK-root --header-backup-file luksHeaderBackup-root $SUDO chown $(whoami) luksHeaderBackup-root - keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/luksHeaderBackup-root luksHeaderBackup-root + keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/luksHeaderBackup-root luksHeaderBackup-root chmod +w luksHeaderBackup-root wipe luksHeaderBackup-root @@ -137,15 +140,15 @@ can be useful during recovery procedures. * Optionally, configure ATA passphrases for the new disk. If so, make sure to save the passphrase in the keyring: - keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/ata + keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/ata ## Mounting Use the following whenever you need to mount the cartridge in another system: - hydractl mount-media $VOLNAME + hydractl mount-media $DISK -This will make the volume available at `/media/$VOLNAME`. +This will make the volume available at `/media/$DISK`. ## Initial user @@ -161,7 +164,7 @@ home folder encrypted, backup your data and then proceed running ### Syncing the user and encrypted home in the new cartridge -Then, with the new TPS cartridge mounted (`hydractl mount-media $VOLNAME`), do +Then, with the new TPS cartridge mounted (`hydractl mount-media $DISK`), do the following to replicate your user and homedir there: $SUDO chroot $DEST adduser $USER @@ -183,7 +186,7 @@ and is logged in there as `root`. From another system, and with the cartridge mounted: - hydractl sync-tpc $VOLNAME + hydractl sync-tpc $DISK This might involve a lot of data being transmitted. If you want `sync-tpc` to ignore all your QEMU virtual machine images hosted at `/var/cache/quemu`, setup @@ -201,7 +204,7 @@ Now that basic provisioning is completed, you can unmount the cartridge. From another system, and with the cartridge mounted: - hydractl umount-media $VOLNAME + hydractl umount-media $DISK ## Booting @@ -209,20 +212,18 @@ The current provisioning procedure (as of 2024-02) may yield into a partially bootable system, and may need some additional manual steps right after installation: - hydractl mount-media $VOLNAME + hydractl mount-media $DISK sudo chroot $DEST update-initramfs -v -u - hydractl umount-media $VOLNAME + hydractl umount-media $DISK During boot, at the `(initramfs)` prompt, type this to successfully unlock the encrypted volumes: - cryptsetup luksOpen /dev/mapper/volumename-root root - cryptsetup luksOpen /dev/mapper/volumename-swap swap + cryptsetup luksOpen /dev/mapper/${DISK}-root root + cryptsetup luksOpen /dev/mapper/${DISK}-swap swap ^Ctrl-D -Where `volumename` is `$VOLNAME`. - Then, in the running system, do: sudo update-initramfs -v -u |