aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
Diffstat (limited to 'docs')
-rw-r--r--docs/backups.md134
-rw-r--r--docs/changelog.md21
-rw-r--r--docs/ideas.md2
-rw-r--r--docs/index.md4
-rw-r--r--docs/todo.md55
-rw-r--r--docs/tpc.md113
6 files changed, 270 insertions, 59 deletions
diff --git a/docs/backups.md b/docs/backups.md
index f6faf8a..d39a85c 100644
--- a/docs/backups.md
+++ b/docs/backups.md
@@ -11,33 +11,73 @@ This mostly has to do with `hydractl` commands such as
* But all this does not exempt a team from manually testing backups and to make
their own offline copies in external drives.
+## Parameters
+
+Start by running the following commands in your terminal, adjusting to your
+case:
+
+ export HYDRA="hydra-name"
+ export DISK="disk-name"
+ export DEVICE="/dev/sdc"
+ export USER="`whoami`"
+ export DOMAIN="`facter domain`"
+
+ if [ "$USER" != "root" ]; then
+ export SUDO="sudo"
+ fi
+
## New external drive
-Proceed as follows
+To add a new external drive into the pool, proceed as follows.
+
+First make sure to create a passphrase for the new disk into your hydra's keyring:
+
+ keyringer $HYDRA pwgen disks/$DISK.$DOMAIN/luks/root
+
+If you're using a USB drive, run this before plugging it on the computer:
+
+ hydractl usb-enable
+
+With the disk plugged in, make a layout with single `${DEVICE}1` partition
+(example: a disk with only one partition like `/dev/sdc1`):
+
+ $SUDO cfdisk ${DEVICE}
+
+Now create the LUKS encryption header in this new disk, using the passphrase
+generated above by keyringer:
- disk=new-disk-name
- device=/dev/sdc
- hydractl usb-enable # run this for USB drives, then connect the disk
- sudo cfdisk ${device} # layout with single ${device}1 partition
- sudo cryptsetup luksFormat ${device}1
- sudo cryptsetup luksOpen ${device}1 $disk
- sudo mkfs.ext4 /dev/mapper/$disk
- sudo mkdir /media/$disk
- sudo mount /dev/mapper/$disk /media/$disk
- sudo mkdir /media/$disk/media.`facter domain`
- sudo chown `whoami`: /media/$disk/media.`facter domain`
+ $SUDO cryptsetup luksFormat ${DEVICE}1
-## NAS
+If you want to make this volume to have a label (helpful for detecting the disk
+etc; downside: third parties can easily get the name), run this command:
+
+ $SUDO cryptsetup config ${DEVICE}1 --label ${DISK}
+
+Open the encrypted volume and create the basic filesystem structure needed
+to sync content:
+
+ $SUDO cryptsetup luksOpen ${DEVICE}1 $DISK
+ $SUDO mkfs.ext4 /dev/mapper/$DISK
+ $SUDO mkdir /media/$DISK
+ $SUDO mount /dev/mapper/$DISK /media/$DISK
+ $SUDO mkdir /media/$DISK/media.$DOMAIN
+ $SUDO chown ${USER}: /media/$DISK/media.$DOMAIN
+
+Finally, umount the drive:
+
+ hydractl umount-media $DISK
+
+## Regular sync
These commands should be enough to sync all media archives:
hydractl usb-enable # run this for USB drives, then connect the disk
- hydractl mount-media $MEDIA
- hydractl sync-media $MEDIA
+ hydractl mount-media $DISK
+ hydractl sync-media $DISK
As this should handle syncing all backups:
- hydractl sync-backups $MEDIA
+ hydractl sync-backups $DISK
## TPC
@@ -57,6 +97,11 @@ Then do the following:
hydractl mount-media $TPC
hydractl sync-tpc $TPC
+If you don't want to do a full TPC sync, but just want to sync the home folders
+(which is faster), use the following instead of the last command above:
+
+ hydractl sync-home $TPC
+
To sync archives and remote backups, proceed with as explained in the NAS
section.
@@ -71,10 +116,10 @@ homedir backups in the external archive/backup volume.
You might want to backup the whole SSD, M-SATA or microSD from your appliances.
If so, proceed as follows with the appliance device connected in your TPC:
- export appliance=appliance-name
- export dest="/var/backups/remote/$appliance.`facter domain`/image/`date +%Y%m%d`"
- sudo mkdir -p $dest
- dcfldd if=/dev/sdb | bzip2 > $dest/$appliance.img.bz2
+ export APPLIANCE="appliance-name"
+ export DEST="/var/backups/remote/${APPLIANCE}.${DOMAIN}/image/`date +%Y%m%d`"
+ $SUDO mkdir -p $dest
+ dcfldd if=/dev/sdb | bzip2 > $DEST/${APPLIANCE}.img.bz2
## Smartphone
@@ -100,10 +145,12 @@ A Backup Kit is a box with the following items:
* External encrypted archive/backup disk.
* Case for SSD transportation.
* Laptop power adapter and cables.
-* Dockstation SATA/USB/M-2/microSD/etc (with power adapter).
+* Dockstation SATA/USB/M.2/microSD/etc (with power adapter -- usually 12V).
+* Power adapter for external hard drives (usually 12V, and might be compatible
+ with the dockstationadapter, so you might carry just one).
* USB power adapter and cable (including USB 2, USB 3 and USB C).
* USB cables (USB 2, USB 3 and USB C) for the Dockstation and the external drive.
-* TPC laptop with battery and TPS (SSD, M-2 etc), optionally with a UltraBase/Dockstation.
+* TPC laptop with battery and TPS (SSD, M.2 etc), optionally with a UltraBase/Dockstation.
* Philips screwdriver and other tools.
* FCR-MG2 adaptor for microSD to USB.
* Anything else your need (like eyeglasses).
@@ -113,7 +160,13 @@ This may be the ultimate disaster recovery kit for your Hydra!
## Restore
-Examples according to the software used to make the backup.
+Having backup data leaked is a serious security issue, and that's why we
+encrypt backups.
+But losing access to the encrypted material is data loss, so it's important
+to make sure in advance that we can get back the material.
+
+Procedures vary, and the following examples are sorted according to the
+software used to make the backup.
### Duplicity
@@ -128,19 +181,50 @@ For [duplicity][]:
For [Borg][]:
+ export PATH_TO_RESTORE="path/to/be/restored"
mkdir ~/temp/misc/restore
cd ~/temp/misc/restore
borg list ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg
- borg extract ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg::$USER-2018-06-11T17:07:39 mnt/crypt/home/$USER/$FILE_OR_FOLDER
+ borg extract ssh://$USER@$SERVER:$PORT//var/backups/users/$USER/borg::$USER-2018-06-11T17:07:39 mnt/crypt/home/$USER/$PATH_TO_RESTORE
Make sure to cleanup `~/temp/misc/restore` after recovering what you need.
+Note on backup keys:
+
+* In the past (before 2024), the Hydra Suite and it's companion [Puppet][]
+ modules used pre-generated [Borg][] repository keys for the sake of automation.
+ This is [not possible anymore][].
+* As it's [important to keep copies of the borg repository key safely
+ elsewhere][], the managed configuration supports OpenPGP-encrypting the
+ repository key and uploading it to the remote repository.
+* This OpenPGP-encrypted key file is named as `keyfile.asc` and is uploaded
+ in the root folder of the remote repository.
+* This OpenPGP-encrypted key file is encrypted and signed with a provided
+ OpenPGP keypair and passphrase (convention is to use the machines's OpenPGP
+ general purpose key, or the machine's role key).
+* This allows the operators to fetch this encrypted keyfile and use their copy
+ of the machine's OpenPGP key to extract the passphrase _on their
+ encrypted-storage workstations_ (recommendation is to not do this on the remote
+ repository).
+
+Just to be sure, let's emphasize Borg's own recommendation:
+
+> IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
+>
+> If you used a repokey mode, the key is stored in the repo, but you should
+> back it up separately.
+> Use "borg key export" to export the key, optionally in printable format.
+> Write down the passphrase. Store both at safe place(s).
+
[Borg]: https://www.borgbackup.org/
+[Puppet]: https://www.puppet.com/
+[not possible anymore]: https://github.com/borgbackup/borg/issues/7047
+[important to keep copies of the borg repository key safely elsewhere]: https://borgbackup.readthedocs.io/en/latest/faq.html#how-important-is-the-home-config-borg-directory
### eCryptfs
For [eCryptfs][]:
- sudo ecryptfs-recover-private /media/$MEDIA/home/.ecryptfs/$USER/.Private
+ $SUDO ecryptfs-recover-private /media/$DISK/home/.ecryptfs/$USER/.Private
[eCryptfs]: https://www.ecryptfs.org/
diff --git a/docs/changelog.md b/docs/changelog.md
new file mode 100644
index 0000000..155459b
--- /dev/null
+++ b/docs/changelog.md
@@ -0,0 +1,21 @@
+# ChangeLog
+
+## 0.3.0 - Unrelased
+
+### hydra
+
+* [x] Deploy:
+ * [x] Automatically fix permission of (or delete) puppet's `devices`
+ folder: https://github.com/puppetlabs/puppet-specifications/blob/master/file_paths.md
+* [x] Keys:
+ * [x] Deprecate generating and deploying/import borg keys, since
+ pre-generation is not a supported behavior right now:
+ https://github.com/borgbackup/borg/issues/7047
+ * [x] Document about how keys are encrypted and backed up in the server:
+ https://borgbackup.readthedocs.io/en/latest/faq.html#how-important-is-the-home-config-borg-directory
+
+### hydractl
+
+* [x] Provision: increase default partition sizes from 20G to 40G, as nowadays
+ distro size increase a lot and a 20G system partition can be filled in
+ easily.
diff --git a/docs/ideas.md b/docs/ideas.md
index 7c96301..dfd549b 100644
--- a/docs/ideas.md
+++ b/docs/ideas.md
@@ -1,5 +1,7 @@
# Ideas
+## Misc
+
* `hydractl`:
* `mount-kvmx-supervised`: open encrypted volumes from supervised [kvmx][] guests.
* `hydra`:
diff --git a/docs/index.md b/docs/index.md
index 58aead8..66915c2 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,5 +1,7 @@
# Hydra Suite - Command and Control
+![Hydra](assets/logo.png){ align=left }
+
Hydra is a suite for orchestration and management of machines composed of
**lots of conventions and assumptions**, and the following basic commands:
@@ -10,7 +12,7 @@ Hydra is a suite for orchestration and management of machines composed of
An Hydra is not a "cloud computing" platform, it's something else: a set of
systems that operate together, where any node can be used to spawn new nodes.
-This is an ongoing experiment is how a person or a collective can manage
+This is an ongoing experiment in how a person or a collective can manage
many computers in an unified way. It's not production ready, and it always
struggles to pass the test of time.
diff --git a/docs/todo.md b/docs/todo.md
new file mode 100644
index 0000000..e13ae1f
--- /dev/null
+++ b/docs/todo.md
@@ -0,0 +1,55 @@
+# TODO
+
+## hydra
+
+* [ ] Deploy:
+ * [ ] Collect basic hardware information along with facts: cpuid,
+ dmidecode, hwinfo, lsblk, lscpu, lshw, lspci, lsusb, sfdisk etc.
+ Store somewhere under the `config/hardware` folder?
+ Could also be a separate action, `hydra <hydra> collect <node>`.
+ * [ ] Command line is broken for ansible when multiple nodes are provided.
+ * [ ] Use console-based GnuPG agent when calling `keyringer`.
+* [ ] Mass:
+ * [ ] Support for [cumin][] ([Debian package](https://tracker.debian.org/pkg/cumin)).
+
+[cumin]: https://github.com/wikimedia/cumin
+
+## hydractl
+
+* [ ] External volume script (replacing new drive procedure from
+ `docs/backups.md`).
+* [ ] Mount/umount media:
+ * [ ] Fix STANDBY handling on external drives:
+ > Disabling STANDBY on drive...
+ > couldn't find field acronym: STANDBY
+ > [perhaps a '--transport=<tn>' or '--vendor=<vn>' option is needed]
+ * [ ] Mount/umount system volume supporting split partiton scheme (`root`,
+ `var`, `home` etc).
+ * [ ] Try to detect the device partition (`/dev/sdb1` etc) based
+ on the LUKS2 label.
+ * [ ] Cryptdisks script needs fixing as `systemd-tty-ask-password-agent` is not
+ catching cryptsetup password prompts anymore. Otherwise, consider to
+ deprecate this script.
+* [ ] Syncing:
+ * [ ] Sync lock.
+ * [ ] Run `git submodule --update --init --recursive` after running
+ `git annex sync`.
+ * [ ] Shall `sync-media` do a `git unannex` on sidecar and other metadata
+ files and manage them through Git instead, or keep them as unlocked
+ files?
+ * [ ] Syncing packages: a frontend to `apt-offline` that uses `git-annex`
+ repositories: getting, installing, cleaning. One node can request
+ packages through an external drive, and another can fetch then.
+ * [ ] Integration with [Baobáxia](https://baobaxia.mocambos.net)?
+ Maybe that will already happen if `git-annex` can run directly
+ on a mucua's repository.
+ * [ ] Merge `sync-home` into `sync-tpc`, or make `sync-tpc` call
+ `sync-home`, to avoid code duplication.
+* [ ] Provision:
+ * [ ] Fix booting issues detailed [here](tpc.md#booting).
+* [ ] Upgrade:
+ * [ ] Support for [fwupd](https://fwupd.org), at least for showing
+ available updates.
+ * [ ] Raspbian does not have the `non-free-firmware` component?
+ Maybe this is not an issue, as we may move away from
+ Raspbian/Raspberry Pi OS.
diff --git a/docs/tpc.md b/docs/tpc.md
index 54f707f..c2a6954 100644
--- a/docs/tpc.md
+++ b/docs/tpc.md
@@ -6,7 +6,10 @@ The Hydra Suite was made with the [Trusted Personal Computer (TPC)][]
This guide documents the lifecycle of a TPC storage device, or a "TPS
Cartridge":
- Cartridge = TPS = Trusted Personal Storage (like and SSD, microSD or M-2)
+ Cartridge = TPS = Trusted Personal Storage
+
+A TPS can usually be whatever removable block device like an SSD, microSD, M.2
+etc.
The idea is that a number of TPCs can have interchangeable cartridges, as long
as each cartridge has an unique name.
@@ -19,12 +22,12 @@ as each cartridge has an unique name.
Start by running the following commands in your terminal, adjusting to your
case:
- export HYDRA="nome-da-hydra"
- export VOLNAME="nome-do-disco"
- export DOMAIN="dominio.example.org"
+ export HYDRA="hydra-name"
+ export DISK="disk-name"
export DEVICE="/dev/sdb"
- export USER=`whoami`
- export DEST=/media/$VOLNAME
+ export DOMAIN="`facter domain`"
+ export USER="`whoami`"
+ export DEST="/media/$DISK"
export CONFIG_FOLDER="`hydra $HYDRA folder`"
if [ "$USER" != "root" ]; then
@@ -46,35 +49,52 @@ procedures from this document can be done from another TPC.
If non-existing, create entries and keys:
- hydra $HYDRA newnode $VOLNAME tpc
+ hydra $HYDRA newnode $DISK tpc
Customize:
- vim $CONFIG_FOLDER/config/provision/$VOLNAME.conf
- vim $CONFIG_FOLDER/puppet/config/node/$VOLNAME.$DOMAIN.yaml
- vim $CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN.yaml
+ vim $CONFIG_FOLDER/config/provision/$DISK.conf
+ vim $CONFIG_FOLDER/puppet/config/node/$DISK.$DOMAIN.yaml
+ vim $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml
### Secrets
Proceed adding other passwords and secrets at
-`$CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN`, make sure to
+`$CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN`, make sure to
encode them with [hiera-yaml][]:
- hydra $HYDRA eyaml $VOLNAME encrypt -q -o block -p -l some::password
+ hydra $HYDRA eyaml $DISK encrypt -q -o block -p -l some::password
If you already have the secret somewhere, you can use a construction like this
which already adds the eyaml block into the hiera config file:
keyringer $HYDA decrypt /path/to/some/secret | \
- hydra $HYDRA eyaml $VOLNAME encrypt -q -o block --sdtin -l some::password >> \
- $CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN.yaml
+ tr -d '\n' | \
+ hydra $HYDRA eyaml $DISK encrypt -q -o block --stdin -l some::password >> \
+ $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml
You can also ensure a fresh random passphrase is used, using your favourite
generator like this:
head -c ${1:-20} /dev/urandom | base64 | \
- hydra $HYDRA eyaml $VOLNAME encrypt -q -o block --sdtin -l some::random:password >> \
- $CONFIG_FOLDER/puppet/config/secrets/node/$VOLNAME.$DOMAIN.yaml
+ tr -d '\n' | \
+ hydra $HYDRA eyaml $DISK encrypt -q -o block --stdin -l some::random::password >> \
+ $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml
+
+For `passwd(5)` and `shadow(5)` hashed passphrases, use something like this:
+
+ mkpasswd -m sha-512 | \
+ tr -d '\n' | \
+ hydra $HYDRA eyaml $DISK encrypt -q -o block --stdin -l some::random::password >> \
+ $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml
+
+Or this, for a random passphrase:
+
+ head -c ${1:-20} /dev/urandom | base64 | \
+ mkpasswd -m sha-512 --stdin | \
+ tr -d '\n' | \
+ hydra $HYDRA eyaml $DISK encrypt -q -o block --stdin -l some::random::password >> \
+ $CONFIG_FOLDER/puppet/config/secrets/node/$DISK.$DOMAIN.yaml
[hiera-yaml]: https://github.com/voxpupuli/hiera-eyaml
@@ -83,7 +103,7 @@ generator like this:
The proceed with the system installation, which should create all volumes in
the TPS and setup the base Operating System:
- hydra $HYDRA provision $VOLNAME
+ hydra $HYDRA provision $DISK
### Metadata
@@ -95,23 +115,23 @@ can be useful during recovery procedures.
1. Write down UUIDS, models and serials from the new disk. Examples:
- lsblk $DEVICE -n -o +UUID | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/lsblk
- udevadm info --query=all --name=$DEVICE | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/udevadm
+ lsblk $DEVICE -n -o +UUID | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/lsblk
+ udevadm info --query=all --name=$DEVICE | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/udevadm
# For non-USB and non-microSD disks only
- $SUDO hdparm -I $DEVICE | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/hdparm
- $SUDO smartctl -i $DEVICE | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/smartctl
+ $SUDO hdparm -I $DEVICE | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/hdparm
+ $SUDO smartctl -i $DEVICE | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/smartctl
2. Save partition tables, the LUKS header and other relevant information:
- $SUDO LC_ALL=C sfdisk -d $DEVICE | keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/sfdisk
+ $SUDO LC_ALL=C sfdisk -d $DEVICE | keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/sfdisk
- $SUDO cryptsetup luksDump /dev/mapper/$VOLNAME-root | keyringer $HYDRA encrypt nodes/houdini.fluxo.info/luksDump-root
- $SUDO cryptsetup luksDump /dev/mapper/$VOLNAME-swap | keyringer $HYDRA encrypt nodes/houdini.fluxo.info/luksDump-swap
+ $SUDO cryptsetup luksDump /dev/mapper/$DISK-root | keyringer $HYDRA encrypt nodes/houdini.fluxo.info/luksDump-root
+ $SUDO cryptsetup luksDump /dev/mapper/$DISK-swap | keyringer $HYDRA encrypt nodes/houdini.fluxo.info/luksDump-swap
- $SUDO cryptsetup luksHeaderBackup /dev/mapper/$VOLNAME-root --header-backup-file luksHeaderBackup-root
+ $SUDO cryptsetup luksHeaderBackup /dev/mapper/$DISK-root --header-backup-file luksHeaderBackup-root
$SUDO chown $(whoami) luksHeaderBackup-root
- keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/luksHeaderBackup-root luksHeaderBackup-root
+ keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/luksHeaderBackup-root luksHeaderBackup-root
chmod +w luksHeaderBackup-root
wipe luksHeaderBackup-root
@@ -120,15 +140,15 @@ can be useful during recovery procedures.
* Optionally, configure ATA passphrases for the new disk. If so, make sure to
save the passphrase in the keyring:
- keyringer $HYDRA encrypt nodes/$VOLNAME.$DOMAIN/ata
+ keyringer $HYDRA encrypt nodes/$DISK.$DOMAIN/ata
## Mounting
Use the following whenever you need to mount the cartridge in another system:
- hydractl mount-media $VOLNAME
+ hydractl mount-media $DISK
-This will make the volume available at `/media/$VOLNAME`.
+This will make the volume available at `/media/$DISK`.
## Initial user
@@ -144,7 +164,7 @@ home folder encrypted, backup your data and then proceed running
### Syncing the user and encrypted home in the new cartridge
-Then, with the new TPS cartridge mounted (`hydractl mount-media $VOLNAME`), do
+Then, with the new TPS cartridge mounted (`hydractl mount-media $DISK`), do
the following to replicate your user and homedir there:
$SUDO chroot $DEST adduser $USER
@@ -166,7 +186,7 @@ and is logged in there as `root`.
From another system, and with the cartridge mounted:
- hydractl sync-tpc $VOLNAME
+ hydractl sync-tpc $DISK
This might involve a lot of data being transmitted. If you want `sync-tpc` to
ignore all your QEMU virtual machine images hosted at `/var/cache/quemu`, setup
@@ -184,7 +204,34 @@ Now that basic provisioning is completed, you can unmount the cartridge.
From another system, and with the cartridge mounted:
- hydractl umount-media $VOLNAME
+ hydractl umount-media $DISK
+
+## Booting
+
+The current provisioning procedure (as of 2024-02) may yield into
+a partially bootable system, and may need some additional manual
+steps right after installation:
+
+ hydractl mount-media $DISK
+ sudo chroot $DEST
+ update-initramfs -v -u
+ hydractl umount-media $DISK
+
+During boot, at the `(initramfs)` prompt, type this to successfully
+unlock the encrypted volumes:
+
+ cryptsetup luksOpen /dev/mapper/${DISK}-root root
+ cryptsetup luksOpen /dev/mapper/${DISK}-swap swap
+ ^Ctrl-D
+
+Then, in the running system, do:
+
+ sudo update-initramfs -v -u
+
+With these steps, the system's boot procedure was fixed.
+
+But there are fixes be done at `hydractl provision` to produce fully working
+installations.
## Deploy
@@ -218,7 +265,7 @@ collected data:
## Adding a new hardware in the TPC pool
-* Proceed with [homologatin a new unit](https://plano.autodefesa.org/specs/tpc.html).
+* Proceed with [homologating a new unit](https://plano.autodefesa.org/specs/tpc.html).
* Provision, deploy, sync-media, sync-vms etc.
* Add the devices' network MAC Addresses at `70-persistent-net.rules`, then make sure
to sync this in the other cartridges.
generated by cgit v1.2.3 (git 2.39.1) at 2025-05-10 03:49:05 +0000