I have a very similar setup to the author, but instead of running Tailscale in my initramfs, I have a Raspberry Pi sitting next to the home server (which is on my Tailscale network) and I use it like a bastion host. Process is something like:
1. SSH into the Pi
2. Issue the Wake-on-LAN packet to boot the server
3. Tunnel that server's SSH port to my laptop
4. SSH into the initramfs SSH server (I use TinySSH), enter the decryption key
5. Wait for server to come up, then access via Tailscale
This is more complicated than the author's setup in that it requires another device (the Pi), but it's simpler in that you don't need to have the initramfs on your Tailnet.
Mandos works with initramfs images created by both initramfs-tools and dracut, and is present in Debian since 2011, so no need to use a third-party package.
Why is this needed at all? As the decrypted key is in memory before the reboot, can’t it just be written to a know location in memory and have kexec be instructed to read it early on?
Oh for sure something is needed for a full start from zero. But the much more common case for a computer with backup power is regular restarts after applying patches that require a reboot. Would be much more pleasant for that to work out of the box with no manual interaction at all.
Good FAQ, clearly stating the weak point of physical access. For a server that threatmodel can work, for a fleet of edge/iot devices in unsecured locations without permanent uptime there is no real solution to be expected without custom silicon logic (like in smartcards) on the soc.
I have something similar set up to unlock the drives on my home server. Just the SSH in initramfs though, tailscale is pretty cool.
I've done stuff with mkinitcpio / initramfs on arch before, can't remember exactly what for. I still run arch on my main laptop. I'm running nixos on my home server though, and adding something like this is so easy by comparison.
Glad to see another example of this! Remote unlocking of your personal server's encrypted hard drive is PITA.
Other options that I've investigated that involve having a second server:
* A second server with Tang, and Clevis in the initramfs OS
* Keylime
Putting tailscale in the initramfs, and then updating the certs on a frequent enough schedule, seems risky to me. I've already played around with limine enough that I know I don't want to install much in the initramfs...
TPM is probably the best solution here. The key can be automatically fetched on reboot unless the boot order is changed or the drive is put in another computer.
Realistically for a home server what you are worried about is someone breaking in and selling your drives on Facebook marketplace rather than the FBI raiding your nextcloud server. So TPM automated unlock is perfectly sufficient.
FYI your decryption key can be MITMed during this process by anyone with physical access to the system, which defeats the purpose of encrypting the disk in the first place.
Police show up and arrest you. Could be with reason, could be by accident. Maybe you did something wrong, maybe you didn’t. They also physically size your servers, and in doing so they unplug the system.
If you have disk encryption, your data now requires the police to force you to produce a password, which may or may not be within their powers, depending on the jurisdiction.
It’s strictly better to have full disk encryption and remote unlocking than no disk encryption at all, because it prevents such „system was switched off by accident“ attacks.
Maybe I have a server at home, with a locked cabinet and vibration sensors, that houses a server or two and they all use full disk encryption, but I still want to be able to reboot them without having to connect a physical keyboard to them. So no one has physical access, not even me, but I still want to be able to reboot them.
Or countless of other scenarios where it could be useful to be able to remotely unlock FDE.
That's not a counter-argument. You are protecting the physical access, and your threat model doesn't include someone willing to bypass your locks and sensors. (or it does and you just didn't go into those details.)
The argument was that physical access gives up the FDE key.
I'd love to see this in the bootloader, along with a selection of binaries useful for recovery. Might sound silly but over the years I have had many a remote system get to the bootloader and then no further after an upgrade. Nowadays we've usually got a nicely sized EFI partition, why not stuff it all in there? Gimme a full Linux userspace from the bootloader, it would feel luxurious when I'm up at 3 am trying to recover a broken system halfway across the country.
Or is there already a solution to this that I've been missing? (Yeah, KVM/IPMI/etc, I know, but not all hosters make it easy to get to that.)
> Because initramfs is just a (mostly) normal Linux system, that means it has its own init PID 1. On Arch, that PID is in fact just systemd.
Debian has (or had; at least my Devuan still has) a simple shell script as first init. Was an interesting read and helped me understand were to add my remote rootfs decryption.
The `base` hook installs the shell PID 1, the `systemd` hook installs systemd as PID1. The default hook setup was changed with the latest'ish release to default too the `systemd` hook setup.
A long time ago, I built my own crashcart adapter with a raspberry pi and a teensy to do something similar. I would sometimes get weird mdadm errors that would hang the boot process and other times, a reboot or power loss wouldn't actually cause the PC to boot back up. The teensy did USB HID emulation for keyboard inputs. I added the ability to push the power button with a fet and some resistors. I had a cheap VGA to composite adapter going into a USB composite capture device so I could at least get screenshots for any weird boot messages. I built a small webpage using flask to display the screenshot, allow for text input, control inputs, and to push the power button. It was a lot of fun building but a basement flood completely wrecked it. Server was sitting on a 6in platform but the crashcart had fallen off the top of the case and was laying on the ground. Oops.
There is an old but still reasonable solution with mkinitcpio hooks encrypt/sd-encrypt + ssh, which is very easy to set up with EFI or grub2 onward. Tailscale is probably overkill for this use case, given that you're already exposing pre-/early- boot to the network by setting up interfaces that early. This became much more hermetic with secureboot and TPMs, too.
TPM definitely rises the effort by a lot to break it. But by default the communication with it is not encrypted, so especially for modules not built into the cpu wire/bus-tapping is a thing.
I currently have dropbear-ssh presenting the LUKS password prompt on my home server, but that has the very annoying quality that there's no way to do it from the console if you set that up too.
It's not a huge problem but it certainly means some recovery scenarios would be painful.
I'm vaguely reminded of some of the third party disk encryption/preboot management utilities that exist in the Windows space that leverage similar technology. Authentication is done against an online source, and only then is the key sent back to the local machine to unlock the disk. The Bitlocker key is kept nowhere near the local TPM.
I've only seen it on some paranoid-level devices in industry (typically devices handling biometric identity verification services).
IIRC this one is a Linux image that boots up, unlocks the normal Bitlocker partition via whatever mechanism you need, then hands control back to the Windows bootloader to continue onwards.
I have a very similar setup to the author, but instead of running Tailscale in my initramfs, I have a Raspberry Pi sitting next to the home server (which is on my Tailscale network) and I use it like a bastion host. Process is something like:
1. SSH into the Pi
2. Issue the Wake-on-LAN packet to boot the server
3. Tunnel that server's SSH port to my laptop
4. SSH into the initramfs SSH server (I use TinySSH), enter the decryption key
5. Wait for server to come up, then access via Tailscale
This is more complicated than the author's setup in that it requires another device (the Pi), but it's simpler in that you don't need to have the initramfs on your Tailnet.
If you want to be able to reboot remotely, and non-interactively (i.e. while you sleep), I (and one other person) created Mandos for this purpose: <https://www.recompile.se/mandos>. It gets the password over the network. If this seems insecure to you, read the FAQ: <https://www.recompile.se/mandos/man/intro.8mandos>
Mandos works with initramfs images created by both initramfs-tools and dracut, and is present in Debian since 2011, so no need to use a third-party package.
I haven't looked deeply into either, but how does this compare to the combination of Clevis and Tang that e.g. Red Hat/Fedora seems to favor?
How does this compare to: https://docs.redhat.com/en/documentation/red_hat_enterprise_...
Why is this needed at all? As the decrypted key is in memory before the reboot, can’t it just be written to a know location in memory and have kexec be instructed to read it early on?
You’re assuming a controlled reboot. Mandos was initially created to deal with intermittent power failures. It’s also good for kernel panics.
Oh for sure something is needed for a full start from zero. But the much more common case for a computer with backup power is regular restarts after applying patches that require a reboot. Would be much more pleasant for that to work out of the box with no manual interaction at all.
Good FAQ, clearly stating the weak point of physical access. For a server that threatmodel can work, for a fleet of edge/iot devices in unsecured locations without permanent uptime there is no real solution to be expected without custom silicon logic (like in smartcards) on the soc.
I have something similar set up to unlock the drives on my home server. Just the SSH in initramfs though, tailscale is pretty cool.
I've done stuff with mkinitcpio / initramfs on arch before, can't remember exactly what for. I still run arch on my main laptop. I'm running nixos on my home server though, and adding something like this is so easy by comparison.
Glad to see another example of this! Remote unlocking of your personal server's encrypted hard drive is PITA.
Other options that I've investigated that involve having a second server:
* A second server with Tang, and Clevis in the initramfs OS
* Keylime
Putting tailscale in the initramfs, and then updating the certs on a frequent enough schedule, seems risky to me. I've already played around with limine enough that I know I don't want to install much in the initramfs...
TPM is probably the best solution here. The key can be automatically fetched on reboot unless the boot order is changed or the drive is put in another computer.
Realistically for a home server what you are worried about is someone breaking in and selling your drives on Facebook marketplace rather than the FBI raiding your nextcloud server. So TPM automated unlock is perfectly sufficient.
This has been a "need" for a long time for home users with FDE. The go-to software solution for Linux has been:
https://github.com/gsauthof/dracut-sshd
Not sure how relevant it is, but I use dropbear to remote unlock over ssh: https://www.cyberciti.biz/security/how-to-unlock-luks-using-...
FYI your decryption key can be MITMed during this process by anyone with physical access to the system, which defeats the purpose of encrypting the disk in the first place.
Just use dm-verity for remote servers.
Police show up and arrest you. Could be with reason, could be by accident. Maybe you did something wrong, maybe you didn’t. They also physically size your servers, and in doing so they unplug the system.
If you have disk encryption, your data now requires the police to force you to produce a password, which may or may not be within their powers, depending on the jurisdiction.
It’s strictly better to have full disk encryption and remote unlocking than no disk encryption at all, because it prevents such „system was switched off by accident“ attacks.
If only everyone shared the same use case :)
Maybe I have a server at home, with a locked cabinet and vibration sensors, that houses a server or two and they all use full disk encryption, but I still want to be able to reboot them without having to connect a physical keyboard to them. So no one has physical access, not even me, but I still want to be able to reboot them.
Or countless of other scenarios where it could be useful to be able to remotely unlock FDE.
That's not a counter-argument. You are protecting the physical access, and your threat model doesn't include someone willing to bypass your locks and sensors. (or it does and you just didn't go into those details.)
The argument was that physical access gives up the FDE key.
Security isn't a binary boolean though.
I'd love to see this in the bootloader, along with a selection of binaries useful for recovery. Might sound silly but over the years I have had many a remote system get to the bootloader and then no further after an upgrade. Nowadays we've usually got a nicely sized EFI partition, why not stuff it all in there? Gimme a full Linux userspace from the bootloader, it would feel luxurious when I'm up at 3 am trying to recover a broken system halfway across the country.
Or is there already a solution to this that I've been missing? (Yeah, KVM/IPMI/etc, I know, but not all hosters make it easy to get to that.)
> Because initramfs is just a (mostly) normal Linux system, that means it has its own init PID 1. On Arch, that PID is in fact just systemd.
Debian has (or had; at least my Devuan still has) a simple shell script as first init. Was an interesting read and helped me understand were to add my remote rootfs decryption.
https://salsa.debian.org/kernel-team/initramfs-tools/-/blob/...
`mkinitcpio` supports both.
The `base` hook installs the shell PID 1, the `systemd` hook installs systemd as PID1. The default hook setup was changed with the latest'ish release to default too the `systemd` hook setup.
Shell `init`; https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio...
A long time ago, I built my own crashcart adapter with a raspberry pi and a teensy to do something similar. I would sometimes get weird mdadm errors that would hang the boot process and other times, a reboot or power loss wouldn't actually cause the PC to boot back up. The teensy did USB HID emulation for keyboard inputs. I added the ability to push the power button with a fet and some resistors. I had a cheap VGA to composite adapter going into a USB composite capture device so I could at least get screenshots for any weird boot messages. I built a small webpage using flask to display the screenshot, allow for text input, control inputs, and to push the power button. It was a lot of fun building but a basement flood completely wrecked it. Server was sitting on a 6in platform but the crashcart had fallen off the top of the case and was laying on the ground. Oops.
I recently ported the WiFi hook to support systemd based initramfs so you could autoconnect via your WiFi to unlock your device.
Give it a go: https://aur.archlinux.org/packages/mkinitcpio-wifi
There is an old but still reasonable solution with mkinitcpio hooks encrypt/sd-encrypt + ssh, which is very easy to set up with EFI or grub2 onward. Tailscale is probably overkill for this use case, given that you're already exposing pre-/early- boot to the network by setting up interfaces that early. This became much more hermetic with secureboot and TPMs, too.
TPM definitely rises the effort by a lot to break it. But by default the communication with it is not encrypted, so especially for modules not built into the cpu wire/bus-tapping is a thing.
https://news.ycombinator.com/item?id=46676919
back before tailscale got good, I used to use tor to remote-unlock on my nixos tower. It worked quite well! https://nixos.wiki/wiki/Remote_disk_unlocking
Man, that's pretty excellent!
I once built a demo-ish encrypted network boot system using similar initrd techniques. It's a fun hack working in the preboot environment.
I currently have dropbear-ssh presenting the LUKS password prompt on my home server, but that has the very annoying quality that there's no way to do it from the console if you set that up too.
It's not a huge problem but it certainly means some recovery scenarios would be painful.
I'm vaguely reminded of some of the third party disk encryption/preboot management utilities that exist in the Windows space that leverage similar technology. Authentication is done against an online source, and only then is the key sent back to the local machine to unlock the disk. The Bitlocker key is kept nowhere near the local TPM.
I've only seen it on some paranoid-level devices in industry (typically devices handling biometric identity verification services).
IIRC this one is a Linux image that boots up, unlocks the normal Bitlocker partition via whatever mechanism you need, then hands control back to the Windows bootloader to continue onwards.
https://winmagic.com/en/products/full-disk-encryption-for-wi...