Some time ago, I was describing how to configure networking crash dumps. In that post, I mentioned that there is also the possibility to encrypt crash dumps. Today we will look into this functionality. Initially, it was implemented during Google Summer of Code 2013 by my friend Konrad Witaszczyk, who made it available in FreeBSD 12. If you can understand Polish, you can also look into his presentation on BSD-PL on which he gave a comprehensive review of all kernel crash dumps features.

The main issue with crash dumps is that they may include sensitive information available in memory during a crash. They will contain all the data from the kernel and the userland, like passwords, private keys, etc. While dumping them, they are written to unencrypted storage, so if somebody took out the hard drive, they could access sensitive data. If you are sending a crash dump through the network, it may be captured by third parties. Locally the data are written directly to a dump device, skipping the GEOM subsystem. The purpose of that is to allow a kernel to write a crash dump even in case a panic occurs in the GEOM subsystem. It means that a crash dump cannot be automatically encrypted with GELI.

To protect your privacy, you should encrypt crash dumps—and FreeBSD now offers such a possibility. The functionality is called EKCD, from the Encrypted Kernel Crash Dumps.  The idea behind is that the administrator or security officer in your company generates a RSA key. The public key is provided to the machine. When the key is loaded, the machine generates a one-time AES key, which is used to encrypt the actual crash dumps. The AES key is encrypted using an RSA key and stored on the machine. Only the owner of the private key can decrypt the AES key, hence the whole crash dump. If you are familiar with cryptography, this is pretty standard practice, because of the limitation of the RSA key.

Knowing all of this theory, let’s look now at how to configure it on the FreeBSD box. First of all, you have to check if your FreeBSD box has an EKCD build in. If you are using GENERIC kernel then you will have it, but you can verify it using, for example, the config(5) command:
$ config -x /boot/kernel/kernel | grep EKCD
options EKCD

The supervisor generates RSA keys, and the public key hides in the safe.
$ openssl genrsa -out private.pem 4096

You may be also interested in encrypting this key with options like aes256. For more details refer to the openssl man pages. After that read a public key from it:
$ openssl rsa -in private.pem -outform PEM -pubout -out public.pem

Now you have to transfer the public.pem to the remote machine, on which you want to encrypt dumps, and import the key:
$ scp public.pem w1:
# dumpon -k public.pem /dev/ada0

And know you have to wait until a crash occurs. That may take a while, but for the test we can panic a kernel ourselves with the following sysctl. Remember to test it only in a safe environment when no critical operation is being performed.
# sysctl debug.kdb.panic=1

When the machine comes back, we can use savecore to save the encrypted core dump from the device. It will create a couple of files:

• key.X - a AES key encrypted with the RSA public key
• vmcore_encrypted.X - an encrypted with AES key core dump

Then we can transfer files to the machine with the private key, and use the decrypt core to decrypt the core.
# savecore /var/crash /dev/ada0
# decryptcore -p ./private.pem -k ./key.0 -e ./vmcore_encrypted.0 -c vmcore.0

After all of that we can analyze our dump which, in this example, is in the vmcore.0.

The EKCD is a pretty interesting new feature in FreeBSD 12. Thanks to it we may be able to protect the privacy of our users on sharing hosting, or generally in our company. The configuration is straightforward, so there is no real reason not to use it.