How to manage SSH keys is something any developer working with servers had to think about. This is the story between me and SSH keys over the past six years, and where I’m now with my current setup. I hope my experiences will inspire you.

The Start: My First VPS

It started in 2018 when I rented my first Virtual Private Server (VPS). My hosting provider had provisioned a basic username/password login. I have used it probably longer than I should have, but eventually it became too inconvenient as I didn’t save the password in PuTTY.

At the time, I was still using PuTTY and Windows. I generated my first SSH key using PuTTY’s key generator (PuTTYgen) and synced it with Dropbox to my laptop. Looking back, that was not so secure, but life was simple then.

Public Key SSH authentication works through a pair of cryptographic keys:

• private key - this must be kept secure and secret
• public key - this can be shared freely

When connecting:

• The server uses your public key to create an encrypted challenge
• Only your matching private key can decrypt and solve this challenge
• Successfully solving the challenge proves your identity without sending your private key to the server

This provides secure authentication since the private key never leaves your computer and the public key alone cannot be used to gain access. More detailed RFC 4252

The Complications of Dual-Booting

In 2020 I dual-booted my desktop system with Linux, and noticed that OpenSSH is a thing, and that PuTTY’s key format is incompatible with it. I also discovered that storing SSH keys in Dropbox was not optimal, this presented two challenges:

1. Converting the PuTTY key to the OpenSSH format, which Linux uses natively.
2. How should I sync my SSH keys going forward

Luckily, PuTTY has an easy way of converting keys to the OpenSSH format (stackoverflow). I also noticed that I only needed to sync my keys once, as they don’t change over time. My solution was to mount my windows drive into Linux and just copy it over.

One-Key-Per-Device

During the end of 2020 I was ready to switch my Laptop to Linux as well. As I felt my current approach was not superb, I researched a little. And found out that it is not common practice to sync keys (because if one device is compromised, so is your private key on all other devices), rather each device has its own private key, maybe even encrypted at rest for bonus points.

My approach evolved to generating a unique SSH key for each device (more precisely for each OS). This strategy worked well, but introduced new challenges:

• Server management: How to keep all my public keys deployed on my servers
• Backup concerns: What happens if I lose access to a device

My Solution was to use GitHub as my source of truth, this meant that only the public keys retrievable on that website are valid. While I set up cron jobs on all my servers to automatically update my authorized_keys file (which stores the valid public keys for my user), deploying the newly added keys daily. This also meant that newly generated keys weren’t immediately usable, but the approach worked well enough.

When I had to reinstall my laptop, this system’s limitation became clear. I was locked out until the next cron job ran to deploy my newly generated public key. This did not spark joy, highlighting the need for a better key management strategy.

Hardware Tokens

While researching better solutions, I discovered hardware tokens and their ability to function as an SSH key while being an actual physical key. I really liked this idea, as it would solve the bootstrapping problem of new devices, or allow me to authenticate to servers on devices I do not own.

In October 2022, an opportunity arose: Cloudflare and YubiKey partnered and offered YubiKey 5’s for around €10. I did not hesitate and bought two.

SSH with YubiKey

YubiKeys are hardware security tokens that support various authentication/security protocols, including FIDO2, WebAuthn and OpenPGP. Their main security feature, compared to a USB drive, is that the cryptographic keys on a hardware token can not be read after they’re written. This means the key can be used across multiple devices while maintaining hardware protection.

For SSH authentication, the YubiKey 5 offers two approaches:

1. ed25519-sk keys

   • (+) Modern, simpler setup
   • (-) Limited support in some systems
   • (-) SSH keys can’t be shared between multiple YubiKeys

2. OpenPGP-based (RSA) Keys

   • (+) Widespread compatibility
   • (+) Keys can be shared between YubiKeys
   • (-) More complex initial setup
   • (-) Requires GPG SSH Agent

ed25519-sk combines the ed25519 elliptic curve algorithm with FIDO/U2F security keys (hence -sk). The private key material never leaves the security key hardware. Instead, the key performs the cryptographic operations internally when touched, signing a challenge from the SSH server.

PGP uses a system of master and sub keys, each serving distinct roles. The master key (certification key) is your core identity. It signs/certifies other keys and creates Subkeys. Think of it as your digital passport, rarely used but proves who you are. Then there are Subkeys which handle daily operations:

• (S)igning: Validates message authenticity, e.g. git commits
• (E)ncryption: Secure files/messages,
• (A)uthentication: Proves identity for services, e.g. SSH

The Master key creates and certifies subkeys, and stays offline and secure. This separation ensures compromised sub keys don’t affect your core identity. Sub keys can be revoked and replaced while maintaining the same trusted master identity. All keys can also have optional expiry dates, forcing you to rotate or renew expired keys.

I initially chose the ed25519-sk approach. I followed the official guide and created new SSH keys on both YubiKeys. Then added both to my servers while maintaining my old keys as a fallback.

This hybrid approach revealed some challenges:

• The physical requirement to insert the YubiKey sometimes led to me taking shortcuts as I still had my old keys
• Compatibility issues with services like Launchpad.net, which didn’t support ed25519-sk keys
• Because the key can by design not leave the YubiKey, how should I approach Backups. Buy more YubiKeys or store non ed25519-sk keys on USB drives?

My prior bootstrapping problems vanished, but because I was not convinced having only two separate YubiKey’s is a perfect backup. This led to me never removing my legacy SSH keys on my hosts. I mean, I could have added a cold SSH key on encrypted USB sticks, but I wanted to move away from caring and deploying multiple public keys.

The OpenPGP Solution

After experiencing these limitations, I revisited the OpenPGP approach, and the desire for a single, unified key solution with proper backups convinced me. I also wanted to start signing git commits, which OpenPGP also supports.

In 2023, I discovered drduh’s YubiKey Guide, an exceptionally detailed resource for implementing this setup. In summary, you boot an air gapped live ISO with the necessary packets installed. Generate your OpenPGP certify key plus all the subkeys for authentication, encryption and signing. You then write the subkeys onto your YubiKey(s), and also create (encrypted) backups of the keys on n drives. Then you can go back to your normal environment, and share your public keys (GPG and authentication key) on your favorite developer platforms.
Note: For SSH authentication, you need to change your SSH agent to gpg-agent. This solution addressed all my prior pain points, and so I configured my YubiKeys.

My Current Setup

Today, I maintain two YubiKeys configured with identical subkeys, all of them are rsa 4096 key with an expiry time of 1 year, meaning I have to extend them yearly, and also have a chance to rethink my current key management. I store encrypted backups of the raw key material on USB drives in multiple physical locations, ensuring I’m protected against both hardware failure and physical loss.
The two things I now need to do regularly are the following.

Bootstrap new device

• Install Dependencies
• Import and assign ultimate trust to my identity (public GPG key)
• Set gpg-agent as my SSH_AUTH_SOCK
• Set git to use my GPG key for signing

Yearly renew of my subkeys

• Start live iso with all Dependencies
• Extend expiry by 1 year on all subkeys
• Create new public gpg key
• Distribute public key to GitHub, GitLab, etc.

Conclusion

The journey to proper SSH key management has been long, but it was worth it. I no longer fear system reinstallation or loosing my devices. My current setup provides both security and convenience for me.

Now, looking back, I appreciate how these small iterations have lead to something amazing.

Security Notes

There have been some incidents in the past and present, where YubiKeys shipped with firmware flaws, making them vulnerable to some attacks Luckily, OpenPGP with rsa keys were not part of these CVE’s. Otherwise, I would have to buy a new YubiKey, because the firmware can not be altered due to security risks. This also all depends on your personal threat model.

• YubiKey Weak RSA 2017
• YubiKey ECDSA Private Key Recovery 2024