SSH CA certificates for hosts and users make using and administering SSH less tedious and more secure. An SSH key is used by the CA to sign host and user public keys resulting in certificates. SSH server and clients are configured to trust certificates that can be verified using specific public keys.

In short

To create an SSH CA key, stored in the file ca:

ssh-keygen -f ca -t ed25519

To create a host certificate:

ssh-keygen -s ca -I "example.org CAv1" -h -n shell.example.com host.key.pub

To create a user certificate:

ssh-keygen -s ca -I "example.org CAv1" -n tomjon tomjon.pub

To configure an SSH client to trust host certificates, add a line like this to the user ~/.ssh/known_hosts file:

@cert-authority *.example.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICSYauGmwHZr8cb7/XUjWREcboVER/irJNtIbY8AbP1H

To configure a server to trust user certificates, first put the public key in /etc/ssh/user-ca-keys:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINC5sITV4bWhfbteOC0z7OC1HqUrV+xI8JT3eC99feFH

Then add this to /etc/ssh/sshd_config:

TrustedUserCAKeys /etc/ssh/user-ca-keys

Then reload the SSH server (using systemd; use whatever command suits your init system):

systemctl reload sshd

Introduction

This document aims to explain the purpose and use of an SSH CA, or certificate authority, and using them for SSH host and user verification. The target audience is system administrators who want to make SSH simpler and more secure to use in their organization.

For practical reasons, this document is currently using examples of filenames and commands that work on Debian systems using systemd: the author only administers such systems. Help making sure they work on other systems is welcome.

Overview

Host certificates

Traditionally, in the world of SSH, servers have host keys that rarely change, but are generated separately for each host. Each user accessing a host for the first time, at a given address, the user is presented with the host’s key, and needs to manually, laboriously, and usually insecurely, check that it’s the right key for that host.

This can be a risky situation: if an attacker manages to trick the user’s SSH client to show a key the attacker has generated, and the user accepts it as the real one, the attacker can see – and change – all the traffic going over the SSH connection. This mostly nullifies the security benefit SSH is meant to provide. (Not entirely: the connection is still protected against other attackers.)

In a situation where there are many hosts, or hosts gets recreated frequently, or change address a lot, such as when using cloud technologies, the risky situation keeps happening frequently. Not only is it risky, it is also tedious and cumbersome to the user. If this keeps happening a lot, users are in effect trained to automatically accept all host keys.

The risky situation can be avoided by having the host keys be communicated to all users ahead of time, but doing this in a secure and convenient way is difficult. It is also unnecessary.

An SSH CA is a way to certify SSH host keys so that the user’s SSH client can trust it without asking the user to verify it. An SSH CA is an SSH key pair used to create host certificates. The client is configured to trust any host certificate that can be verified using the SSH CA public key. The CA public key still needs to be communicated to the user in a secure way, but the CA key is only one key and rarely changes, so the tiresome risky situation happens very rarely. After the user has the CA key, an attacker can’t trick the user to accept a false host key.

With host certificates, the client never needs to ask its user if the host key of a new host is valid, and the user never needs to try to verify it. If the host’s identity (host key or address) changes, such as when a VM is re-created, the client doesn’t need to bother the user about it.

Overall, this leads to a much smoother and more secure experience for people using SSH.

User certificates

Traditionally, a user authenticates to an SSH server using a password, or a user key. We will not discuss passwords here.

A user SSH key is an SSH key pair of which the user has the private part. The public part is added to the ~/.ssh/authorized_keys file on the server for the user’s account. At login time, the client proves to the server that it has a private key that corresponds to one of the public keys in that file, and this proves to the server that the user is authorized to log in. This is great, because the user does not need to remember a strong password, nor type it in every time they log in, and the server does not need to store the user’s password at all, even in an encrypted way.

The result is an easy, secure way for the user to log into the server. However, this only works if the list of authorized keys is kept up to date.

If the user needs to, for any reason, change to a new key, perhaps as part of a regular key rotation strategy, the list of authorized keys needs to be updated to add the new key, and remove the old key. The update procedure is often a risky, tedious step. If an attacker manages to get the attacker’s key into the list, they can log into the server as the user. Given that the authorized_keys file is usually user-editable, the user may add any SSH public keys to that file, including keys for other people, or keys stored on machines that are insecure. The user may do this intentionally, or because they’ve been tricked or coerced into doing it.

An SSH CA can create a user certificate, which ties a user’s SSH public key to a username. An SSH server can be configured to trust such certificates, made with specific CA keys, and to act as if that user’s public key is in their authorized_keys file, even if that file doesn’t exist. The result is that there is no need to maintain that file. This also means it’s feasible to revoke access with specific certificates.

Overall, this leads to system administrators having an easier way to control who has access their SSH servers.

Certificate automation

An organization that wishes to use SSH CA probably needs to find or build automation for issuing certificates. For host certificates this is fairly easy and can be done entirely by the system administrators.

For user certificates it’s harder. Your author doesn’t yet know of any good solutions here.

Creating certificates

Create an SSH CA key

Generate an SSH ca key pair using the following command:

ssh-keygen -f ca -t ed25519

This creates two files, named ca (private key) and ca.pub (public key), of type ed25519.

An SSH CA key is a normal SSH key and is created in the normal way. Keys of type ed25519 are smaller, and thus more convenient, but any type will work. Every type of CA key can certify every type of key.

Because CA keys are more valuable than most SSH keys, it’s best to store them especially securely and choose an especially strong passphrase. If possible, storing the key on a hardware token is probably best.

Create a certificate for a host or user

To create a certificate need the private or public key of the host or the user. The public key is preferred: there is no need to expose private keys to the SSH CA person or machinery. A host’s public keys is in the /etc/ssh directory (for example, /etc/ssh/ssh_host_ed25519_key.pub). A user’s public key is usually in their ~/.ssh directory (e.g., ~/.ssh/id_rsa.pub).

The certificate itself is created with the ssh-keygen command:

ssh-keygen -s ca -I "example.org CAv1" -n tomjon tomjon.pub
ssh-keygen -s ca -I "example.org CAv1" -h -n shell.example.com host.key.pub

The certificate is written to the tomjon-cert.pub or host.key-cert.pub file.

The -s filename option to ssh-keygen creates a certificate. The ca file has the CA private key, from above.

The -I description option sets a description of the certificate key. It gets logged by the server and can be used, for example, to document which SSH CA key is used.

The -h option tells the program to create a host certificate. Without this option, it creates a user certificate.

The -n hostname or -n username specifies the host or domain name, or the username, for which the the certificate is valid. For host certificates this is optional, but then the certificate is valid for any host. For user certificates, -n username is mandatory, or the server won’t know which user is authenticating. Multiple names can be given, separated by commas.

Using certificates

Configure SSH server to provide a host certificate

Copy the host certificate you created above as /etc/ssh/ssh_host_ed25519_key-cert.pub on the serve.

Edit the /etc/ssh/sshd_config file to add the following at the top (the HostKey must come before the HostCertificate):

HostKey /etc/ssh/ssh_host_ed25519_key
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

Reload or restart the SSH daemon:

systemctl reload sshd

The filenames above assume keys of type ed25519; adjust as needed. You can also use other names.

Configure an SSH server to accept user certificates

Create file /etc/ssh/user-ca-keys with a line for each SSH CA public key used to create user certificates:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINC5sITV4bWhfbteOC0z7OC1HqUrV+xI8JT3eC99feFH

Configure the SSH server to trust SSH CA keys listed in that file: edit /etc/ssh/sshd_config to contain the following line:

TrustedUserCAKeys /etc/ssh/user-ca-keys

Reload or restart the SSH server:

systemctl reload sshd

You can, again, use another name for the file with CA keys for user certificates.

Configure an SSH client to accept host certificates

Edit the user’s ~/.ssh/known_hosts file, or the system /etc/ssh/ssh_known_hosts file, to contain a line giving the SSH CA public key:

@cert-authority * ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICSYauGmwHZr8cb7/XUjWREcboVER/irJNtIbY8AbP1H

where everything after * comes from the SSH CA public key file (ca.pub in the example above). The * specifies which hosts the user trusts the CA for. Use a more restrictive pattern, such as *.example.com as needed.

This pattern is an additional limitation on top of the -n hostname restriction specified when creating the certificate. The SSH CA might not want to restrict the certificate, since they trust themselves to not issue certificates for hosts outside their organization, but a user has no need to trust them to not do that.

Configure an SSH client to use a user certificate

Get your user certificate from the SSH CA. Tell the SSH client to use it as if it were your public key. Store the certificate in ~/.ssh/id_xxx-cert.pub, where xxx is the type of your key. Alternatively, tell the client to use a specific certificate with the -i option.

ssh server.example.com
ssh -i ~/.ssh/my-cert.pub server.example.com

Assuming your SSH client has the private key corresponding to a the public key specified by the certificate , you can log in as the user specified in the certificate.

Features of certificates

An SSH certificate can be tied to one or more hosts or users, called principals. Use the -n option to ssh-keygen for this. For host certificates this is optional, but it’s mandatory for user certificates. If more than one principal is listed, separate them with commas. This can be useful when the same host can be reached using multiple names. The default is that the certificate is valid for any host, which seems rather insecure.

A certificate is valid for all time, by default. It can be more secure to limit the validity time of a certificate to be short and create a new certificate often. This limits how long a stolen certificate can be used, but requires generating new certificates regularly. See the -V option to ssh-keygen for details.

A certificate has a serial number. This is useful if the same CA creates multiple certificates for the same principal, and one needs to be revoked. See the -z option to ssh-keygen for details.

Revoking certificates

SSH has support for revoking certificates. However, to use that requires distributing revocation lists to clients, which is tedious and not something your author has experience with. A compromise would be to have a short validity time for a certificate, such as a week, and create and deploy a new certificate just before the validity time expires.

User certificates can also be revoked, and such revocations only need to be distributed the servers that may need to accept them, which is probably a simpler problem.

That probably means automating things, but that’s outside the scope of this document.

Thanks

While writing this, your author got feedback and reviews of drafts from David Leggett.