SSH keys

An SSH login is a pair of files and a proof. The private file stays on the machine you type at. The server stores one public line, in the account you want to enter. It never receives the private file. It sends data from this connection and checks a signature. Password login is the other method: once the session is encrypted, the client sends the account password and the server checks it, usually through PAM. The password can be typed into the wrong box. The private key stays on the client, so it is not something you can paste into a prompt by mistake.

This note is OpenSSH on Linux, the ssh a Debian or Ubuntu server runs and the client a Mac speaks as well. Windows OpenSSH uses the same key text. An administrator account reads it from a different path, noted below. The aim is a key the server accepts, and a reading of the log when it does not.

The host key is not your login key

SSH checks the server first. On a new address the client shows the server host-key fingerprint and asks you to accept it. That pair belongs to sshd. The client stores what it accepted in ~/.ssh/known_hosts and compares later connections with it. A warning that the host key changed is this check, not a bad login key. Reinstalling the system changes the host key. Remove the stale line with ssh-keygen -R and the name or address, then check the new fingerprint by a channel you already trust.

Your key is considered only after that, inside the encrypted session. Time spent in authorized_keys will not clear a known_hosts mismatch.

The two files

Ed25519 is the key to generate. The comment is a label for ssh-add -l and the end of the public line. The server does not match it to the username. ada@laptop can log in as deploy.

ssh-keygen -t ed25519 -a 100 -C "ada@laptop" -f ~/.ssh/id_ed25519

-a 100 is the work factor used to encrypt the private file when you set a passphrase. You get two files. id_ed25519 is private, mode 600, owned by you. In the current format it begins with -----BEGIN OPENSSH PRIVATE KEY-----. That block does not belong on the server, in a ticket, in chat, or in git. id_ed25519.pub is a single line: ssh-ed25519, a Base64 blob, and the comment. That line is what you install. Someone who has only the .pub line still cannot log in.

The passphrase encrypts the private file on disk. It is not a server password and it is not sent to sshd. Leave it empty only for a key that must run unattended, and treat a copy of that file as a copy of the login. A laptop key should have a passphrase. ssh-keygen -lf ~/.ssh/id_ed25519.pub prints the fingerprint. ssh-keygen -y -f ~/.ssh/id_ed25519 prints the public line for a private file, which is the check that a copied pair still matches.

If something old cannot do Ed25519, ssh-keygen -t rsa -b 4096 -a 100 -C "ada@laptop" -f ~/.ssh/id_rsa still works. Retire a 1024-bit RSA key rather than offering it.

Use a separate key for each place the private file has to live. The laptop, a build runner, and a backup host are three files, and three lines in authorized_keys. When the runner is retired you delete its line and the laptop key keeps working. One key copied onto every machine is one leak away from rotating all of them, including the laptop you still have. The comment on each public line should say which machine it is, because that label is the only thing that distinguishes the lines once they are installed. The server still does not enforce the comment. You do, when you decide which line to remove.

Password login and key login

Both run after the host key is accepted. Password authentication sends the password. A guess, a reused password, and a prompt on the wrong host are all the same protocol working as designed. You can turn it off.

Key authentication offers a public key and a signature. A wrong passphrase fails on the laptop, before any login packet, because the client cannot unlock the file. A public key the server does not hold fails on the server. Leave both methods on while you install the key. Prove it in a second terminal with password authentication switched off for that one command, so a successful login cannot be the password in disguise:

ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes -o PreferredAuthentications=publickey deploy@203.0.113.10

If that command prints Permission denied (publickey), the key is not accepted yet. A password prompt during an ordinary ssh has not answered the question. Only when the command above opens a session should you install the end state:

PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
AuthenticationMethods publickey

PasswordAuthentication no is not sufficient on a PAM system. Keyboard-interactive can still ask for the account password. Set KbdInteractiveAuthentication no, or require AuthenticationMethods publickey. PubkeyAuthentication already defaults to yes. Test the config, then reload. A reload does not drop the session you have open, which is the one you use if the file is wrong:

sudo sshd -t && sudo systemctl reload ssh

The unit is ssh on Debian and Ubuntu, and sshd on RHEL and Fedora. If sshd -t fails, do not reload.

sshd -t only checks that the file parses. It does not tell you which of two conflicting lines won. For most keywords the first value sshd reads is the one that sticks. Debian and Ubuntu pull in /etc/ssh/sshd_config.d/*.conf from an Include near the top of sshd_config, so a drop-in is read before the main file. A drop-in that sets PasswordAuthentication yes wins over a PasswordAuthentication no typed later in sshd_config. sudo sshd -T | grep -Ei 'passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication|authenticationmethods' prints the values that will actually be used. Put a local override in a file that sorts first, such as /etc/ssh/sshd_config.d/00-local.conf, and run sshd -T again before the reload.

authorized_keys

The default file is ~/.ssh/authorized_keys of the account you log into. AuthorizedKeysFile can move it. One public key per line. Options, if you need them, come before the type. # comments and blank lines are ignored.

From the laptop, while a password still works:

ssh-copy-id -i ~/.ssh/id_ed25519.pub deploy@203.0.113.10

That appends the .pub line named by -i and creates ~/.ssh if needed. The key it installs is the file you named, which is not necessarily the key the agent used to log in for the copy. Check the new line is the fingerprint you expect, with ssh-keygen -lf on the .pub file and the same command against the server's authorized_keys. A second run appends a duplicate. Harmless, and worth deleting. On the server, with the public file already copied up:

install -d -m 700 ~/.ssh
touch ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
cat id_ed25519.pub >> ~/.ssh/authorized_keys

The installed line is the .pub file with nothing wrapped around it:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... ada@laptop

That blob is the shape, not a key. Paste the real line, and keep it on one line. Folded Base64 is not a key sshd will parse.

The path is the account. A line in /home/deploy/.ssh/authorized_keys does nothing for ssh root@203.0.113.10. On Windows OpenSSH, administrators are read from C:\ProgramData\ssh\administrators_authorized_keys, with a tight ACL, not from the profile .ssh directory. The line itself is unchanged.

A backup key that should only run one program is still one line:

restrict,command="/usr/local/bin/backup-shop" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... backup

restrict turns off forwarding and the other extras. command replaces whatever the client requested. The client still has to hold the matching private key.

The agent, and which key is offered

The agent holds an unlocked key and signs on request, so a passphrase is typed once. Desktops often start one for you. ssh-add -l lists it. The agent has no identities means this shell's agent socket is empty.

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
ssh-add -l

Skip the eval when a desktop agent is already running. A second eval starts an empty agent and points this shell at it. The keys are still in the first one. AddKeysToAgent yes in ~/.ssh/config loads a key the next time you unlock it. ssh -A forwards the agent so the remote host can ask it to sign. Leave forwarding off unless you mean that host to use your key for the session.

The client offers id_ed25519, id_rsa, and whatever else the agent has. A file named shop_ed25519 is skipped until you name it. Too many agent keys hit MaxAuthTries (default 6) and the server closes with Too many authentication failures before the right key is tried. Pin the key:

Host shop
  HostName 203.0.113.10
  User deploy
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes

~/.ssh/config has to be owned by you and not writable by others. Mode 600 is the usual one. IdentitiesOnly yes stops the client offering every agent key. The one-shot form is ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes deploy@203.0.113.10. ssh -G shop prints the settings that will be used, including a Host * block you forgot was overriding IdentityFile.

ssh -v deploy@203.0.113.10 is the trace when -G already shows the right file and the login still fails. A working attempt offers the Ed25519 path, records Server accepts key, and finishes with Authenticated to 203.0.113.10 using "publickey". Offering public key: /home/ada/.ssh/id_rsa followed by Authentications that can continue: publickey,password means the server does not hold that public key for this account. No more authentication methods to try after only the wrong filenames means the client never offered id_ed25519. Fix the offer before you generate another key. The server log confirms the same event from the other side: Accepted publickey for deploy names the account that matched. Failed publickey for deploy means a key was tried and was not in that account's file.

Permissions

StrictModes defaults to yes. sshd then refuses the key if the home directory, .ssh, or authorized_keys is writable by anyone but the owner. Group write is enough. The client just sees a refusal or a password prompt. The server log has the path:

Authentication refused: bad ownership or modes for directory /home/deploy

The same sentence ending in authorized_keys is the file. Home created as 775, .ssh at 775, and an editor saving the file as 664 are the common cases. On RHEL and Fedora a copied file can also have the wrong SELinux label. restorecon -Rv ~/.ssh restores it after the modes are fixed.

chmod 755 /home/deploy
chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys
chmod 600 ~/.ssh/id_ed25519

755 on the home directory is normal. 775 fails the check. The last command is the client side: a private key at 644 is never sent.

Permissions 0644 for '/home/ada/.ssh/id_ed25519' are too open.

The public file may stay 644. Logs are journalctl -u ssh -S today on Debian and Ubuntu, journalctl -u sshd where the unit has that name, and /var/log/auth.log if rsyslog is collecting it. Accepted publickey for deploy is the key. Accepted password means a password, or keyboard-interactive, is still open.

Pitfalls

• The private file starts with BEGIN OPENSSH PRIVATE KEY or an older BEGIN RSA PRIVATE KEY. authorized_keys gets the one .pub line. A private key that has been pasted anywhere is finished: generate a new pair, install the new public line, delete the old one.
• A public line folded by mail, or a file named authorized_keys.txt, is not read as a key.
• Group-writable home, .ssh, or authorized_keys is refused while StrictModes is on. Read the path in the server log.
• A private key left at 644 is ignored on the client, before sshd is involved.
• The key is installed for deploy and the command is ssh root@.... The comment is not a username. Windows administrators are not read from the profile .ssh folder.
• An unusually named file is not offered without -i or IdentityFile. A crowd of agent keys trips MaxAuthTries. IdentitiesOnly yes is the fix.
• PasswordAuthentication no can leave a PAM password prompt via keyboard-interactive. Confirm with a new connection.
• Host-key warnings are known_hosts. Login-key warnings are authorized_keys.

ssh -G is the key you meant. ssh -v is the key you offered. The auth log is whether the server took it.