You paste the command a teammate sent, press enter, and the server says no.

$ ssh deploy@10.0.4.21
deploy@10.0.4.21: Permission denied (publickey).

That single line is the most common first wall in Linux remote access. The server is up, the network is fine, and it still will not let you in. By the end of this part you will know why the message appears and the exact steps that make it stop appearing.

What SSH actually does

SSH stands for secure shell. A program on your machine, the client, opens a connection to a program on the far machine, the server, called sshd. By default that connection lands on TCP port 22. Once it is open, everything you type and everything the server sends back travels inside an encrypted tunnel, so anyone sniffing the wire between you sees noise, not your commands or your password.

The tunnel gets built first. Only after the encryption is agreed does the server ask the real question: who are you, and can you prove it. That proof step is where password login and key login part ways.

Passwords work. Keys work better.

You can log in with the account password, and on a fresh server that is often the only option at first. It also means every bot on the internet that finds port 22 gets to guess that password a few thousand times an hour. Key based login replaces the guessable secret with a pair of files that belong together: a private key that never leaves your laptop, and a public key you hand to the server.

Make your first key pair

One command builds the pair. The comment after -C is just a label so you can tell your keys apart later.

$ ssh-keygen -t ed25519 -C "pranay@laptop"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/pranay/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/pranay/.ssh/id_ed25519
Your public key has been saved in /home/pranay/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:9Xk2p... pranay@laptop

Press enter to accept the default path. When it asks for a passphrase, type one. A passphrase means a stolen laptop does not hand over a working key. You now have two files: id_ed25519, the private half, and id_ed25519.pub, the public half you are allowed to share.

Why ed25519 and not RSA

Old guides tell you to run ssh-keygen with no flags or with -t rsa -b 4096. Ed25519 is the better default on any current system. A 256 bit ed25519 key gives you security in the same league as a 4096 bit RSA key, the keys are shorter, and the signing is faster. It is also the type modern OpenSSH picks on its own if you leave off -t. Reach for RSA only when you must talk to something genuinely old that has never heard of ed25519.

Get the public key onto the server

The server keeps a list of public keys it trusts in ~/.ssh/authorized_keys for each user. Put your public key on that list and the door opens. The tidy way is ssh-copy-id, which appends the key and fixes permissions for you.

$ ssh-copy-id deploy@10.0.4.21
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s)
deploy@10.0.4.21's password:
Number of key(s) added: 1

Now try logging into the machine, with:   "ssh deploy@10.0.4.21"
$ ssh deploy@10.0.4.21
Welcome to Ubuntu 24.04.2 LTS
deploy@web01:~$

That last password prompt is the account password, used one time to place the key. After this you log in with the key and never type that password again. Once the key works, turn password login off entirely, which is the next section.

When ssh-copy-id is not available, do it by hand. This does the same thing the tool does:

$ ssh deploy@10.0.4.21 "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys" < ~/.ssh/id_ed25519.pub

$ chmod 700 ~/.ssh
$ chmod 600 ~/.ssh/authorized_keys

Keep this map in your head. When a key login fails, walk down it and check each mode before you touch anything else. Directories need the execute bit so you can enter them, which is why they read 700 and not 600.

Stop typing the address every time

Once you manage more than two servers, remembering IP addresses and usernames gets old. The client reads ~/.ssh/config, where you name a host once and then connect by that name.

Host web01
    HostName 10.0.4.21
    User deploy
    IdentityFile ~/.ssh/id_ed25519
    ServerAliveInterval 60

Now ssh web01 is the whole command. ServerAliveInterval 60 sends a quiet keepalive every minute so the session does not die when you step away for coffee. This file is also where you copy files with scp web01:/var/log/app.log . using the same short name.

Lock down the server side

The server behaviour lives in /etc/ssh/sshd_config. Edit it as root, change a few lines, and reload sshd. These four changes carry most of the weight.

Test the file before you reload, so a typo does not lock everyone out: sshd -t prints nothing if the config is valid. Keep your current session open while you test the change from a second terminal. If the new session works, you are safe to close the old one.

The Ubuntu 24.04 reload trap

Here is the one that catches people. On RHEL family systems the service is sshd, and you apply changes with systemctl reload sshd. On Debian and Ubuntu the service is ssh, not sshd. From Ubuntu 22.10 onward the standard install also uses socket activation, so sshd only starts when a connection arrives. If you change Port and restart ssh.service, the server keeps listening on 22 because the socket unit still owns the port. On Ubuntu 24.04 you restart ssh.socket, or better, edit the socket with systemctl edit ssh.socket to set the new port there.

# Debian / Ubuntu
$ sudo sshd -t && sudo systemctl reload ssh

# RHEL / Rocky / Alma
$ sudo sshd -t && sudo systemctl reload sshd

An opinion on the port

Half the hardening guides online start with ‘move SSH off port 22’. I disagree that it counts as security. Moving to port 2222 cuts the noise in your logs from dumb bots, and that is a real, pleasant benefit. It stops a targeted attacker for about four seconds, because a port scan finds the new port immediately. The change that actually matters is PasswordAuthentication no. Do that one first. Treat the port move as log hygiene, not defence, and you will spend your effort in the right place.

The whole thing, start to finish

Here is a real sequence on a fresh server, including the failure you will probably hit and how it clears.

$ ssh-keygen -t ed25519 -C "pranay@laptop"
  ... key saved to ~/.ssh/id_ed25519 ...

$ ssh deploy@10.0.4.21
deploy@10.0.4.21: Permission denied (publickey).   # no key on the server yet

$ ssh-copy-id deploy@10.0.4.21
deploy@10.0.4.21's password:
Number of key(s) added: 1

$ ssh deploy@10.0.4.21
deploy@web01:~$ sudo sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
deploy@web01:~$ sudo sshd -t && sudo systemctl reload ssh
deploy@web01:~$ exit

$ ssh deploy@10.0.4.21          # still works, now key only
deploy@web01:~$

The first Permission denied is expected, not a bug. It is the server telling you it has no key for you yet. The copy step fixes it, and the reload closes the password door behind you.

$ ssh-keygen -t ed25519 -C "test-key"
$ ssh-keygen -lf ~/.ssh/id_ed25519.pub
256 SHA256:9Xk2p... test-key (ED25519)

Questions I actually get

Do I need a separate key for every server?
No. One key pair can be trusted by any number of servers, because the same public key can sit in many authorized_keys files. Some people keep separate keys per role or per client for cleaner revocation, which is reasonable, but it is a choice, not a rule.

Can I copy my private key to my other laptop?
Avoid it. The fewer copies of a private key exist, the fewer places it can leak from. Generate a fresh key on the second laptop and add its public key to the servers you need. Two identities are easier to reason about than one shared secret on two disks.

The connection freezes when I leave it idle. Why?
A firewall or NAT in the middle dropped the idle session. Add ServerAliveInterval 60 to your ~/.ssh/config so the client sends a small packet every minute and the path stays open.

Is PuTTY still a thing?
On Windows, yes, though you no longer need it. Windows ships the OpenSSH client now, so ssh user@host works from PowerShell the same way it does on Linux. PuTTY is fine if you like its saved sessions, but the commands in this part run natively.

What is the host key warning the first time I connect?
SSH shows the server's fingerprint and asks you to confirm it. That is the client learning the server's identity so it can warn you later if the server ever changes underneath you. Say yes on a machine you trust, and it is remembered in ~/.ssh/known_hosts.

Where to take this next

SSH is the door to every server in the rest of this series and the rest of your career. The moment you can log in reliably, everything else opens up: editing config files, reading logs, running services. Next comes staying productive once you are inside, starting with the editor that is always installed and always intimidating at first.

Set up key login on one machine today, even a spare laptop, and turn off its password login. Doing it once by hand teaches you more than reading ten guides. If you are heading toward cloud or virtualization work, the same keys get you into cloud instances and ESXi hosts, so this skill carries straight across to the Cloud for Beginners and VMware for Beginners guides.

References

OpenSSH ssh-keygen manual
OpenSSH sshd_config manual
Ubuntu Server, OpenSSH server

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts