How to Use Secure Shell (SSH) for Remote Logins in Linux: Complete Connection and Security Guide

SSH is probably the single Linux tool I’ve used more than any other, full stop. It’s how I manage every remote server I touch, how I move files, how I tunnel traffic through restrictive networks, and how I make automation scripts run unattended without ever storing a plaintext password anywhere. This guide covers SSH from a first connection through key-based auth, hardening the daemon, and the productivity features (tunneling, agent forwarding, config files) that make it genuinely pleasant to work with once you know them.

What SSH Actually Does

SSH (Secure Shell) provides an encrypted channel between a client and a server for remote command execution, file transfer, and port forwarding. Under the hood, an SSH session goes through:

1. TCP connection to the server, typically port 22.
2. Key exchange — client and server negotiate a shared session key using asymmetric cryptography (commonly Diffie-Hellman or elliptic-curve variants), establishing a symmetric encryption key for the actual session without ever transmitting it.
3. Host verification — the client checks the server’s host key against its known_hosts file, protecting against man-in-the-middle attacks.
4. Authentication — password, public key, keyboard-interactive (2FA), or GSSAPI/Kerberos.
5. Session — once authenticated, all further traffic (interactive shell, file transfer, port forwarding) is encrypted inside this single channel.

Basic Connection

ssh user@remotehost
ssh user@192.168.1.50
ssh -p 2222 user@remotehost     # non-standard port

First connection to a new host:

The authenticity of host 'remotehost (192.168.1.50)' can't be established.
ED25519 key fingerprint is SHA256:abcd1234...
Are you sure you want to continue connecting (yes/no/[fingerprint])?

This fingerprint check matters — in a genuinely secure workflow, you’d verify this fingerprint against one provided by the server administrator through a separate channel before typing “yes,” rather than accepting blindly.

Key-Based Authentication (What You Should Actually Be Using)

Password authentication over SSH is encrypted, but it’s still vulnerable to brute-force attempts and doesn’t scale well for automation. Key-based authentication is both more secure and more convenient.

Generate a Key Pair

ssh-keygen -t ed25519 -C "your_email@example.com"

• -t ed25519 — modern, fast, and secure key type (preferred over the older rsa type for new keys)
• -C — a comment, usually your email, to help identify the key later

For environments requiring RSA specifically (some older systems, some compliance requirements):

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

You’ll be prompted for a passphrase — use one. An unencrypted private key sitting on disk is a single-file compromise away from full account takeover; a passphrase-protected key at least requires an attacker to also crack the passphrase.

Copy the Public Key to the Server

ssh-copy-id user@remotehost

Or manually, if ssh-copy-id isn’t available:

cat ~/.ssh/id_ed25519.pub | ssh user@remotehost "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

Test It

ssh user@remotehost

You should be prompted for your key’s passphrase (if set) instead of the account’s server-side password — and if you’re using ssh-agent (below), not even that.

ssh-agent: Avoid Retyping Your Passphrase

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

Now your passphrase is cached in memory for the duration of the agent’s life (or until you explicitly remove it), and every SSH connection using that key authenticates silently.

List currently loaded keys:

ssh-add -l

Remove all cached keys:

ssh-add -D

The SSH Config File: Making Connections Effortless

~/.ssh/config lets you define shortcuts and per-host settings instead of typing full connection strings every time:

Host prodweb
    HostName 203.0.113.10
    User deploy
    Port 2222
    IdentityFile ~/.ssh/id_ed25519_prod

Host *.internal.example.com
    User admin
    ProxyJump bastion.example.com

Host bastion
    HostName bastion.example.com
    User jump-user
    IdentityFile ~/.ssh/id_ed25519_bastion

Now instead of ssh -p 2222 -i ~/.ssh/id_ed25519_prod deploy@203.0.113.10, you just run:

ssh prodweb

The ProxyJump directive is genuinely one of my favorite SSH features — it transparently routes your connection through a bastion/jump host, which is an extremely common production pattern for reaching private-network servers without exposing them directly to the internet.

Copying Files: scp and sftp

scp localfile.txt user@remotehost:/tmp/
scp user@remotehost:/var/log/app.log ./
scp -r localdir/ user@remotehost:/tmp/localdir/

Interactive file transfer session:

sftp user@remotehost
sftp> put localfile.txt
sftp> get remotefile.txt
sftp> ls
sftp> cd /var/log

For serious file syncing, rsync over SSH is generally the better tool — it only transfers changed data:

rsync -avz -e ssh /local/dir/ user@remotehost:/remote/dir/

Port Forwarding (Tunneling)

Local Forwarding — Reach a Remote-Only Service Through Your Local Machine

ssh -L 8080:localhost:80 user@remotehost

Now http://localhost:8080 on your machine reaches port 80 on the remote host — useful for reaching an internal admin panel that isn’t directly exposed.

Remote Forwarding — Expose a Local Service to the Remote Side

ssh -R 9000:localhost:3000 user@remotehost

Now connections to port 9000 on the remote host reach port 3000 on your local machine — useful for letting a remote server temporarily reach a service running on your laptop during development.

Dynamic Forwarding — a SOCKS Proxy Through SSH

ssh -D 1080 user@remotehost

Configure your browser or another application to use localhost:1080 as a SOCKS5 proxy, routing that traffic through the encrypted SSH tunnel — a lightweight, no-additional-software VPN alternative for browsing through a trusted remote network.

Running Commands Without an Interactive Session

ssh user@remotehost "df -h"
ssh user@remotehost "systemctl status nginx"

Useful in scripts, combined with key-based auth for fully unattended automation:

#!/bin/bash
ssh backup-server "tar czf /backup/daily-$(date +%F).tar.gz /data"
scp backup-server:/backup/daily-$(date +%F).tar.gz /local/backups/

Hardening the SSH Server (sshd_config)

Configuration lives at /etc/ssh/sshd_config. Key settings I check on every server I stand up:

# Disable root login entirely — use sudo after logging in as a normal user
PermitRootLogin no

# Disable password auth once keys are set up and confirmed working
PasswordAuthentication no

# Only allow protocol version 2 (version 1 is ancient and broken; modern sshd doesn't even offer it, but explicit is good)
Protocol 2

# Limit which users/groups can SSH in at all
AllowUsers deploy admin

# Change the default port (mild obscurity benefit — reduces automated scan noise, not a real security boundary on its own)
Port 2222

# Disable empty passwords
PermitEmptyPasswords no

# Limit authentication attempts per connection
MaxAuthTries 3

# Set an idle timeout
ClientAliveInterval 300
ClientAliveCountMax 2

Apply changes:

sudo sshd -t                     # test config syntax before restarting — always do this
sudo systemctl restart sshd

Test the new configuration from a second, separate SSH session before closing your current one. This is the single most important operational habit for SSH hardening — if PasswordAuthentication no gets applied before you’ve confirmed key-based auth actually works, and your current session drops, you may be locked out entirely without console access.

Multi-Factor Authentication with SSH

For an additional layer beyond key-based auth, PAM-based TOTP (via Google Authenticator PAM module) is a common addition:

sudo apt install libpam-google-authenticator    # Debian/Ubuntu
sudo dnf install google-authenticator           # RHEL/Fedora

google-authenticator

Then in /etc/pam.d/sshd, add:

auth required pam_google_authenticator.so

And in sshd_config:

ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

This requires both a valid SSH key and a TOTP code, combining something-you-have with something-you-know.

A Deeper Look at the SSH Protocol Handshake

Understanding the sequence a bit more precisely helps when debugging connection issues with -vvv, since the verbose output maps directly onto these stages:

1. Version exchange — client and server each announce their SSH protocol version and software identifier.
2. Algorithm negotiation — both sides propose supported key exchange algorithms, ciphers, MACs, and compression methods; they agree on the strongest mutually-supported option.
3. Key exchange (KEX) — using algorithms like curve25519-sha256 (the modern default), client and server derive a shared session key without ever transmitting it directly, even over an otherwise-observed connection.
4. Server host key verification — the client checks the server’s public host key against ~/.ssh/known_hosts; a mismatch from a previously-seen host produces the well-known (and important) warning about a potential man-in-the-middle attack.
5. User authentication — publickey, password, keyboard-interactive, or GSSAPI, attempted in an order controlled by both client preference and server’s AuthenticationMethods/PubkeyAuthentication settings.
6. Channel/session establishment — once authenticated, a logical channel is opened inside the encrypted connection for the shell, command execution, port forward, or SFTP subsystem.

ssh -vvv user@remotehost 2>&1 | grep -E "kex|Server host key|Authentication"

This kind of targeted grep against verbose output is genuinely useful for isolating exactly which stage a connection is failing at, rather than scrolling through the full firehose of debug output.

Understanding known_hosts and Host Key Rotation

cat ~/.ssh/known_hosts | head -3

Each line records a hostname (or hashed hostname, if HashKnownHosts yes is set), the key type, and the base64-encoded public host key. When a server’s host key legitimately changes — a reinstall, a migration to new hardware, an intentional key rotation — every client that’s previously connected will see the “REMOTE HOST IDENTIFICATION HAS CHANGED” warning and refuse to connect until the stale entry is removed.

ssh-keygen -R oldhostname   # remove a specific stale entry safely

Modern OpenSSH also supports host key rotation notifications via the UpdateHostKeys client option, letting a server proactively inform already-connected clients of additional or replacement host keys through an authenticated in-band mechanism — a meaningful improvement over the old “figure out if the change is legitimate by out-of-band means” workflow, though it still requires the first connection to have been trustworthy.

Certificate-Based SSH Authentication (Beyond Individual Keys)

For anything beyond a handful of servers, distributing individual public keys to every host and managing revocation by hand doesn’t scale. OpenSSH supports a certificate authority model, where a central CA key signs both user keys and host keys, and individual servers/clients simply trust the CA rather than needing every individual key added manually.

Generate a CA key (done once, kept extremely secure):

ssh-keygen -t ed25519 -f ssh_ca -C "internal SSH CA"

Sign a user’s public key, producing a short-lived certificate:

ssh-keygen -s ssh_ca -I "alice-cert" -n alice -V +8h -z 1 alice_key.pub

• -V +8h — certificate expires 8 hours from signing, a genuinely powerful control that individual static keys don’t offer.
• -n alice — restricts the certificate to authenticating as the alice principal.
• -z 1 — a serial number, used for targeted revocation later.

On the server side, configure sshd_config to trust the CA:

TrustedUserCAKeys /etc/ssh/ssh_ca.pub

This is the pattern behind most modern “ephemeral SSH access” tooling in larger organizations — rather than managing hundreds of individual authorized_keys entries across a fleet, servers trust a CA, and short-lived certificates are issued per-session, often gated behind an additional approval or SSO step.

SSH Multiplexing for Faster Repeated Connections

If you connect to the same host repeatedly in a short span (common during active administration or scripting), SSH’s connection multiplexing avoids repeating the full handshake for every single connection:

Host prodweb
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 10m

mkdir -p ~/.ssh/sockets

The first connection to prodweb establishes the underlying TCP/SSH session normally; every subsequent connection within the ControlPersist window reuses that same already-authenticated connection as a transport, making each additional ssh/scp call to the same host nearly instantaneous.

Comparing SSH to Other Remote Access Approaches

Approach	Encryption	Auth model	Typical use case
SSH	Full session	Keys, passwords, certificates, MFA	General-purpose remote administration
Telnet	None	Plaintext password	Legacy device management only, avoid otherwise
RDP	Full session (modern versions)	Password, certificate, NLA	Windows GUI remote access
VNC	Varies by implementation, often weak by default	Password, sometimes weak	GUI remote access, often needs tunneling through SSH for real security
Mosh	Full session (built on SSH for initial auth)	Delegates to SSH for authentication	Unstable/high-latency connections, roaming between networks

Mosh in particular is worth knowing about if you frequently work over unreliable connections — it authenticates via a normal SSH handshake, then hands off to its own UDP-based, session-persistent protocol that survives IP changes and network interruptions far more gracefully than a raw SSH TCP session does.

Auditing Who Has SSH Access to a Server

# List all authorized keys across every user's home directory
sudo find /home /root -name "authorized_keys" -exec echo {} \; -exec cat {} \;

# Check sshd's actual effective configuration (merges config file with compiled defaults)
sudo sshd -T | grep -E "permitrootlogin|passwordauthentication|allowusers"

Running sshd -T (test/dump effective config) rather than just reading sshd_config directly is worth doing periodically — it shows exactly what the daemon is actually enforcing, including any defaults not explicitly set in the file, which can differ from what a quick read of the config alone would suggest.

Troubleshooting

Connection refused — sshd isn’t running or isn’t listening on the expected port:

sudo systemctl status sshd
sudo ss -tulnp | grep ssh

Permission denied (publickey) — verify permissions on the server side; SSH is strict about this and silently refuses keys with overly permissive file modes:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Host key verification failed — the server’s host key changed (legitimate reinstall, or a potential MITM attack). Investigate before blindly removing the old entry:

ssh-keygen -R remotehost   # only after confirming the change is legitimate

Verbose debugging for any connection issue:

ssh -vvv user@remotehost

Best Practices Summary

• Use ed25519 keys with a passphrase, protected by ssh-agent for convenience.
• Disable password authentication once keys are confirmed working.
• Disable direct root login; use sudo from a normal account instead.
• Use ~/.ssh/config and ProxyJump instead of manually chaining connections.
• Test every sshd_config change with sshd -t and from a second session before closing your current one.
• Consider MFA for anything internet-facing.

Summary

SSH is the backbone of secure remote Linux administration: encrypted transport, cryptographic host verification, and flexible authentication that scales from a single interactive login to fully unattended automation. The habits that matter most are the boring ones — key-based auth over passwords, testing config changes before you commit to them, and never closing your only working session until you’ve verified the new one works.

References

• OpenSSH Project
• man sshd_config(5)
• man ssh(1)
• Red Hat Documentation – Using secure communications between two systems with OpenSSH
• Ubuntu Server Guide – OpenSSH