Server Hardening Checklist

This checklist distills the security guides into an actionable sequence of steps. It is divided into three phases: Day-1 Hardening (performed when the server is first provisioned), Ongoing Maintenance (repeated on a regular schedule), and Monitoring and Incident Response (continuous). Use it as a baseline and adapt it to your environment and compliance requirements. Not every item applies to every server, but every item should be consciously evaluated.

Hub: Linux Security Hardening | See also: SSH Hardening, Firewall Best Practices

Phase 1: Day-1 Hardening

Perform these steps immediately after provisioning and before exposing the server to production traffic. The order matters: update first, then lock down access, then enable monitoring.

1.1 Update All Packages

# Debian/Ubuntu
sudo apt update && sudo apt upgrade -y

# Fedora/RHEL
sudo dnf upgrade -y

# Enable automatic security updates (Debian/Ubuntu)
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

# Enable automatic security updates (Fedora/RHEL)
sudo dnf install dnf-automatic
sudo systemctl enable --now dnf-automatic-install.timer

# Verify automatic updates are configured
cat /etc/apt/apt.conf.d/20auto-upgrades    # Debian/Ubuntu
systemctl status dnf-automatic-install.timer   # Fedora/RHEL

Starting from a fully patched base eliminates all known vulnerabilities from day one. Many breaches exploit CVEs for which patches have been available for months.

1.2 Configure the Firewall

Set a default deny policy and allow only the services you need:

# UFW (Debian/Ubuntu)
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 2222/tcp comment 'SSH'
sudo ufw allow 443/tcp comment 'HTTPS'
sudo ufw enable
sudo ufw status verbose

# firewalld (RHEL/Fedora)
sudo firewall-cmd --set-default-zone=drop
sudo firewall-cmd --zone=public --add-port=2222/tcp --permanent
sudo firewall-cmd --zone=public --add-service=https --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --zone=public --list-all

See Firewall Best Practices for detailed configuration including rate limiting, conntrack, logging, and IPv6.

1.3 Harden SSH

# Generate a strong host key (if not already present)
sudo ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ""

# Apply hardening directives via a drop-in configuration file
sudo tee /etc/ssh/sshd_config.d/hardening.conf <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowUsers deploy admin
Port 2222
MaxAuthTries 3
X11Forwarding no
AllowTcpForwarding no
AllowAgentForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2
EOF

# Validate the configuration and restart
sudo sshd -t && sudo systemctl restart sshd

See SSH Hardening for ssh-agent, ProxyJump, fail2ban, port knocking, and algorithm hardening.

1.4 Disable Root Login and Create a Deploy User

# Create a non-root user for administration
sudo adduser deploy
sudo usermod -aG sudo deploy    # Debian/Ubuntu
sudo usermod -aG wheel deploy   # RHEL/Fedora

# Copy your SSH public key to the new user
sudo mkdir -p /home/deploy/.ssh
sudo cp ~/.ssh/authorized_keys /home/deploy/.ssh/
sudo chown -R deploy:deploy /home/deploy/.ssh
sudo chmod 700 /home/deploy/.ssh
sudo chmod 600 /home/deploy/.ssh/authorized_keys

# Lock the root password (SSH root login is already disabled above)
sudo passwd -l root

# Verify sudo works for the new user before logging out as root
su - deploy -c "sudo whoami"

1.5 Remove Unnecessary Services

# List all enabled services
systemctl list-unit-files --state=enabled

# Disable services you do not need (examples)
sudo systemctl disable --now cups
sudo systemctl disable --now avahi-daemon
sudo systemctl disable --now rpcbind
sudo systemctl disable --now bluetooth

# Uninstall packages that are not needed
sudo apt purge telnetd ftp vsftpd rsh-client    # Debian/Ubuntu
sudo dnf remove telnet-server vsftpd             # RHEL/Fedora

# Verify which services are listening
sudo ss -tulnp

Every running service is a potential attack vector. Minimise the installed package set. If you do not need it, remove it.

1.6 Set Up Basic Auditing

# Install and enable auditd
sudo apt install auditd audispd-plugins   # Debian/Ubuntu
sudo dnf install audit                     # RHEL/Fedora
sudo systemctl enable --now auditd

# Add baseline audit rules
sudo tee /etc/audit/rules.d/baseline.rules <<'EOF'
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/ssh/sshd_config -p wa -k sshd
-w /usr/bin/sudo -p x -k priv_esc
-w /usr/bin/su -p x -k priv_esc
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
EOF

sudo augenrules --load
sudo auditctl -l

See Audit & Intrusion Detection for AIDE filesystem integrity, fail2ban jails, and log analysis.

1.7 Enable MAC

# RHEL/Fedora: verify SELinux is enforcing
getenforce   # Should show "Enforcing"

# Ubuntu/Debian: verify AppArmor is loaded
sudo aa-status

See SELinux & AppArmor for policy management, troubleshooting, and profile creation.

Phase 2: Ongoing Maintenance

2.1 Patch Management

# Check for available updates (at least weekly)
sudo apt update && apt list --upgradable    # Debian/Ubuntu
sudo dnf check-update                       # RHEL/Fedora

# Apply all updates
sudo apt upgrade -y
sudo dnf upgrade -y

# Check if a reboot is required after kernel updates
[ -f /var/run/reboot-required ] && echo "Reboot needed"  # Debian/Ubuntu
needs-restarting -r                                       # RHEL/Fedora

Schedule a maintenance window at least monthly. For critical CVEs (CVSS 9.0 or higher), apply patches within 24-48 hours. Subscribe to your distribution's security mailing list and monitor CVE databases.

2.2 Log Review

# Review authentication failures over the past week
journalctl -u sshd --since "7 days ago" | grep "Failed"

# Run logwatch for a weekly summary
sudo logwatch --detail high --range "between -7 days and today"

# Check fail2ban activity
sudo fail2ban-client status sshd

# Review auditd reports
sudo aureport --auth --start this-week
sudo aureport --failed --start this-week

# Check AIDE for unexpected filesystem changes
sudo aide --check

Automate log delivery to a central log server or SIEM for correlation and long-term retention. Logs on the server itself can be tampered with by an attacker who gains root.

2.3 Backup Verification

# Verify backup integrity (example with restic)
restic -r /mnt/backup check
restic -r /mnt/backup snapshots

# Test a restore to a temporary location
restic -r /mnt/backup restore latest --target /tmp/restore-test
ls -la /tmp/restore-test/

# Ensure LUKS header backups are current
sudo cryptsetup luksHeaderBackup /dev/sda2 \
  --header-backup-file /root/sda2-header-$(date +%F).bak

Backups that have never been restored are not backups -- they are hopes. Test restores quarterly at minimum. Verify that you can rebuild the server from backups alone.

2.4 Credential Rotation

Rotate SSH keys, API tokens, database passwords, and TLS certificates on a regular schedule. Automate where possible:

# Renew Let's Encrypt certificates (usually handled by the certbot timer)
sudo certbot renew --dry-run

# Rotate an SSH key
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_new
ssh-copy-id -i ~/.ssh/id_ed25519_new.pub user@server
# Test the new key, then remove the old key from authorized_keys

# Rotate a LUKS passphrase
sudo cryptsetup luksAddKey /dev/sda2       # add new passphrase
sudo cryptsetup luksRemoveKey /dev/sda2    # remove old passphrase

Phase 3: Monitoring and Incident Response

3.1 Monitoring with Prometheus and node_exporter

Prometheus with node_exporter provides real-time visibility into system health metrics:

# Install node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/\
  node_exporter-1.7.0.linux-amd64.tar.gz
tar xzf node_exporter-*.tar.gz
sudo cp node_exporter-*/node_exporter /usr/local/bin/

# Create a systemd service
sudo tee /etc/systemd/system/node_exporter.service <<'EOF'
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --collector.processes
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo useradd -rs /bin/false node_exporter
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

# Verify it is running
curl -s http://localhost:9100/metrics | head

Configure Prometheus to scrape node_exporter on port 9100 and set up Alertmanager rules for disk space, CPU, memory, failed SSH logins, and service availability.

3.2 Alerting Rules

Example Prometheus alert rules:

# prometheus/rules/security.yml
groups:
  - name: security
    rules:
      - alert: Fail2banDown
        expr: node_systemd_unit_state{name="fail2ban.service",state="active"} != 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "fail2ban is not running on {{ $labels.instance }}"

      - alert: DiskSpaceLow
        expr: >
          node_filesystem_avail_bytes{mountpoint="/"} /
          node_filesystem_size_bytes{mountpoint="/"} < 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Root filesystem below 10% on {{ $labels.instance }}"

      - alert: HighLoadAverage
        expr: node_load15 > 4
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "15-minute load average above 4 on {{ $labels.instance }}"

3.3 Incident Response Plan

Prepare a documented incident response plan before you need it:

  1. Detect -- monitoring alerts, log anomalies, AIDE reports, user reports.
  2. Contain -- isolate the affected system by tightening firewall rules or moving it to a quarantine VLAN. Do not shut down the system unless necessary (volatile memory contains forensic evidence).
  3. Investigate -- review audit logs (ausearch), file integrity (aide --check), network captures, and process listings.
  4. Eradicate -- remove the attacker's access, patch the vulnerability, rotate all compromised credentials.
  5. Recover -- restore from verified backups, re-harden, test, and return to service.
  6. Review -- conduct a blameless post-incident review. Document what happened, what worked, what failed, and what changes will prevent recurrence.

3.4 CIS Benchmarks

The Center for Internet Security (CIS) publishes detailed benchmarks for Linux distributions with hundreds of specific configuration recommendations, each scored as Level 1 (essential, minimal performance impact) or Level 2 (defense in depth, may affect functionality). Download the benchmark for your distribution from cisecurity.org and use it as a comprehensive audit checklist.

Automated tools can scan your system against the benchmark:

# OpenSCAP (open source)
sudo apt install libopenscap8 ssg-debian    # Debian/Ubuntu
sudo oscap xccdf eval \
  --profile xccdf_org.ssgproject.content_profile_cis \
  --results /tmp/cis-results.xml \
  --report /tmp/cis-report.html \
  /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml

# Open the HTML report
xdg-open /tmp/cis-report.html

Security is a journey, not a destination. This checklist is a starting point. Revisit it with every new deployment, every security incident, and at least quarterly. The goal is continuous improvement: each pass through the checklist should leave your systems measurably more resilient than before.