How to Monitor Nginx with Prometheus and Grafana

For a long time, my approach to Nginx monitoring was reactive: something breaks, I SSH in, I grep through logs, I figure out what happened after the fact. That worked, barely, until I was managing enough servers that I couldn’t reasonably keep tabs on all of them by hand anymore. Setting up Prometheus and Grafana changed that entirely — instead of finding out about problems from angry users, I now have dashboards that show me request rates, error rates, and latency trends in real time, plus alerts that fire before things get bad. This guide walks through exactly how I set this up, from exporter installation to building a genuinely useful dashboard.

The Monitoring Stack, at a Glance

• Nginx — the web server being monitored
• nginx-prometheus-exporter (or the stub_status/vts module, depending on approach) — exposes Nginx metrics in a format Prometheus understands
• Prometheus — scrapes and stores metrics over time
• Grafana — visualizes those metrics in dashboards and handles alerting

Requirements

• A Linux server running Nginx
• Root or sudo access
• Basic familiarity with editing config files and running services
• A separate server or the same server with enough resources to also run Prometheus and Grafana (for production, I recommend running these on a dedicated monitoring host rather than the same box as your web server)

Step 1: Enable the Nginx stub_status Module

Nginx has a built-in (but very basic) status module that exposes connection counts. Most prebuilt Nginx packages already include this module — verify with:

nginx -V 2>&1 | grep -o with-http_stub_status_module

If it’s listed, you’re good. Now add a status endpoint to your Nginx config:

server {
    listen 127.0.0.1:8080;
    server_name localhost;

    location /stub_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}

I always bind this to 127.0.0.1 and explicitly deny external access — this endpoint shouldn’t be reachable from the internet, only from the exporter running locally.

Reload Nginx:

sudo nginx -t
sudo systemctl reload nginx

Test it:

curl http://127.0.0.1:8080/stub_status

You’ll see output like:

Active connections: 3
server accepts handled requests
 1023 1023 5391
Reading: 0 Writing: 1 Waiting: 2

This is useful but limited — it only exposes connection-level metrics, not per-status-code request counts, latency, or anything more granular. For that, we need the exporter and a richer log-based or module-based approach.

Step 2: Install nginx-prometheus-exporter

This is a small Go binary maintained by the Nginx/F5 team that scrapes stub_status and re-exposes it in Prometheus’s metrics format.

cd /tmp
curl -LO https://github.com/nginx/nginx-prometheus-exporter/releases/latest/download/nginx-prometheus-exporter_linux_amd64.tar.gz
tar xvzf nginx-prometheus-exporter_linux_amd64.tar.gz
sudo mv nginx-prometheus-exporter /usr/local/bin/

Create a systemd service so it runs persistently:

sudo nano /etc/systemd/system/nginx-exporter.service

[Unit]
Description=Nginx Prometheus Exporter
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/nginx-prometheus-exporter -nginx.scrape-uri=http://127.0.0.1:8080/stub_status
Restart=on-failure
User=nobody

[Install]
WantedBy=multi-user.target

Enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable nginx-exporter
sudo systemctl start nginx-exporter
sudo systemctl status nginx-exporter

By default, the exporter listens on port 9113. Confirm it’s working:

curl http://127.0.0.1:9113/metrics

You should see Prometheus-formatted output like:

nginx_connections_active 3
nginx_connections_reading 0
nginx_connections_writing 1
nginx_http_requests_total 10234

Step 3: Getting Richer Metrics with nginx-vts-module (Optional but Recommended)

stub_status only gives connection counts — no per-status-code breakdowns, no request timing, no per-upstream metrics. For serious monitoring, I install the nginx-module-vts (Virtual host Traffic Status) module instead, which provides far more detail: requests by status code, response times, bytes transferred, and per-server-block breakdowns.

On distributions with a dynamic module package available:

sudo apt install libnginx-mod-http-vhost-traffic-status -y

If not packaged for your distro, you’ll need to compile Nginx from source with the module included — check the module’s GitHub repository for current build instructions, since this changes with Nginx versions.

Configure it in nginx.conf:

http {
    vhost_traffic_status_zone;

    server {
        listen 127.0.0.1:8080;

        location /status {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
        }

        location /status/format/json {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format json;
        }
    }
}

This gives you a much richer JSON metrics endpoint at /status/format/json, which some Prometheus exporters can consume directly, or which you can scrape with a small custom exporter if needed. In practice, for most of my deployments, stub_status plus the official exporter is sufficient — I reserve the VTS module for cases where I specifically need per-status-code or per-vhost granularity.

Step 4: Install Prometheus

On your monitoring server:

cd /tmp
curl -LO https://github.com/prometheus/prometheus/releases/latest/download/prometheus-2.53.0.linux-amd64.tar.gz
tar xvzf prometheus-2.53.0.linux-amd64.tar.gz
sudo mv prometheus-2.53.0.linux-amd64 /opt/prometheus

(Check the Prometheus releases page for the current version number before downloading.)

Create a config file:

sudo nano /opt/prometheus/prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'nginx'
    static_configs:
      - targets: ['your-nginx-server-ip:9113']

Replace your-nginx-server-ip with the actual IP or hostname of the server running the exporter. If Prometheus and Nginx are on the same box, use 127.0.0.1.

Set up a systemd service:

sudo nano /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
After=network.target

[Service]
Type=simple
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data
Restart=on-failure
User=prometheus

[Install]
WantedBy=multi-user.target

sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus
sudo chown -R prometheus:prometheus /opt/prometheus
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Access the Prometheus web UI at http://your-monitoring-server:9090 and check Status > Targets — your nginx job should show as UP.

Step 5: Install Grafana

sudo apt install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt update
sudo apt install grafana -y

sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Access Grafana at http://your-monitoring-server:3000 — default login is admin / admin, and you’ll be prompted to change the password on first login.

Step 6: Connect Grafana to Prometheus

In Grafana:

1. Go to Connections > Data sources > Add data source
2. Select Prometheus
3. Set the URL to http://127.0.0.1:9090 (or your Prometheus server’s address)
4. Click Save & Test — you should see a confirmation that the connection succeeded

Step 7: Build (or Import) a Dashboard

Rather than building from scratch, I usually start from the official nginx-prometheus-exporter community dashboard, which covers the key metrics out of the box:

1. In Grafana, go to Dashboards > New > Import
2. Enter dashboard ID 12708 (a well-maintained community Nginx exporter dashboard) into the import field
3. Select your Prometheus data source
4. Click Import

This gives you panels for active connections, request rate, connection handling rates (accepted vs. handled), and reading/writing/waiting connection states out of the box.

From there, I typically add custom panels for things the default dashboard doesn’t cover, using PromQL queries like:

Request rate (requests/sec):

rate(nginx_http_requests_total[5m])

Active connections over time:

nginx_connections_active

Connection acceptance vs. handling gap (a nonzero gap here indicates dropped connections, often due to worker_connections limits being hit):

rate(nginx_connections_accepted[5m]) - rate(nginx_connections_handled[5m])

Step 8: Setting Up Alerts

Alerting is where this stack really pays off — instead of discovering problems after the fact, I get notified as they’re happening.

In Grafana, go to Alerting > Alert rules > New alert rule. A few alerts I set up on essentially every server:

High connection count (potential overload):

nginx_connections_active > 1000

Nginx exporter unreachable (Nginx or exporter down):

up{job="nginx"} == 0

Sudden drop in request rate (potential outage or upstream failure):

rate(nginx_http_requests_total[5m]) < 0.1

Configure a notification channel (Slack, email, PagerDuty, etc.) under Alerting > Contact points, and attach it to these rules so you actually get notified when they fire.

Testing the Full Pipeline

Generate some traffic to confirm metrics flow end-to-end:

for i in {1..100}; do curl -s http://your-nginx-server/ > /dev/null; done

Then check:

1. curl http://nginx-server:9113/metrics shows updated counters
2. Prometheus’s Status > Targets page shows the target as UP and recently scraped
3. A Prometheus query for nginx_http_requests_total shows the counter incrementing
4. Your Grafana dashboard reflects the traffic in near real-time

Troubleshooting Common Issues

Exporter target shows “DOWN” in Prometheus — Check network connectivity between the Prometheus server and the Nginx server on port 9113 (firewall rules are the usual culprit), and confirm the exporter service is actually running (systemctl status nginx-exporter).

stub_status returns 403 Forbidden — The allow/deny rules in your stub_status location block don’t include the IP the exporter is actually connecting from. If the exporter runs on the same host, allow 127.0.0.1; should work; if on a different host, adjust accordingly (though I’d recommend keeping stub_status local-only and running the exporter on the same host as Nginx instead).

Metrics exist but Grafana dashboard shows “No data” — Usually a data source mismatch; confirm the dashboard’s panels are pointed at the correct Prometheus data source, especially if you have multiple data sources configured.

Counters resetting unexpectedly — This is expected behavior whenever Nginx restarts (not just reloads) — Prometheus counters reset to zero, which rate() and irate() functions handle gracefully by design, so don’t be alarmed by an occasional dip in rate-based panels around deploy times.

Security Considerations

• Never expose stub_status or the exporter’s /metrics endpoint publicly. Bind both to 127.0.0.1 or restrict access via firewall rules — these endpoints reveal operational details useful to attackers.
• Put Prometheus and Grafana behind authentication and, ideally, a VPN or private network, not directly exposed to the public internet.
• Use HTTPS for Grafana in production, especially if accessed outside a trusted internal network.
• Restrict who can create/modify alert rules and dashboards using Grafana’s built-in role-based access control if multiple people have access.

Performance Tips

• Keep Prometheus’s scrape_interval reasonable (15-30s is typical) — overly frequent scraping adds unnecessary load for marginal benefit on most workloads.
• Set appropriate retention (--storage.tsdb.retention.time=30d, for example) to avoid Prometheus’s local storage growing unbounded.
• For monitoring many Nginx servers, consider Prometheus’s federation or a remote-write setup to a centralized long-term storage backend (Thanos, Cortex, Mimir) rather than a single monolithic Prometheus instance.
• Run the exporter and Prometheus on separate hardware from your production Nginx servers where possible, so monitoring overhead never competes with your actual application’s resources.

Real-World Use Cases

• Capacity planning: Watching request rate and connection trends over months has helped me make informed, data-backed scaling decisions rather than guessing.
• Incident detection: A Grafana alert firing on a sudden request-rate drop caught an upstream application crash minutes before customers started reporting it.
• Post-incident analysis: Having historical dashboards to look back on made root-causing a past outage dramatically faster than trying to reconstruct events from raw logs alone.
• SLA reporting: Aggregated uptime and error-rate metrics feed directly into monthly reports for clients who need documented service reliability.

Best Practices I Follow

1. Bind stub_status and exporter metrics endpoints to localhost only — never expose them publicly.
2. Start with the official community dashboard rather than building from scratch, then extend as needed.
3. Set up alerts for both “server is overloaded” and “server/exporter is unreachable” — the latter is easy to forget but critical.
4. Use rate() rather than raw counter values in Grafana panels, since Nginx metrics are cumulative counters.
5. Run Prometheus and Grafana on a separate host from production Nginx servers where feasible.
6. Set sane data retention policies to keep Prometheus storage manageable long-term.
7. Layer in the VTS module when you need per-status-code or per-vhost granularity beyond what stub_status provides.
8. Review and tune alert thresholds periodically as traffic patterns and capacity change.

Wrapping Up

Moving from reactive log-grepping to a proper Prometheus and Grafana monitoring setup was one of the more impactful operational changes I’ve made — not because the setup is complicated (it genuinely isn’t, once you’ve done it once), but because of what it enables: catching problems before users report them, understanding traffic trends over time, and having real data during incident response instead of guesswork. If you’re currently running Nginx without any metrics collection, I’d treat this as one of the higher-value improvements you can make to your infrastructure, and it’s a comfortable weekend project even if you’re doing it for the first time.