How to Secure Nginx with ModSecurity

Rate limiting and HTTP auth solve certain classes of problems, but neither of them does anything against a SQL injection attempt, a cross-site scripting payload, or a malicious file upload disguised as an image. For that level of protection, I install ModSecurity — an open-source web application firewall (WAF) that inspects request and response traffic against a rule set and blocks anything that looks malicious before it ever reaches your application. I started using it after a client’s WordPress site got compromised through a plugin vulnerability that a WAF rule would have caught outright. This guide covers everything involved in getting ModSecurity running with Nginx, from compiling the module to tuning rules so it doesn’t block legitimate traffic.

What ModSecurity Actually Does

ModSecurity is a rule-based engine that inspects HTTP requests (and optionally responses) and takes action — usually blocking or logging — based on pattern matches. Paired with the OWASP Core Rule Set (CRS), it provides out-of-the-box protection against:

• SQL injection
• Cross-site scripting (XSS)
• Remote and local file inclusion
• Command injection
• Protocol violations and malformed requests
• Common scanner/bot fingerprints
• Session fixation and various OWASP Top 10 categories

Unlike Nginx’s built-in rate limiting or access control, ModSecurity actually inspects the content of requests — headers, query strings, POST bodies — against a large rule database, which is fundamentally more powerful but also more resource-intensive and requires tuning.

Requirements

• A Linux server (Ubuntu/Debian commands shown; adjust package manager for RHEL/CentOS)
• Nginx — note that ModSecurity for Nginx is delivered as a dynamic module, which usually means compiling it yourself unless your distro provides a prebuilt package
• Root or sudo access
• Development tools for compilation (build-essential, various libraries)
• Patience — the CRS requires tuning to avoid false positives, and this is genuinely the most time-consuming part of the whole process

Step 1: Install Build Dependencies

sudo apt update
sudo apt install -y git build-essential libpcre3 libpcre3-dev libssl-dev \
  libtool automake autoconf libxml2-dev libcurl4-openssl-dev \
  libyajl-dev pkg-config zlib1g-dev

Step 2: Build the ModSecurity Library

cd /usr/local/src
sudo git clone --depth 1 -b v3/master https://github.com/owasp-modsecurity/ModSecurity
cd ModSecurity
sudo git submodule init
sudo git submodule update
sudo ./build.sh
sudo ./configure
sudo make -j$(nproc)
sudo make install

This step takes a while — ModSecurity v3 has a fair number of dependencies to compile. Grab a coffee.

Step 3: Build the Nginx Connector Module

ModSecurity v3 doesn’t integrate with Nginx directly — it needs the separate ModSecurity-nginx connector, compiled as a dynamic Nginx module:

cd /usr/local/src
sudo git clone --depth 1 https://github.com/owasp-modsecurity/ModSecurity-nginx.git

Now you need to compile this against your exact installed Nginx version, using the same configure arguments Nginx itself was built with:

nginx -V

Copy the configure arguments line from the output, then:

cd /usr/local/src
sudo wget http://nginx.org/download/nginx-1.26.0.tar.gz  # match your installed version
sudo tar xzf nginx-1.26.0.tar.gz
cd nginx-1.26.0

sudo ./configure --with-compat [paste your original configure arguments here] \
  --add-dynamic-module=../ModSecurity-nginx

sudo make modules
sudo cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/

Load the module in nginx.conf, near the top:

load_module modules/ngx_http_modsecurity_module.so;

Verify Nginx still starts cleanly:

sudo nginx -t

Step 4: Configure ModSecurity’s Base Configuration

Copy the recommended base config:

sudo mkdir -p /etc/nginx/modsec
sudo cp /usr/local/src/ModSecurity/modsecurity.conf-recommended /etc/nginx/modsec/modsecurity.conf
sudo cp /usr/local/src/ModSecurity/unicode.mapping /etc/nginx/modsec/

Edit /etc/nginx/modsec/modsecurity.conf and change the critical line:

SecRuleEngine DetectionOnly

to:

SecRuleEngine On

I always start with DetectionOnly in a fresh deployment (logs matches without blocking), get a feel for what’s triggering on real traffic, then switch to On once I’ve tuned out false positives. Jumping straight to blocking mode on a production site without this step is how you accidentally block legitimate customers.

Step 5: Install the OWASP Core Rule Set (CRS)

The base ModSecurity install has no rules by default — you need the CRS separately:

cd /etc/nginx/modsec
sudo git clone https://github.com/coreruleset/coreruleset.git
cd coreruleset
sudo cp crs-setup.conf.example crs-setup.conf

Now create a main include file that pulls everything together:

sudo nano /etc/nginx/modsec/main.conf

Include /etc/nginx/modsec/modsecurity.conf
Include /etc/nginx/modsec/coreruleset/crs-setup.conf
Include /etc/nginx/modsec/coreruleset/rules/*.conf

Step 6: Enable ModSecurity in Your Nginx Server Block

server {
    listen 80;
    server_name example.com;

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Test and reload:

sudo nginx -t
sudo systemctl reload nginx

Step 7: Testing ModSecurity Is Actually Working

With SecRuleEngine DetectionOnly set, trigger an obviously malicious-looking request and check the logs rather than expecting a block:

curl "http://example.com/?id=1' OR '1'='1"

Check the ModSecurity audit log:

sudo tail -f /var/log/modsec_audit.log

You should see a detailed entry showing the matched rule ID, the rule message, and the offending request data.

Once you switch to SecRuleEngine On, the same request should be blocked outright, typically returning a 403 Forbidden:

curl -I "http://example.com/?id=1' OR '1'='1"

Step 8: Tuning to Reduce False Positives

This is the part that actually takes real time. The CRS is aggressive by design, and legitimate traffic — especially from CMS platforms, rich text editors, or API clients sending complex JSON — will trigger false positives regularly at first.

Set the anomaly scoring threshold in crs-setup.conf — CRS uses an anomaly scoring model rather than a strict “one rule match = block” approach:

SecAction \
  "id:900110,\
  phase:1,\
  pass,\
  t:none,\
  nolog,\
  setvar:tx.inbound_anomaly_score_threshold=5,\
  setvar:tx.outbound_anomaly_score_threshold=4"

Lower thresholds (more sensitive) catch more attacks but risk more false positives; higher thresholds are more permissive.

Exclude specific rules for specific paths when you identify a genuine false positive. Say your CMS’s rich text editor legitimately submits content that trips the XSS detection rule on POST /admin/content:

location /admin/content {
    modsecurity on;
    modsecurity_rules 'SecRuleRemoveById 941100';
    proxy_pass http://127.0.0.1:3000;
}

I always look up the specific rule ID from the audit log before excluding it, and I scope the exclusion as narrowly as possible (specific location, not globally) rather than disabling a whole rule category site-wide.

Use the paranoia level setting for a broader tuning lever — CRS defines paranoia levels 1 through 4, where level 1 (the default) catches the most obvious attacks with the fewest false positives, and higher levels add progressively more (and more sensitive) rules:

SecAction "id:900000,phase:1,pass,nolog,setvar:tx.paranoia_level=1"

I keep most production sites at paranoia level 1 — going higher generates significantly more false positives that require more tuning time than most projects can justify.

Troubleshooting Common Issues

Nginx fails to start after enabling the module — Almost always a version mismatch between the Nginx binary and the compiled connector module. The ModSecurity-nginx module must be compiled against the exact same Nginx version and configure arguments as your running Nginx binary.

Everything gets blocked, including normal traffic — You likely jumped straight to SecRuleEngine On without a DetectionOnly tuning period first, or your paranoia level is set too high for your application’s traffic patterns.

High latency after enabling ModSecurity — Rule processing does add CPU overhead, especially at higher paranoia levels or with request body inspection enabled on large payloads. Check SecRequestBodyLimit and SecResponseBodyLimit settings — very large limits mean ModSecurity inspects (and buffers) huge amounts of data per request.

Audit log growing very large very fast — Adjust SecAuditLogParts to log only what you need, and make sure this log is included in your log rotation configuration (see the companion guide on rotating Nginx log files).

Can’t find which rule blocked a request — Check the audit log entry for the request; it will include the specific rule ID(s) that matched, which you can then look up in the CRS rule files for context on what it was detecting.

Security Considerations

• Don’t rely on ModSecurity alone. It’s a strong additional layer, not a replacement for secure application code, input validation, and proper authentication/authorization at the app level.
• Keep the CRS updated. New attack patterns emerge constantly; an outdated rule set misses newer techniques. I check for CRS updates quarterly at minimum.
• Protect the audit log itself — it can contain sensitive request data (including attempted credential values from blocked requests), so restrict file permissions appropriately.
• Don’t disable rules broadly to “fix” false positives — always scope exclusions as narrowly as possible (specific rule ID, specific path) rather than turning off whole rule categories, which reopens the exact attack surface those rules exist to close.
• Monitor for ModSecurity being silently bypassed — confirm the module loads and rules apply after every Nginx config change or version upgrade, since a broken connector module can fail in ways that don’t necessarily stop Nginx from starting.

Performance Tips

• Limit request body inspection size to avoid excessive memory/CPU use on large uploads:

SecRequestBodyLimit 13107200
SecRequestBodyNoFilesLimit 131072

• Disable response body inspection (SecResponseBodyAccess Off) unless you specifically need it — inspecting outbound responses roughly doubles the inspection overhead and is less commonly needed than request inspection.
• Use paranoia level 1 unless you have a specific compliance or threat-model reason to go higher — each level up meaningfully increases rule evaluation overhead.
• Exclude static asset paths from ModSecurity entirely — there’s little value in WAF-inspecting requests for .css, .js, or image files:

location ~* \.(css|js|jpg|jpeg|png|gif|ico|svg)$ {
    modsecurity off;
    # normal static file serving config
}

Real-World Use Cases

• Protecting CMS platforms (WordPress, Drupal, Joomla) from the constant stream of automated exploit attempts targeting known plugin/theme vulnerabilities — this is genuinely one of the highest-value applications of ModSecurity I’ve deployed.
• Shielding legacy applications that can’t easily be patched or refactored, buying time against known vulnerability classes while a proper fix is developed.
• Compliance requirements — some standards (PCI-DSS, for instance) explicitly call for a WAF in front of applications handling payment data.
• API protection — blocking injection attempts and malformed payloads before they reach backend services, especially useful for APIs aggregating multiple backend microservices with varying levels of input validation maturity.

Best Practices I Follow

1. Always start in DetectionOnly mode and review audit logs before switching to blocking mode.
2. Keep paranoia level at 1 unless you have a specific, well-understood reason to increase it.
3. Scope rule exclusions as narrowly as possible — specific rule ID and specific location, never a blanket category disable.
4. Exclude static asset paths from inspection to reduce unnecessary overhead.
5. Keep the OWASP Core Rule Set updated on a regular schedule.
6. Set sane request body size limits to control memory and CPU overhead.
7. Treat the audit log as sensitive data and include it in your log rotation and access control policies.
8. Never treat ModSecurity as a substitute for secure application code — layer it as defense in depth, not the only defense.

Wrapping Up

Setting up ModSecurity with Nginx is genuinely more involved than most of the other configuration changes I make to a server — compiling a connector module against your exact Nginx build, tuning the Core Rule Set, and working through false positives all take real time. But for anything handling sensitive data, running third-party CMS software, or exposed to the general internet at meaningful scale, it closes off an entire category of attacks that no amount of rate limiting or access control alone will catch. I’d recommend setting this up on a staging environment first, running it in detection mode against realistic traffic for at least a week, and only then moving to blocking mode in production — the tuning period is not optional if you want a WAF that protects you without breaking your own application.