Debugging Common Issues

The Debugging Mindset

When something breaks, resist the urge to change random settings. Follow a systematic approach:

1. Reproduce — Can you make it fail consistently?
2. Isolate — What's different between working and broken?
3. Gather data — Check logs, status codes, timing
4. Form hypothesis — What could cause these symptoms?
5. Test — Change one thing, observe result
6. Document — Record what you found for next time

502 Bad Gateway

Debugging Steps

# 1. Check if upstream is running
systemctl status your-app
ps aux | grep node
ps aux | grep php-fpm

# 2. Check if upstream is listening
ss -tlnp | grep 3000    # TCP port
ls -la /var/run/php/php-fpm.sock  # Unix socket

# 3. Test connection directly
curl -v http://127.0.0.1:3000/
curl --unix-socket /var/run/php/php-fpm.sock http://localhost/

# 4. Check Nginx error log
tail -20 /var/log/nginx/error.log
# Look for: "connect() failed" or "upstream prematurely closed"

# 5. Check if port is correct in config
grep -r "proxy_pass\|fastcgi_pass" /etc/nginx/

Error Log Examples

# Upstream not running
connect() failed (111: Connection refused) while connecting to upstream

# Wrong socket path
connect() to unix:/var/run/php/wrong.sock failed (2: No such file or directory)

# Upstream crashed mid-request
upstream prematurely closed connection while reading response header

# Permission denied on socket
connect() to unix:/var/run/php/php-fpm.sock failed (13: Permission denied)

Fixes

# Start the upstream service
systemctl start your-app
systemctl start php-fpm

# Fix socket permissions
chown www-data:www-data /var/run/php/php-fpm.sock
chmod 660 /var/run/php/php-fpm.sock

# Or configure PHP-FPM to allow nginx
# /etc/php/8.1/fpm/pool.d/www.conf
listen.owner = www-data
listen.group = www-data
listen.mode = 0660

504 Gateway Timeout

Debugging Steps

# 1. Check how long requests take
curl -w "Time: %{time_total}s\n" -o /dev/null -s http://localhost/slow-endpoint

# 2. Check current timeout settings
grep -r "timeout" /etc/nginx/nginx.conf /etc/nginx/sites-enabled/

# 3. Monitor application during request
top -p $(pgrep -d',' node)  # Watch CPU/memory
tail -f /var/log/your-app/app.log  # Watch application logs

# 4. Check for slow queries
# Enable slow query log in MySQL/PostgreSQL
tail -f /var/log/mysql/slow-query.log

# 5. Profile the application
# Add timing logs, use APM tools (New Relic, Datadog)

Increasing Timeouts

# Nginx - increase proxy timeouts
location /api/export {
    proxy_pass http://backend;

    # Increase timeouts for slow endpoint
    proxy_connect_timeout 60s;
    proxy_send_timeout 300s;
    proxy_read_timeout 300s;
}

# FastCGI timeout
location ~ \.php$ {
    fastcgi_pass unix:/var/run/php/php-fpm.sock;
    fastcgi_read_timeout 300s;
}

403 Forbidden

Debugging Steps

# 1. Check error log for specific reason
tail -20 /var/log/nginx/error.log
# Look for: "directory index of" or "forbidden" or "Permission denied"

# 2. Check file permissions
ls -la /var/www/html/
ls -la /var/www/html/index.html
namei -l /var/www/html/problematic-file

# 3. Check ownership
stat /var/www/html/index.html
# Owner should be readable by nginx user (www-data/nginx)

# 4. Check Nginx configuration for deny rules
grep -r "deny\|allow" /etc/nginx/

# 5. Check SELinux (if enabled)
getenforce
ls -Z /var/www/html/
ausearch -m avc -ts recent

# 6. Check if index file exists
ls /var/www/html/index.*

Error Log Examples

# Permission denied
open() "/var/www/html/file.html" failed (13: Permission denied)

# No index file, listing denied
directory index of "/var/www/html/" is forbidden

# Access rule denied
access forbidden by rule, client: 192.168.1.100

Fixes

# Fix file permissions
chmod 644 /var/www/html/index.html
chmod 755 /var/www/html/

# Fix ownership
chown -R www-data:www-data /var/www/html/

# Fix parent directory permissions (execute needed to traverse)
chmod +x /var /var/www /var/www/html

# Fix SELinux context
restorecon -Rv /var/www/html/
# Or allow httpd to read user content
setsebool -P httpd_read_user_content 1

# Add index file or enable directory listing
location / {
    index index.html index.htm;
    # Or enable listing:
    # autoindex on;
}

404 Not Found

Debugging Steps

# 1. Check what path Nginx is looking for
tail -20 /var/log/nginx/error.log
# Shows: open() "/actual/path/file" failed

# 2. Verify the file exists
ls -la /var/www/html/expected/path/

# 3. Check document root in config
grep -r "root\|alias" /etc/nginx/sites-enabled/

# 4. Check if location block matches
nginx -T | grep -A5 "location"

# 5. Test URL rewriting
curl -v http://localhost/test-path
# Compare requested path to what error log shows

# 6. Check symlinks
ls -la /var/www/html/
# If symlinks exist, check if they're valid and nginx follows them

Common Misconfigurations

# Wrong: Missing trailing slash causes path issues
location /app {
    alias /var/www/app;  # Request for /app/style.css → /var/www/appstyle.css
}

# Correct: Trailing slashes match
location /app/ {
    alias /var/www/app/;  # Request for /app/style.css → /var/www/app/style.css
}

# Alternative: Use root instead of alias
location /app {
    root /var/www;  # Request for /app/style.css → /var/www/app/style.css
}

# Enable symlink following
location / {
    disable_symlinks off;  # Default, but check if changed
}

# SPA routing - send all paths to index.html
location / {
    try_files $uri $uri/ /index.html;
}

Using curl for Debugging

curl is your best friend for debugging HTTP issues:

# Basic request with response code
curl -I http://localhost/

# Verbose output (shows headers, connection info)
curl -v http://localhost/

# Follow redirects and show each step
curl -vL http://localhost/redirect

# Test specific Host header (virtual host)
curl -H "Host: example.com" http://localhost/

# Test POST request
curl -X POST -d '{"key":"value"}' -H "Content-Type: application/json" http://localhost/api

# Show timing breakdown
curl -w "@curl-format.txt" -o /dev/null -s http://localhost/

Create curl-format.txt for timing analysis:

# curl-format.txt
     time_namelookup:  %{time_namelookup}s\n
        time_connect:  %{time_connect}s\n
     time_appconnect:  %{time_appconnect}s\n
    time_pretransfer:  %{time_pretransfer}s\n
       time_redirect:  %{time_redirect}s\n
  time_starttransfer:  %{time_starttransfer}s\n
                      ----------\n
          time_total:  %{time_total}s\n

Testing Configuration Changes

Always validate configuration before applying:

# Test Nginx config syntax
nginx -t

# Test and show parsed config
nginx -T

# Test specific config file
nginx -t -c /etc/nginx/nginx.conf

# Reload (graceful - doesn't drop connections)
nginx -s reload

# Or using systemctl
systemctl reload nginx

# If reload fails, check what's wrong
journalctl -u nginx -n 50

Network-Level Debugging

When HTTP-level debugging isn't enough:

Check What's Listening

# Show all listening ports
ss -tlnp

# Show only nginx ports
ss -tlnp | grep nginx

# Check specific port
ss -tlnp | grep :80

# Show established connections
ss -tn state established

DNS Issues

# Check DNS resolution
dig example.com
nslookup example.com

# Check what IP nginx resolves
# (Add resolver directive to nginx config)
resolver 8.8.8.8;

# Test from server
curl -v http://upstream-hostname/

tcpdump for Deep Debugging

# Capture HTTP traffic on port 80
tcpdump -i any -A port 80

# Capture and save to file
tcpdump -i any -w capture.pcap port 80

# Filter by host
tcpdump -i any host upstream.example.com

# Show only packet headers
tcpdump -i any -q port 80

Common Error Patterns

Debug Logging

Enable verbose logging temporarily to diagnose issues:

# Increase error log verbosity
error_log /var/log/nginx/error.log debug;

# Debug specific connection (by IP)
events {
    debug_connection 192.168.1.100;
    debug_connection 192.168.1.0/24;
}

# Debug specific location
location /problem {
    error_log /var/log/nginx/problem-debug.log debug;
}

Debugging Checklist

When things break, work through this systematically:

• Check error log: tail -f /var/log/nginx/error.log
• Check access log for the failed request
• Test with curl directly to the server
• Verify upstream is running: systemctl status app
• Test upstream directly: curl http://127.0.0.1:3000/
• Check file permissions: ls -la /var/www/
• Verify configuration syntax: nginx -t
• Check what's listening: ss -tlnp
• Check disk space: df -h
• Check memory: free -m
• Check for recent config changes: git log -5 /etc/nginx/

Emergency Procedures

When production is down:

Quick Recovery

# If config is broken, revert to last known good
cp /etc/nginx/nginx.conf.backup /etc/nginx/nginx.conf
nginx -t && nginx -s reload

# If nginx won't start, check what's wrong
nginx -t 2>&1
journalctl -u nginx --no-pager -n 100

# Check if something else is using port 80
ss -tlnp | grep :80

# Kill stuck nginx processes (last resort)
pkill -9 nginx
nginx

Traffic Management

# Temporarily return maintenance page
server {
    listen 80 default_server;
    return 503;
    error_page 503 /maintenance.html;
    location = /maintenance.html {
        root /var/www/maintenance;
        internal;
    }
}

# Drain connections before shutdown
nginx -s quit  # Graceful shutdown, waits for connections

Summary

• Always check error logs first — They usually tell you exactly what's wrong
• 502 = upstream connection problem (not running, wrong address)
• 504 = upstream too slow (optimize app, increase timeout)
• 403 = permission denied (file perms, SELinux, access rules)
• 404 = wrong path (document root, alias config, rewrites)
• Use curl -v for HTTP-level debugging
• Always nginx -t before reload
• Debug logging temporarily for deep issues
• Systematic approach — reproduce, isolate, gather data, hypothesize, test