Enriching Server Logs

Custom log formats, Client Hints, and script-to-header techniques for richer server-side analytics

From Basic to Rich Logs

Server access logs are the oldest and most reliable analytics data source — every request is recorded without any client-side code. But the default Common Log Format captures only 7 fields. With configuration changes and a few techniques, you can transform logs into a rich analytics dataset that rivals client-side collection for many use cases.

Log Formats — From Common to Custom

The Common Log Format (CLF) gives you IP, identity, user, timestamp, request line, status, and size — only 7 fields, with no browser info, no referrer, and no timing. The Combined format adds Referer and User-Agent, making it the de facto standard for analytics. But custom formats can capture any HTTP header, server variable, cookie, or timing metric.

Log Format Fields Analytics Value
Common (CLF) IP, identity, user, timestamp, request line, status, size Basic hit counting only — no browser or referrer data
Combined CLF + Referer + User-Agent Traffic sources, browser/OS breakdown — the analytics minimum
Custom / Extended Any header, cookie, variable, or timing metric Rich analytics: response time, language, viewport, device hints

Both Apache and Nginx support custom log formats that can capture any request header:

# Apache — LogFormat + CustomLog LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D \"%{Accept-Language}i\" \"%{X-Viewport}i\"" enriched CustomLog /var/log/apache2/access.log enriched # %D = response time in microseconds # %{HeaderName}i = any request header # Nginx — log_format + access_log log_format enriched '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time "$http_accept_language" "$http_x_viewport"'; access_log /var/log/nginx/access.log enriched; # $request_time = response time in seconds # $http_headername = any request header (lowercase, dashes become underscores)

The key insight: %{HeaderName}i (Apache) and $http_headername (Nginx) let you log any request header. This is what makes the techniques below possible.

Client Hints — Structured Device Information

User-Agent strings are chaotic, bloated, and increasingly frozen by browsers. Client Hints are the modern replacement: structured, opt-in headers the server requests. The server sends an Accept-CH header (or <meta http-equiv="Accept-CH">) listing desired hints — the browser then includes them on subsequent requests, and the server logs them automatically.

Client Hint What It Provides Example Value
Sec-CH-UABrowser brand and version"Chromium";v="124", "Chrome";v="124"
Sec-CH-UA-MobileMobile device??0 (no) or ?1 (yes)
Sec-CH-UA-PlatformOperating system"macOS"
Sec-CH-UA-Platform-VersionOS version"14.5"
Sec-CH-UA-Full-Version-ListDetailed browser versionsFull version strings
Sec-CH-UA-ModelDevice model"Pixel 8"
Sec-CH-UA-ArchCPU architecture"arm"
Sec-CH-Viewport-WidthViewport width in CSS pixels1440
Sec-CH-DPRDevice pixel ratio2.0
ECTEffective connection type4g, 3g, 2g, slow-2g
DownlinkEstimated bandwidth (Mbps)10.0
RTTEstimated round-trip time (ms)50

Client Hints "upscale" your logs: structured device info, network quality, and viewport data — all from headers, logged automatically. However, they are Chromium-only (Chrome, Edge, Opera). Firefox and Safari don't support them, so User-Agent remains necessary as a fallback.

Script-to-Header — Bridging Client and Server Logs

JavaScript knows viewport, scroll depth, errors, and color scheme. Server logs know timing, status codes, and upstream latency. Merging both is ideal. The technique: JavaScript sets a cookie with client-side data — the browser sends it on subsequent requests — the server log format captures it.

// Set client-side data as a cookie for server log capture document.cookie = '_viewport=' + window.innerWidth + 'x' + window.innerHeight + ';path=/;SameSite=Lax;max-age=1800'; // Or set a custom header on fetch requests fetch('/api/data', { headers: { 'X-Viewport': window.innerWidth + 'x' + window.innerHeight } });

The server captures the cookie via %{_viewport}C (Apache) or $cookie__viewport (Nginx). The advantage: client data appears in server logs without a separate beacon endpoint. The limitation: cookie-based data is one request behind (the cookie is set on one request and sent on the next), and cookies add overhead to every request. This technique works best for data that changes infrequently: viewport, timezone, color scheme, device pixel ratio.

Log Forwarding — Third-Party Analysis of First-Party Data

A hybrid model: collect logs first-party, then forward them to a third party for analysis. This gives you the privacy benefits of first-party collection with the analysis power of third-party tools.

  Web Server ──writes──▶ access.log ──▶ Log Shipper ──ships over HTTPS──▶ Analysis Service
                                       (Filebeat,                        (ELK, Splunk,
                                        Fluentd,                          Datadog, or
                                        Fluent Bit)                       custom)

Common log shippers include rsyslog/syslog-ng (traditional), Filebeat/Fluentd/Fluent Bit (modern), cloud agents, or even a simple pipe-to-script. The privacy advantage is significant: the browser never talks to the third party. You decide what fields to forward and what to redact.

This approach blurs the first-party / third-party line — collection is first-party, analysis may be third-party.