HOWTO: log the user's IP, not the proxy's, in Lighttpd access log

Posted by & filed under geek, howto, linux.

Lighttpd - fly lightWhen you run a webserver behind a reverse proxy or HTTP accelerator like Squid or Varnish, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP.  This not only breaks any kind of tracking or reporting you want to run against your webserver logs, but it also takes away a datapoint I’ve had use for in general server admin tasks. This server runs Varnish in front of Lighttpd, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making Lighttpd (lighty) use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple; in lighttpd.conf, enter this:

accesslog.format = "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b /
\"%{Referer}i\" \"%{User-Agent}i\""

For the definition of these variables, and plenty more, hit Lighty’s wiki. Props to the poster on the Varnish mailing list for bringing this up and reminding me to fix it! I’ve sent this link to the list so now it’s out there.

home
  • Anon

    You can also do this with mod_extforward:
    http://trac.lighttpd.net/trac/wiki/Docs/ModExtForward

    With some caveats … modifying the conf file seems more straightforward.

  • Anon

    You can also do this with mod_extforward:
    http://trac.lighttpd.net/trac/wiki/Docs/ModExtForward

    With some caveats … modifying the conf file seems more straightforward.

  • jt

    This works fine but I’m getting the port # appended to the IP address. I’m setting the X-Forward-For in Varnish like this:

    # Capture the client IP address
    remove req.http.X-Forwarded-For;
    set req.http.X-Forwarded-For = client.ip;

    And the accesslog.format in lighttpd as shown above. What I get in the lighttpd log is:

    IPAddress:Port, IPAddress like this:

    208.96.54.73:51575, 208.96.54.73 – - [01/Apr/2008:23:27:41 -0400] “GET /robots.txt HTTP/1.1″ 404 345 “-” “Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)”

    AWStats treats IP’s with the port #’s as separate hosts obviously and I want to remove the port # from the ip address. Any way to configure Varnish to just send the client IP and not the port? I’d rather not have to parse & fix the log files if possible.

  • jt

    This works fine but I’m getting the port # appended to the IP address. I’m setting the X-Forward-For in Varnish like this:

    # Capture the client IP address
    remove req.http.X-Forwarded-For;
    set req.http.X-Forwarded-For = client.ip;

    And the accesslog.format in lighttpd as shown above. What I get in the lighttpd log is:

    IPAddress:Port, IPAddress like this:

    208.96.54.73:51575, 208.96.54.73 – - [01/Apr/2008:23:27:41 -0400] “GET /robots.txt HTTP/1.1″ 404 345 “-” “Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)”

    AWStats treats IP’s with the port #’s as separate hosts obviously and I want to remove the port # from the ip address. Any way to configure Varnish to just send the client IP and not the port? I’d rather not have to parse & fix the log files if possible.

  • http://fak3r.com/ fak3r

    @jt
    I don’t know of a way to do that, currently I’m running nginx w/o varnish so I haven’t had that issue. I want to re-implement varnish in the mix soon, possibly with nginx acting as a proxy in front of it so it only sends dynamic requests back to varnish, while handling static requests by itself (nginx is said to be faster at serving static that varnish, which makes sense since varnish stores files via disk.

  • http://fak3r.com fak3r

    @jt
    I don’t know of a way to do that, currently I’m running nginx w/o varnish so I haven’t had that issue. I want to re-implement varnish in the mix soon, possibly with nginx acting as a proxy in front of it so it only sends dynamic requests back to varnish, while handling static requests by itself (nginx is said to be faster at serving static that varnish, which makes sense since varnish stores files via disk.

  • Pingback: fak3r » HOWTO: log the user’s IP, not the proxy’s, in nginx access log

  • http://bummercloud.com/ Bummer

    Thanks! This is exactly what I was looking for…

  • http://bummercloud.com/ Bummer

    Thanks! This is exactly what I was looking for…

  • http://bummercloud.com/ Bummer

    Thanks! This is exactly what I was looking for…

  • Pingback: Remote Lighttpd Logging and Cloudflare/Reverse Proxy « A No Nonsense Blog