HOWTO: log the user’s IP, not the proxy’s, in nginx access log
nginx
So back in January I had a post about HOWTO: log the user’s IP, not the proxy’s, in Lighttpd access log, but today I switched that system to run nginx (actually nginx has been running since early this year, I just got lazy on running Varnish) fronted again by Varnish. I had the same issue, but not much trouble solving it. Since I often refer to my own notes on fak3r, I’m recording it here for myself, and anyone streaming in from Google. So, as I talked about before, when you run a webserver behind Varnish doing http acceleration, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP. This not only breaks any kind of tracking or reporting you want to run against your webserver logs. Since this server runs Varnish in front of nginx, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making nginx use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple. Edit your nginx.conf file:
vi /etc/nginx/nginx.conf
Once in the file, find the block about logging, and add the following to it:
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"' ;
access_log /var/log/nginx/access.log main;and finally, restart nginx
/etc/init.d/nginx restart
If you look at the logfiles now you’ll see the IP of the original requester!
HOWTO: Configure nginx for Debian / Ubuntu
UPDATE: I’m reworking my config blending in the security ideas found on camomel.org they’re really thought things through on this, this should make for a very secure environment.
I’m always trying new software, and with the webserver I’ve moved from Apache 1.3 to 2.0 to 2.2, and then later I moved everything over to Lighttpd, which I’ve liked, save for some memory issues that popped up. Now, enter a web server named nginx (engine x), written by a Russian hacker. It’s already proved it’s meddle by running some of the largest Russian sites for years now. It has the speed of Lighttpd, but with none of that memory weirdness, plus it uses a fraction of the CPU, so scaling should be smooth for highly visited sites. It also does cool things like load balancing, reverse proxy, IMAP and POP proxy, etc, so I can see it being used in a variety of ways on a network. It took me some time to understand how to configure it, which was a case of me just making it harder than it really is, so I wanted to post it here. Look for updates as we go along, but this is currently backing a Production site I manage.
user www-data www-data;
worker_processes 5;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr $host $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" '
'"$request_time" "$gzip_ratio"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
sendfile on;
tcp_nopush on;
tcp_nodelay off;
keepalive_timeout 65;
gzip on;
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_buffers 16 8k;
#gzip_proxied expired no-cache no-store private auth;
gzip_proxied any;
gzip_min_length 1000;
gzip_types text/plain text/html text/css application/json application/x-javascript
text/xml application/xml application/xml+rss text/javascript;
server {
listen 80;
client_max_body_size 50M;
server_name server.domain.com;
root /var/www;
index index.html index.php;
access_log /var/log/nginx/access.log main;
error_page 500 502 503 504 /500.html;
location = /500.html {
root /var/www;
}
location ~* ^.+.(jpg|jpeg|gif)$ {
root /var/www;
expires 30d;
}
location ~ \.php$ {
include /etc/nginx/fastcgi_params;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /var/www$fastcgi_script_name;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
}
}
}HOWTO: determine optimal fastcgi settings for Lighttpd
Anyone building a server with a LAMP stack today has tons of options, mine have evolved to using Varnish -> Lighttpd -> Xcache -> PHP5 -> MySQL. Once I had Lighttpd (aka Lighty) installed and running PHP pages I looked to optimize the configuration and push it as hard as possible for more speed. Of course lately I’ve been getting unexplained slowdowns, with many instances of php5-cgi appearing to be taking up almost all of my available CPU on `top`. Reading up on things it appears that I had max_procs, along with PHP_FCGI_CHILDREN, set far too high for the load I’m getting. When you start lighty it gives you the number of processes you’ve define, and then those in turn spawn the number of children you’ve specified. While my settings were too high, they were really overshooting things when you take into account that I’m using Xcache (which provides PHP pre-caching) and Varnish (for HTTP acceleration). So even though one of my dynamic sites that I’m working on to ‘monetize’ things is getting 700-800 hits each day, my caching strategy is taking the load away from the ever available Lighty. Because of this, lighty has much less to do, so giving it a ton of processes to just sit there and eat memory until they’re zombified is a waste. After reading the lighty FAQ and other posts specific to this, I’ve settled on the this for my fastcgi config block within my lighttpd.conf file. (more…)
HOWTO: use monit to keep Lighttpd and Varnish running
Thanks to a post from Steve over at debian-administration.org, I finally got around to setting up monit, the little monitoring app we use at work to keep things sane. I was getting around to installing it at home, but it became more urgent when Varnish went down last week; without it running there’s nothing to handle requests on :80, so as a webserver it’s dead. So here’s my monitrc for the webserver Lighttpd fronted by Varnish, acting in the reverse proxy/http accel role. Varn is listening on 80, then, if things aren’t cached, it forwards things on to Lighttpd listening on 82. Lighty also listens on the standard 443 for HTTPS requests, so we check that as well.
check process varnish with pidfile /var/run/varnishd.pid start program = "/etc/init.d/varnish start" stop program = "/etc/init.d/varnish stop" if cpu > 60% for 2 cycles then alert if cpu > 80% for 5 cycles then restart if totalmem > 200.0 MB for 5 cycles then restart if children > 250 then restart if loadavg(5min) greater than 10 for 8 cycles then stop if failed host 127.0.0.1 port 80 protocol http then restart if 3 restarts within 5 cycles then timeout check process lighttpd with pidfile /var/run/lighttpd.pid start program = "/etc/init.d/lighttpd start" stop program = "/etc/init.d/lighttpd stop" if cpu > 60% for 2 cycles then alert if cpu > 80% for 5 cycles then restart if totalmem > 200.0 MB for 5 cycles then restart if children > 250 then restart if loadavg(5min) greater than 10 for 8 cycles then stop if failed host 127.0.0.1 port 82 protocol http then restart if failed host 127.0.0.1 port 443 type tcpssl protocol http with timeout 15 seconds then restart if 3 restarts within 5 cycles then timeout
So now we have monit watching Lighttpd, Varnish, Postifx, MySQL and OpenSSH – restarting things if they fail, and emailing me the status when they do. Next on to some long term trending with Cacti providing some rrd graphing and then we’ll really have an idea of what this box is doing and be able to tune it accordingly.
HOWTO: log the user’s IP, not the proxy’s, in Lighttpd access log
When you run a webserver behind a reverse proxy or HTTP accelerator like Squid or Varnish, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP. This not only breaks any kind of tracking or reporting you want to run against your webserver logs, but it also takes away a datapoint I’ve had use for in general server admin tasks. This server runs Varnish in front of Lighttpd, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making Lighttpd (lighty) use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple; in lighttpd.conf, enter this:
accesslog.format = "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b /
\"%{Referer}i\" \"%{User-Agent}i\""For the definition of these variables, and plenty more, hit Lighty’s wiki. Props to the poster on the Varnish mailing list for bringing this up and reminding me to fix it! I’ve sent this link to the list so now it’s out there.
Hi, I'm fak3r and I've been online for over 15 years, blogging for 10. I work with open source software, various hardware and whatever else that I can hack on. I enjoy learning by doing, painting infrequently and listening to more music than the law allows. Feel free to follow me and comment on my thoughts and ideas; we're all in it together kid.
