Tag Archives: nginx

geek howto

HOWTO sort web-server logs to find top users

March 31, 2010 – 11:07 am

The other day I came across a situation where a web-server was getting hammered, and we needed to know who the offend(ers) were. While watching a logfile tail by at high speeds is always fun, we wanted to be able to sort the web-server access log and find top users, to be able to narrow down where the traffic was coming from. While we don’t want to block users that want to access our data, sometimes we need to throttle things back so one requester doesn’t overwhelm all the available bandwidth and make the site unusable for others. So after some playing around and digging on Google, we came up with a nice, succinct one liner to do this, here it is:

cat /path/to/access.log | awk '{print $1}' | sort | uniq -c | sort -n | tail
This is a preview of HOWTO sort web-server logs to find top users. Read the full post (161 words, 1 image, estimated 39 secs reading time)
Tags , , , , , , , , , , , , , , | Permalink | Comments (5)
geek howto

HOWTO: log the user's IP, not the proxy's, in nginx access log

December 18, 2008 – 12:55 pm
nginx

nginx

So back in January I had a post about HOWTO: log the user’s IP, not the proxy’s, in Lighttpd access log, but today I switched that system to run nginx (actually nginx has been running since early this year, I just got lazy on running Varnish) fronted again by Varnish. I had the same issue, but not much trouble solving it. Since I often refer to my own notes on fak3r, I’m recording it here for myself, and anyone streaming in from Google. So, as I talked about before, when you run a webserver behind Varnish doing http acceleration, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP. This not only breaks any kind of tracking or reporting you want to run against your webserver logs. Since this server runs Varnish in front of nginx, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making nginx use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple.  Edit your nginx.conf file:

This is a preview of HOWTO: log the user's IP, not the proxy's, in nginx access log. Read the full post (252 words, 1 image, estimated 1:00 mins reading time)
Tags , , , , | Permalink | Comments (7)
geek linux

HOWTO: Configure nginx for Debian / Ubuntu

May 5, 2008 – 12:20 pm

nginxUPDATE: I’m reworking my config blending in the security ideas found on camomel.org they’re really thought things through on this, this should make for a very secure environment.

This is a preview of HOWTO: Configure nginx for Debian / Ubuntu. Read the full post (349 words, 1 image, estimated 1:24 mins reading time)
Tags , , , , , | Permalink | Comments (12)