Varnish web accelerator: the right tool for the job?

Note: The following testing and writeup occurred in the Fall of 2007 following months of research and conjecture.  I repost it now because it was not originally posted publicly, and because the results are still a driving factor in how I architect systems for web production.  This week I am implementing Varnish to enhance an [...]

Read the full story »
More posts from this section »
More posts from this section »
art

geek

howto

music

politics

geek, howto »

HOWTO: log the user’s IP, not the proxy’s, in nginx access log
nginx

nginx

So back in January I had a post about HOWTO: log the user’s IP, not the proxy’s, in Lighttpd access log, but today I switched that system to run nginx (actually nginx has been running since early this year, I just got lazy on running Varnish) fronted again by Varnish. I had the same issue, but not much trouble solving it. Since I often refer to my own notes on fak3r, I’m recording it here for myself, and anyone streaming in from Google. So, as I talked about before, when you run a webserver behind Varnish doing http acceleration, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP. This not only breaks any kind of tracking or reporting you want to run against your webserver logs. Since this server runs Varnish in front of nginx, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making nginx use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple.  Edit your nginx.conf file:

vi /etc/nginx/nginx.conf

Once in the file, find the block about logging, and add the following to it:

  log_format main '$remote_addr - $remote_user [$time_local] '
                '"$request" $status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for"' ;
  access_log /var/log/nginx/access.log main;

and finally, restart nginx

/etc/init.d/nginx restart

If you look at the logfiles now you’ll see the IP of the original requester!

HOWTO: disable IPv6 networking in Debian HOWTO: disable IPv6 networking in Debian

Tonight I did ran netstat (`netstat -plunt`) on my Debian server and saw that I had some ports listening via IPv6.  It’s a shame IPv6 hasn’t caught on as it’s better than IPv4 in virtually every way, and it should, especially since TCP/IPv4 was standardized in ARPANET RFC’s… in 1981!  Also, IPv6 provides network [...]

Olbermann’s commentary on Prop 8 Olbermann’s commentary on Prop 8

The fact that Proposition 8 passed this last week in Califonia is a sad commentary on people’s morays and the discrimination that they still think is ok to pass on to others.  Of course the statistic that African Americans (whose strong turnout is being attributed to Obama) voting 2 to 1 *for* the measure is [...]

HOWTO: install Ruby on Rails on Debian or Ubuntu Linux easily HOWTO: install Ruby on Rails on Debian or Ubuntu Linux easily

In the early days of this blog I used to run it on Typo, which *was* a great Ruby on Rails blogging platform (at one time).  Unfortunately the project stalled (for years) and I ended up jumping ship after a few months of bugs and the ever crashing Rails server, WEBrick.  Yes, if you search [...]

Equal Rights for All Equal Rights for All

Here’s a great, recent News of the Weird article,”Roy Hollander filed a civil rights lawsuit against Columbia University in New York City in August, claiming that its “women’s studies” curriculum teaches a religion-like philosophy that oppresses men by blaming them for nearly all social problems. (When interviewed by the New York Daily News, Hollander declined [...]

Wassup 2008 Wassup 2008

File under: it’s funny cause it’s true. I mean, tru, tru.

A rural county pining for a racist past A rural county pining for a racist past

With all the polls it feels like there’s been a sea change in how America sees race, or maybe not. Maybe it’s W’s 22% approval rating skewing the numbers, because there are obvious still some serious misconceptions and racism playing big in the election. Interviews in the ‘bootheal’ region of Missouri (only a few [...]

Army: Twitter could be a terrorist tool Army: Twitter could be a terrorist tool

Hmmm…so the Army has claimed that terrorists may be ‘tweeting’ along to plan and organize attacks. Well yeah, I guess they could use Gmail, Slashdot comments and other things the same way, it seems their claim is that since this is more ‘real time’ it could be a danger. Ok, oh, and they [...]

Distributing biodiversity data globally Distributing biodiversity data globally

My current project at work will take me far into next year, and that’s good because I’m facing an unprecedented amount of data, that will only continue to grow.  Because of this I’m finally getting to put my money where my mouth is.  For years I’ve talked about my ideas and theories about how I [...]

How to become a hacker How to become a hacker

There has long been a movement in the geek community to expunge the negative thoughts attached to the word hacker, the image to the right The Glider, being one of the latest and most visible. In the beginning there were hackers (people who worked on computers, programmed and made things work) and crackers (people [...]

Volunteering for change Volunteering for change

This afternoon I volunteered at the local field office in South St. Louis for Barack Obama. When I got there there was a flurry of activity, and I could tell people were there because they wanted to be. As we waited for others to show up to get started, I saw a pile [...]

HOWTO: send commandline email with attachments HOWTO: send commandline email with attachments

Are you like me, do you have scripts running on servers and you need to know what they know? If there’s output in a file you can sed/grep/awk info out of them and have them emailed to you, but if you don’t know specifically what you’re looking for you may need the entire file/log/whatever. [...]

McCain’s acceptance speech echos Bush’s McCain’s acceptance speech echos Bush’s

This is glorious, The Daily Show obliterates McCain’s acceptance speech by interspersing clips of Dubya’s acceptance speech. It’s funny that much of it is the same - WORD FOR WORD! So I guess that’s the change they keep talking about. Damn McCain, you crazy Mavrick! (sic)

HOWTO: webserver in 100 lines of Bash HOWTO: webserver in 100 lines of Bash

I’m a big Bash fan, I know Perl is the more popular scripting language, and I’m slowly using it more, but hey, if I need something done, I can do it quicker in Bash (keeping in mind that I’m a systems guy, not a dev guy). While at work looking up Bash related syntax [...]