HOWTO: log the user’s IP, not the proxy’s, in nginx access log
nginx
So back in January I had a post about HOWTO: log the user’s IP, not the proxy’s, in Lighttpd access log, but today I switched that system to run nginx (actually nginx has been running since early this year, I just got lazy on running Varnish) fronted again by Varnish. I had the same issue, but not much trouble solving it. Since I often refer to my own notes on fak3r, I’m recording it here for myself, and anyone streaming in from Google. So, as I talked about before, when you run a webserver behind Varnish doing http acceleration, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP. This not only breaks any kind of tracking or reporting you want to run against your webserver logs. Since this server runs Varnish in front of nginx, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making nginx use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple. Edit your nginx.conf file:
vi /etc/nginx/nginx.conf
Once in the file, find the block about logging, and add the following to it:
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"' ;
access_log /var/log/nginx/access.log main;and finally, restart nginx
/etc/init.d/nginx restart
If you look at the logfiles now you’ll see the IP of the original requester!
HOWTO: disable IPv6 networking in Debian
IPv6 is ready?
Tonight I did ran netstat (`netstat -plunt`) on my Debian server and saw that I had some ports listening via IPv6. It’s a shame IPv6 hasn’t caught on as it’s better than IPv4 in virtually every way, and it should, especially since TCP/IPv4 was standardized in ARPANET RFC’s… in 1981! Also, IPv6 provides network level security via IPSec, which enables authentication of sender and encryption of communication path, to secure communications, all fun stuff, but while some point to the fact that the Beijing Olympics used IPv6 exclusively as a point in how far it’s come, that’s hardly saying much when the protocol went Alpha… in 1996! I mean I put things off and get distracted, sure, but come on! So while its adoption can be argued to be a case of the chicken before the egg, since I’m not using anything IPv6, nor do I or my ISP even have the capability to use it, it’s silly and perhaps dangerous to leave it running with open ports. So, if you’re not using it, disable it – it’s easy, just put on your pointy hat and follow along… (more…)
HOWTO: install Ruby on Rails on Debian or Ubuntu Linux easily
In the early days of this blog I used to run it on Typo, which *was* a great Ruby on Rails blogging platform (at one time). Unfortunately the project stalled (for years) and I ended up jumping ship after a few months of bugs and the ever crashing Rails server, WEBrick. Yes, if you search Netcraft you could see that was my *exposed* server at the time…not good! 
Now if you look, Typo is still kicking, and it *may* be a solid platform now, I hope it is, as I even contributed a ton of the achieved themes that live on in the ‘Theme Garden’ there. But on I moved into the world of MySQL/PHP front end sites via great apps like Drupal and Wordpress, fast forward, Ruby on Rails is a mature platform now, and I am evaluating webapps at work, so I needed to install Rails on Debian GNU/Linux (but of course these directions would work just as well in Ubuntu Linux. It’s amazing simple, I took some steps from the Ruby on Rails wiki, first install the dependencies for good measure: (more…)
Distributing biodiversity data globally
My current project at work will take me far into next year, and that’s good because I’m facing an unprecedented amount of data, that will only continue to grow. Because of this I’m finally getting to put my money where my mouth is. For years I’ve talked about my ideas and theories about how I could network disparate systems together and have them leverage each other to keep everything in sync. So, while working with Open Source to push boundaries I seem to find more ways to do more complex things. One basic idea that I’m working on now is that data sets are huge, and are only going to get huger (and hugerer) as time goes on, how to handle this has been solved a few different ways. Usually it’s someone like the Internet Archive who have 1000s of computers networked together to share the data (they are using some parts of hadoop for the distributed file system, and then nutch for search indexing) – but it’s still working from one central point of failure. I started doing research to find out how this has been solved before, and if my idea of building a BitTorrent network was sound – and I found some great information to build on. As I’m setting up my demo BitTorrent tracker in Debian, this info keeps me thinking of the best ways to implement my ideas. Much of my progress is due to the very helpful advice of Paul at Geograph Torrent Archive, a project that has somewhat similar goals. (more…)
HOWTO: conky config (conkyrc) for Debian Part 2
I changed around my Conky config, and it’s something you could do forever, but it’s great because it can be as heavy or light as you want it. Recently I dropped Gnome almost all together to run Openbox (full HOWTO on this forthcoming). I found a panel that will house things like nm-applet output, but was missing things like a simple clock, network activity, etc. So now, using most of the same code/look that I used here, I have a small, transparent strip at the bottom of the screen showing me time, date, proc, proc temp, network up, network down, and power status (battery, AC and the level of charge). It looks good, it’s light, it’s all I need. Nice to bring some of the memory requirements down from Gnome as well.
# Create own window instead of using desktop (required in nautilus)
own_window true
own_window_hints undecorated,below,skip_taskbar
background no
# Use double buffering (reduces flicker, may not work for everyone)
double_buffer true
# fiddle with window
use_spacer right
use_xft true
# Update interval in seconds
update_interval 3.0
# Minimum size of text area
minimum_size 10000 5
# Draw shades?
draw_shades yes
# Text stuff
draw_outline no # amplifies text if yes
draw_borders no
uppercase no # set to yes if you want all text to be in uppercase
# Stippled borders?
stippled_borders 8# border margins
border_margin 1
# border width
border_width 1
# Default colors and also border colors, grey90 == #e5e5e5
default_color white
default_shade_color black
default_outline_color white
own_window_colour brown
own_window_transparent yes
# Text alignment, other possible values are commented
#alignment top_left
#alignment top_right
alignment bottom_left
#alignment bottom_right
# Gap between borders of screen and text
gap_x 10
gap_y 5
# stuff after ‘TEXT’ will be formatted on screen
override_utf8_locale no
#xftfont Terminus:size=8
xftfont Terminus:size=10
xftalpha 0.8
#Mail:${color}${execi 300 python ~/scripts/gmail.py}
TEXT${offset 0}${color }${time %H:%M} ${color slate grey}${time %Z }Date: ${color }${time %a, } ${time %e %B %G} ${offset 0} ${offset 0} ${color slate grey}Proc:${color} $cpu%${offset 5}${acpitemp}C${offset 5}${cpugraph 16,100 000000 ffffff} ${offset 0} ${color slate grey}Net:${offset 5}${color}Up:${upspeed wlan0}k/s${offset 5}${upspeedgraph wlan0 16,100 000000 ffffff}${offset 0} ${color}Dn:${downspeed wlan0}k/s${color}${offset 5}${downspeedgraph wlan0 16,100 000000 ffffff} ${color slate grey} Power:${offset 5}${color}${battery}
Try it, you might like it – I’ll keep working on it, I’m sure I’ll find more things to add/improve. Conky rocks.
HOWTO: convert an AVI or to DVD (VOB) in Linux
I have some AVIs that I needed to get into the VOB format so I could burn them to DVD. I knew I could do this in Linux, but didn’t know how. Here is how I did it with Debian GNU/Linux (testing – Lenny). First I installed the GTK+ app, Avidemux (don’t worry, we’ll get back to the commandline soon). The I opened the AVI in Avidemux – after it imported it I clicked on FILE -> SAVE -> SAVE VIDEO – then choose where to save the file. I saved it as movie.mpg so it would work with my next step. This took some time for me, even with my duo-core 1.6Mhz 1Gig RAM laptop, but once it was complete I could play the mpg file in multimedia apps, so it worked. Now we need to make it into the format that you can burn to DVD. These are the weird looking folders named VIDEO_TS and AUDIO_TS that you’ll burn to the root of the DVD. To build this from an mpg is pretty easy, you need to install the commandline app dvdauthor. Once this is downloaded we need a simple XML file to tell dvdauthor what to do, so I created a base/simple one – open your text editor of choice (vim ftw!) and create a new file:
vi dvdauthor.xml
with the contents (NOTE: change all ( and )s to brackets, apparently neither the pre or code tag accepts brackets in them in wordpress – reminder to self, fix this):
(dvdauthor dest="DVD")
(vmgm /)
(titleset)
(titles)
(pgc)
(vob file="movie.mpg" chapters="0,15:00,30:00,45:00,1:00:00"/)
(/pgc)
(/titles)
(/titleset)Now run dvdauthor referring to new XML file:
dvdauthor -x dvdauthor.xml
When it’s done you’ll have a new directory called DVD, with the contents VIDEO_TS and AUDIO_TS. Open your favorite DVD burning app (I recommend GnomeBaker) and place those two direcotries in the root of the DVD and burn. Notice that in the XML file we called out chapters as 0, 15, 30, 45, 1 hour – obviously these can be further tweaked to be ‘real’ chapters, or left out all together. I’m sure there’s a GUI DVD authoring app that helps you do this, if you figure it out post below and let me know. HTH!
HOWTO: fix fonts in Debian Lenny/Sid
UPDATE: also, before you try this, make sure you have some good fonts installed, after a fresh install of Lenny at work, I needed to run this first: apt-get install ttf-mscorefonts-installer msttcorefonts
After a…slight slip up, I finally had the chance to install Linux from scratch on my laptop (Dell Vostro 1500) the way I’ve always wanted it with Debian GNU/Linux – Lenny and partitioned with LVM (Linux Volume Management). After that I set out to get the desktop fonts to look as good in Debian as they did (by default) in Ubuntu. After much scouring around online I found a pretty easy tweak that got me most of the way. As root:
dpkg-reconfigure fontconfig-config
In the dialog choose these options; Autohinter, Automatic and No. Now issue this command:
dpkg-reconfigure fontconfig
Logout and log back into your desktop and your fonts should be *noticeably* nicer looking. Of course after that you have to drive yourself crazy tweaking the settings for autohinting and RGB lines, installing any and all ttf-* fonts apt-get sees, but hey, that’s what choice is all about!
HOWTO: automatically reconfigure Xorg in Debian
If you’re like me, you’ve messed up your xorg.conf before and wanted to start over with the default that you know dpkg-reconfigure can set it to. Because of this I’m posting here because I’ve needed it multiple times in the past and have tired of looking it up! To automatically reconfigure Xorg in Debian or Ubuntu issue the following:
sudo dpkg-reconfigure -phigh xserver-xorg
Then logout/login or restart X via contrl-alt-backspace. As one who tweaks things a bit more than he should, this has saved me a few times now. Props go to a poster on this page.
HOWTO: sound after hibernate in Linux (Gusty/Lenny)
With all the tweaking to get my Dell Vostro 1500 working with Ubuntu, it’s still been an annoyance to get sound working evertime after hibernation. It goes to sleep fine, it wakes up fine, it obeys all of the power preferences I defined within Gnome fine too, it’s just that when it comes out of hibernation, the sound is usually off. It’s not muted, it’s off. Trying to restart alsa (the sound server) is a lession in frustration, so until now I’ve been ignoring it since it was rare that I would need it, but still…come on. This week I came across a solution in the Debian Forums that creates a new task for Linux to do before it shuts down and before it starts up.: “Create the file /etc/pm/sleep.d/49sound…
mkdir /etc/pm vi /etc/pm/sleep.d49sound
with the following contents:
function kill_sound_apps() {
pidsnd=$(lsof | grep /dev/snd | awk '{ print $2 }')
pidmixer=$(lsof | grep /dev/mixer | awk '{ print $2 }')
piddsp=$(lsof | grep /dev/dsp | awk '{ print $2 }')
kill $pidsnd $pidmixer $piddsp
}
case "$1" in
hibernate|suspend)
kill_sound_apps
modprobe -r snd_hda_intel
;;
thaw|resume)
modprobe snd_hda_intel
;;
*)
;;
esac
exit $?Then just make it executable:
# chmod +x /etc/pm/sleep.d/49sound
So before shutting down, Linux properly shuts down the sound, and when it comes back it, it properly starts the sound. As always, this *should* work, but the fact that it hasn’t been updated in Ubuntu Gusty is one of the reasons I’m shifting to Debian Lenny on this ‘top.
NOTE: yes, I am thinking of making Ignignokt my official HOWTO mascot. “Using a key to gouge expletives on another’s vehicle is a sign of trust and friendship“
HOWTO: use monit to keep Lighttpd and Varnish running
Thanks to a post from Steve over at debian-administration.org, I finally got around to setting up monit, the little monitoring app we use at work to keep things sane. I was getting around to installing it at home, but it became more urgent when Varnish went down last week; without it running there’s nothing to handle requests on :80, so as a webserver it’s dead. So here’s my monitrc for the webserver Lighttpd fronted by Varnish, acting in the reverse proxy/http accel role. Varn is listening on 80, then, if things aren’t cached, it forwards things on to Lighttpd listening on 82. Lighty also listens on the standard 443 for HTTPS requests, so we check that as well.
check process varnish with pidfile /var/run/varnishd.pid start program = "/etc/init.d/varnish start" stop program = "/etc/init.d/varnish stop" if cpu > 60% for 2 cycles then alert if cpu > 80% for 5 cycles then restart if totalmem > 200.0 MB for 5 cycles then restart if children > 250 then restart if loadavg(5min) greater than 10 for 8 cycles then stop if failed host 127.0.0.1 port 80 protocol http then restart if 3 restarts within 5 cycles then timeout check process lighttpd with pidfile /var/run/lighttpd.pid start program = "/etc/init.d/lighttpd start" stop program = "/etc/init.d/lighttpd stop" if cpu > 60% for 2 cycles then alert if cpu > 80% for 5 cycles then restart if totalmem > 200.0 MB for 5 cycles then restart if children > 250 then restart if loadavg(5min) greater than 10 for 8 cycles then stop if failed host 127.0.0.1 port 82 protocol http then restart if failed host 127.0.0.1 port 443 type tcpssl protocol http with timeout 15 seconds then restart if 3 restarts within 5 cycles then timeout
So now we have monit watching Lighttpd, Varnish, Postifx, MySQL and OpenSSH – restarting things if they fail, and emailing me the status when they do. Next on to some long term trending with Cacti providing some rrd graphing and then we’ll really have an idea of what this box is doing and be able to tune it accordingly.
HOWTO: notes on securing Debian
Looking over the Debian own harden-doc guide online, (which is a monster of a resource) as well as Debian Help’s security page gave me some excellent new ideas on how to secure Debian and Linux in general. Also today i found a netstat command with some nice switches to help you figure out what is listening on each port in an easy to read layout, -plunt:
netstat -plunt
Plus it’s fun to say, ‘plunt’. Lastly there’s a good overview of deborphan (which assists you in keeping your system clear of unneeded packages) with coverage on how to use it at Debian Adminstrator.org. But in the comments a thread talks about how it’s better to use aptitude, as this does it automatically.
Hi, I'm fak3r and I've been online for over 15 years, blogging for 10. I work with open source software, various hardware and whatever else that I can hack on. I enjoy learning by doing, painting infrequently and listening to more music than the law allows. Feel free to follow me and comment on my thoughts and ideas; we're all in it together kid.
