look out honey 'cause I'm using technology

howto

HOWTO monitor your servers via Twitter

Alert: your server has failed!

The other day I got inspired to write a script that would allow me to monitor my servers via Twitter. The idea of having a column in Tweetdeck set aside to inform me of my servers’ statuses’ sounded cool, plus, it’s quicker than checking email.  I know sending tweets from the command-line had been done before, but after seeing briealeida’s post titled Tweeting Cron Jobs I really got inspired. While hers was written in Perl, I didn’t want to go that route since I had a few, self imposed, restrictions I wanted to stick to. One, I only wanted to use standard shell commands, the ones you get by default in Linux, so you would have absolutely no dependencies to install for this to work. Two, I wanted to see how much info I could stuff into a 140 character tweet, and still have it make sense. While I’m still working on adding more info, the current state of the script gives me a quick snapshot of seven specifics metrics on a selected server, which I’m quite happy with. To try it yourself only takes a few minutes. (more…)


HOWTO use monit to monitor sites and alert users

Ok, I’ve used the process management software, monit, since at least 2004, and it is simply an indespensible tool in my sysadmin cache. Basically it watches a process, say like Apache, and restarts it if it dies. But wait, that’s not all, it does tons of other things. Want it to watch it and restart it at a certain time? Sure. How about if it uses 50% of system memory in 5 cycles (cycles are checks, 120 seconds by default)? Yep, it’ll take care of that. How about watching a file and stopping a service and/or issuing an alert by email or web if the file’s UID, permission, or whatever has changed?   No problem. Disk space is greater than 90% on one partition you want an email to go out to the admin? Easy. Seriously, once you start using monit you’ll be amazed at what you can cover, it’s truly one of the best tools I’ve ever used, and of course it’s GPL’d open source.

So, this week we had an issue where a some of our sites were down, and the monitor that watches them were internal to our network, and relied on some of the same resources; which is lees than ideal. I have a remote server running at one of our partner’s sites, so it’s the perfect canidate to watch our sites from a ‘real world’ view. (more…)


HOWTO sort web-server logs to find top users

Wario is being greedy...

The other day I came across a situation where a web-server was getting hammered, and we needed to know who the offend(ers) were. While watching a logfile tail by at high speeds is always fun, we wanted to be able to sort the web-server access log and find top users, to be able to narrow down where the traffic was coming from. While we don’t want to block users that want to access our data, sometimes we need to throttle things back so one requester doesn’t overwhelm all the available bandwidth and make the site unusable for others. So after some playing around and digging on Google, we came up with a nice, succinct one liner to do this, here it is:

cat /path/to/access.log | awk '{print $1}' | sort | uniq -c | sort -n | tail

(more…)


HOWTO securely delete files in OS X on the commandline

So I’ve had my MacBook Pro for a few months now, and since I have a 500 Gig harddrive, I haven’t bothered to empty my trash yet. Now I’m on a work trip in China, and it makes me think about the privacy (internet and otherwise) that I have in the US, that I don’t expect here. In fact, since we’re blocked from posting to either Facebook or Twitter, I know this post will only make it there because this site will post if for me after I post it to my site (again, not something you’d think about just living in most other parts of the world). So what a good time to learn how to securely emptying my trash! The first thing I did was use the ‘Secure delete’ feature of the OS X trash folder, but with over 190,000 files to remove, it sat there at 0% while the fan spun up for about 15 minutes. That was it for me, it was clear it was going to take years for this to happen, so canceled that and hit Google to learn the right way to do it via the commandline. One of the best pages talks about srm a secure file deletion for posix systems that is installed by default on OS X. I’ve crafted my srm command to use the nice command to reduce the amount of overhead the process causes (again, the GUI version was taking over the system and heating things up quickly) and the sudo command to ensure all files would be deleted regardless of permission/ownership. In the end in looks like this:

nice -19 srm -rfv ~/.Trash/*

Yeah, while the -v flag will slow things down slightly, I prefer to have ‘verbose’ output from the command to understand exactly what it’s doing. Does anyone have better/more secure way to do this? Leave a message in the comments if you do, I’d love to learn more about this.


Ruby on Rails: gem install versus apt-get

railsUPDATE: Thanks to Ryan, Ant and Fern for the tips.  With that in mind I found an online Slicehost tutorial that contained the steps and explained how to install ruby via apt-get, then get the latest rubygems, install that manually, ran gem to update itself, then run gem to install rails – as suggested.  The steps I took from that page:
(more…)


HOWTO build your own open source Dropbox clone

I KAN HAZ OPEN-SRC DROPBX?UPDATE: Thanks to everyone who has contributed to this, and the Reddit thread, as it has provided some great ideas building off of my concept.  I’m starting to rethink about how we could have version control on top of things, and I’ll update things when I have more to share.  Also, does anyone have iFolder (thanks for the proper linksalubrium) working?  It looks like you need SUSE Linux, which I don’t have access to, plus I know most Novell projects need a *ton* of Mono dependencies installed to have any of their stuff working, at least on the server side; but it sounds like they have Mac, Linux and Windows clients, which is encouraging.  While for my needs something a bit more ‘close to the bone’ (as below) might be better for the server side, having it be inter-operable with something like iFolder could provide a lot more functionality for others.

First off, if you haven’t tried Dropbox, you should check it out; sync all of your computers via the Dropbox servers, their basic free service gives you 2Gigs of space and works cross-platform (Windows, Mac, Linux).  I use it daily at home and work, and just having a live backup of my main data for my work workstation, my home netbook, and any other computer I need to login to is a huge win.  Plus, I have various ‘shared’ folders that distribute certain data to certain users that I’ve granted access to, this means work details can be updated and automatically distributed to the folks I want to review/use the data.  I recommend everyone try it out, and see how useful it is, it’s turned into a game changer for me.  So a few months ago they made headlines on supporting Linux as they released the client as open source. While this got hopes up for many, it was only the client that was open source, the server is still proprietary.  While slightly disappointing, this is fine, they’re a company trying to make money.  I don’t fault them for this, it’s just that a free, portable service like that would be a killer app. (more…)


HOWTO: install Fedora-commons repository software on Debian

newlogo2So I’ve been using Fedora-commons for almost a year, first off, no it is NOT the Linux distribution, it is a digital repository used by libraries, museums, etc, worldwide to keep track of their digital collections.  For this Fedora-commons is very good at its job, but there was a steep learning curve when I first jumped in with, a complaint I’ve heard repeated by many who aren’t Java jockeys (just made that up).  Today I had to reinstall Fedora-commons on a new development server, and to be honest I had a couple of missteps along the way as I tried to remember my super cool moves to install this app.  Once something like this happens it’s a prime candidate for a HOWTO, so here is my super, simple HOWTO get Feodora-commons up and running in a development environment in Debian GNU/Linux (I’m running Lenny – 5.0) or Ubuntu Linux.  Notice that for simplicity’s sake, this uses Fedora-commons built in Tomcat implementation, for developing this is fine, for production I highly recommend installing a Tomcat via dpkg/apt-get, or whatever package manager you use, so that any security patches will be available for apt-get to automatically update.  I’ve also set it to use a local MySQL store, change to a different database if you need to. (more…)


HOWTO: configure MySQL’s my.cnf file

mysql-logoUPDATE: I recently used this MySQL tuner script, I basically went with what it told me, but I’m using a higher query_cache_size than it recommends, basically because I don’t see anything online saying it will hurt things.  So I’m now using the following values on my server:

[mysqld]
user=mysql
bind-address=127.0.0.1
datadir=/var/lib/mysql
pid-file=/var/run/mysqld/mysqld.pid
socket=/var/run/mysql/mysql.sock
port=3306
tmpdir=/tmp
language=/usr/share/mysql/english
skip-external-locking
query_cache_limit=64M
query_cache_size=32M
query_cache_type=1
max_connections=15
max_user_connections=300
interactive_timeout=100
wait_timeout=100
connect_timeout=10
thread_stack=128K
thread_cache_size=128
myisam-recover=BACKUP
key_buffer=64M
join_buffer=1M
max_allowed_packet=32M
table_cache=512M
sort_buffer_size=1M
read_buffer_size=1M
read_rnd_buffer_size=768K
max_connect_errors=10
thread_concurrency=4
myisam_sort_buffer_size=32M
skip-locking
skip-bdb
expire_logs_days=10
max_binlog_size=100M
server-id=1
[mysql.server]
user=mysql
basedir=/usr
[safe_mysqld]
bind-address=127.0.0.1
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
open_files_limit=8192
SAFE_MYSQLD_OPTIONS=”–defaults-file=/etc/my.cnf –log-slow-queries=/var/log/slow-queries.log”
[mysql]
[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M
[mysqlhotcopy]
interactive-timeout
max_heap_table_size = 64 M
tmp_table_size = 64 M
!includedir /etc/mysql/conf.d/

(more…)


HOWTO: serve jpeg2000 images with a scalable infrastructure

page1At the Biodiversity Heritage Library, we have replaced a proprietary jpeg2000 image server, that was straining under the load, with a new, open source jpeg2000 server, djatoka. Chris Freeland and Chris Moyers cover the background in far more detail on the BHL Blog, so here I’ll cover my rationale and decisions I made to provide a scalable, stable infrastructure to provide the images as efficiently as possible.

When I started sketching out how I wanted to run djatoka, I knew I wanted it to provide security, caching for performance and scalability and fault tolerance. Our server runs Tomcat, which I didn’t want to be public facing. Because of this I proxy Tomcat requests through Apache with the use of ajp_proxy, the successor to the old mod_jk. Initially I was using nginx in place of Apache, but after reading about all the functionality and performance improvements ajp_proxy offered, it was a no brainier; this is how to present Tomcat in a production environment.

(more…)


HOWTO: log the user’s IP, not the proxy’s, in nginx access log

nginx

nginx

So back in January I had a post about HOWTO: log the user’s IP, not the proxy’s, in Lighttpd access log, but today I switched that system to run nginx (actually nginx has been running since early this year, I just got lazy on running Varnish) fronted again by Varnish. I had the same issue, but not much trouble solving it. Since I often refer to my own notes on fak3r, I’m recording it here for myself, and anyone streaming in from Google. So, as I talked about before, when you run a webserver behind Varnish doing http acceleration, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP. This not only breaks any kind of tracking or reporting you want to run against your webserver logs. Since this server runs Varnish in front of nginx, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making nginx use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple.  Edit your nginx.conf file:

vi /etc/nginx/nginx.conf

Once in the file, find the block about logging, and add the following to it:

  log_format main '$remote_addr - $remote_user [$time_local] '
                '"$request" $status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for"' ;
  access_log /var/log/nginx/access.log main;

and finally, restart nginx

/etc/init.d/nginx restart

If you look at the logfiles now you’ll see the IP of the original requester!


HOWTO: disable IPv6 networking in Debian

IPv6 is ready?

Tonight I did ran netstat (`netstat -plunt`) on my Debian server and saw that I had some ports listening via IPv6.  It’s a shame IPv6 hasn’t caught on as it’s better than IPv4 in virtually every way, and it should, especially since TCP/IPv4 was standardized in ARPANET RFC’s… in 1981!  Also, IPv6 provides network level security via IPSec, which enables authentication of sender and encryption of communication path, to secure communications, all fun stuff, but while some point to the fact that the Beijing Olympics used IPv6 exclusively as a point in how far it’s come, that’s hardly saying much when the protocol went Alpha… in 1996!  I mean I put things off and get distracted, sure, but come on!  So while its adoption can be argued to be a case of the chicken before the egg, since I’m not using anything IPv6, nor do I or my ISP even have the capability to use it, it’s silly and perhaps dangerous to leave it running with open ports.  So, if you’re not using it, disable it – it’s easy, just put on your pointy hat and follow along… (more…)


HOWTO: install Ruby on Rails on Debian or Ubuntu Linux easily

In the early days of this blog I used to run it on Typo, which *was* a great Ruby on Rails blogging platform (at one time).  Unfortunately the project stalled (for years) and I ended up jumping ship after a few months of bugs and the ever crashing Rails server, WEBrick.  Yes, if you search Netcraft you could see that was my *exposed* server at the time…not good! ;)   Now if you look, Typo is still kicking, and it *may* be a solid platform now, I hope it is, as I even contributed a ton of the achieved themes that live on in the ‘Theme Garden’ there.  But on I moved into the world of MySQL/PHP front end sites via great apps like Drupal and WordPress, fast forward, Ruby on Rails is a mature platform now, and I am evaluating webapps at work, so I needed to install Rails on Debian GNU/Linux (but of course these directions would work just as well in Ubuntu Linux.  It’s amazing simple, I took some steps from the Ruby on Rails wiki, first install the dependencies for good measure: (more…)


How to become a hacker

Glider - ESR's hacker emblem

The Glider: A Universal Hacker Emblem

There has long been a movement in the geek community to expunge the negative thoughts attached to the word hacker, the image to the right The Glider, being one of the latest and most visible. In the beginning there were hackers (people who worked on computers, programmed and made things work) and crackers (people who would use computers for nefarious purposes, crimes, viruses, etc), these were two distinct camps, with some miscreants jumping the fence back and forth to confuse the issue. Regardless, somewhere along the way popular culture (movies, news, your teachers probably) began to equate hacking as being the bad, crime ridden activity that cracker was supposed to cover. I think it’s a moot point now, as even my Dad was shocked when he learned my annual DefCon trip is billed as “largest hacking conference in the world”.  I gave him the above explanation, but I’m unsure if he really believes it. Regardless, the original “How to become a hacker” paper written by Eric S Raymond is always cited as the quintessential word on the use of the word hacker. I found it mirrored online, and it’s a worthwhile read if you have any interest in the topic, or want to cement your own views of your hobby.  For now, if you don’t want to read the entire verbiage, here’s the intro to learn and take with you. (more…)


HOWTO: send commandline email with attachments

You've got mail!

Are you like me, do you have scripts running on servers and you need to know what they know? If there’s output in a file you can sed/grep/awk info out of them and have them emailed to you, but if you don’t know specifically what you’re looking for you may need the entire file/log/whatever. You’ll need a utility called uuencode, which is a utility that, (more…)


HOWTO: webserver in 100 lines of Bash

I’m a big Bash fan, I know Perl is the more popular scripting language, and I’m slowly using it more, but hey, if I need something done, I can do it quicker in Bash (keeping in mind that I’m a systems guy, not a dev guy). While at work looking up Bash related syntax I came across a page describing how to run a webserver with 100 lines of Bash. It uses the old school GNU utility Netcat (nc) for communication between the pipes, and just a ton of basic logic and functions to pass it on to the user. It’s one of those things I look at and can’t believe it works, but it does. Of course security is unknown, as is the original author, but I consider this a reference on how to do networking things in Bash; who knows what I’ll use (parts) of it for. If anyone has details on who originally wrote this I’m all ears.[sourcecode language='xml']#!/bin/bash

function debug {
local severity=”$1″
shift
local message=”$@”

echo -n “`date -u`” 1>&2
echo -ne ‘\t’ 1>&2
echo -n “$severity” 1>&2
echo -ne ‘\t’ 1>&2
echo “$message” 1>&2
}

function fix_path {
echo -n “$1″ | head -n 1 | sed ‘s|^[/.-]*||’ | sed ‘s|/\.*|/|g’
}

function serve_dir {
local dir=”`fix_path “$1″`”
if [ "$dir" = "" ]; then
dir=”./”
fi
echo ‘HTTP/1.1 200 OK’
echo ‘Content-type: text/html;charset=UTF-8′
echo
echo LISTING “$dir”
echo ‘

ls -p “$dir” | sed -e ‘s|^\(.*\)$|\1
|’
}

function serve_file {
echo ‘HTTP/1.1 200 OK’
echo ‘Content-type: application/x-download-this’
echo
local file=”`fix_path “$1″`”
debug INFO serving file “$file”
cat “$file”
}

function process {
local url=”`gawk ‘{print $2}’ | head -n 1`”
case “$url” in
*/)
debug INFO Processing “$url” as dir
serve_dir “$url”
break
;;
*)
debug INFO Processing “$url” as file
serve_file “$url”
;;
esac
}

function serve {
local port=”$1″
local sin=”$2″
local sout=”$3″

while debug INFO Running nc; do

nc -l -p “$port” < "$sin" > “$sout” &amp;
pid=”$!”

debug INFO Server PID: “$pid”

trap cleanup SIGINT
head -n 1 “$sout” | process > “$sin”
trap – SIGINT

debug INFO Killing nc

kill “$pid”
done

debug INFO Quiting server
}

function cleanup {
debug INFO Caught signal, quitting…
rm -Rf “$tmp_dir”
exit
}

tmp_dir=”`mktemp -d -t http_server.XXXXXXXXXX`”
sin=”$tmp_dir”/in
sout=”$tmp_dir”/out
pid=0
port=”$1″

mkfifo “$sin”
mkfifo “$sout”

debug INFO Starting server on port “$port”
serve “$port” “$sin” “$sout”
cleanup[/sourcecode]


HOWTO: convert an AVI or to DVD (VOB) in Linux

I have some AVIs that I needed to get into the VOB format so I could burn them to DVD. I knew I could do this in Linux, but didn’t know how. Here is how I did it with Debian GNU/Linux (testing – Lenny). First I installed the GTK+ app, Avidemux (don’t worry, we’ll get back to the commandline soon). The I opened the AVI in Avidemux – after it imported it I clicked on FILE -> SAVE -> SAVE VIDEO – then choose where to save the file. I saved it as movie.mpg so it would work with my next step. This took some time for me, even with my duo-core 1.6Mhz 1Gig RAM laptop, but once it was complete I could play the mpg file in multimedia apps, so it worked. Now we need to make it into the format that you can burn to DVD. These are the weird looking folders named VIDEO_TS and AUDIO_TS that you’ll burn to the root of the DVD. To build this from an mpg is pretty easy, you need to install the commandline app dvdauthor. Once this is downloaded we need a simple XML file to tell dvdauthor what to do, so I created a base/simple one – open your text editor of choice (vim ftw!) and create a new file:

vi dvdauthor.xml

with the contents (NOTE: change all ( and )s to brackets, apparently neither the pre or code tag accepts brackets in them in wordpress – reminder to self, fix this):

(dvdauthor dest="DVD")
  (vmgm /)
   (titleset)
     (titles)
       (pgc)
         (vob file="movie.mpg" chapters="0,15:00,30:00,45:00,1:00:00"/)
       (/pgc)
      (/titles)
   (/titleset)

Now run dvdauthor referring to new XML file:

dvdauthor -x dvdauthor.xml

When it’s done you’ll have a new directory called DVD, with the contents VIDEO_TS and AUDIO_TS. Open your favorite DVD burning app (I recommend GnomeBaker) and place those two direcotries in the root of the DVD and burn. Notice that in the XML file we called out chapters as 0, 15, 30, 45, 1 hour – obviously these can be further tweaked to be ‘real’ chapters, or left out all together. I’m sure there’s a GUI DVD authoring app that helps you do this, if you figure it out post below and let me know. HTH!


HOWTO: recursively download only specific file types

Have you ever found a batch of mp3s online on someone’s ‘Index of’ page?  I know you have (and if not, do a search for ‘google hacks’ in google to learn about the fun)  The issue always comes up that I find an album I want to grab, but the individual files are in a directory, so you have something like band_name-album_name/01-songone.mp3, and so on.  To grab all of them I used to issue a wget command, with the -r (recursive) switch like this:

wget -r http://www.someurl.com/band_name*

but then I’d end up with a ton of other files from the root directory that would take time and confuse the download so I’d have to search around for the mp3 payload.  I found a better way to do it, still using the -r for recursive search, but then only downloading the mp3s, forgoing any html pages or other directories in the root.  The command goes something like this:

wget -A mp3,mpg,mpeg,avi -r -l 3 http://www.someurl.com/band_name*

The curl command operates in a similar way. Its advantage is that it’s actively developed. Other similar commands that you can use are snarf, fget, and fetch, but I don’t see a direct advantage over wget with the proper filters.


Chat on Skype via Pidgin on Linux (or Adium on Mac)

Skype on Linux - FTW!This is a big deal for me, I played with Skype back in the day, but never really used it much since it required a second client, and I have always used Gaim (which is now Pidgin) to consolidate all of my accounts into one client and didn’t want to break out of that mold, but now I don’t have to.  Using the Skype API, Eion Robb has created a Pidgin plugin called Skype API plugin for Pidgin/libpurple/Adium.  Now I just add my user to the Pidgin accounts tab and I can now chat via Skype in Pidgin just like I chat with all my other contacts.  Note that you can’t do the video of Skype on Pidgin.  Mac users note that you can use this on Adium (my fav OS X chat client), which uses libpurple, which is the backend for Pidgin, on Mac.  So now I’m using Skype again, which is a propreitary app, thanks to them providing an API for the Open Source community to latch on to.  Ah, the circle of life…


HOWTO: make old Firefox extensions install in 3.0 betas

If you’re like me you’re already running one of the Firefox 3 betas and loving the new features and stability lacking in the earlier series. The only issue now is that it’s taking extension writers time to update their extension to be compatible with the new Betas. Some of these extensions can almost hold me back to the 2.x series since they’re so useful, but 3 is just so much better in many ways. So, to install an old (2.x) extension into a new (3.x) Firefox just takes a little tweak to the install file. By default the extensions are hardcoded to something like 2.0.12 or the like, and *generally* nothing else needs to change to make it work with the 3.x series. (please re-read the *generally* part…done? ok). To do this, just right click on the extension (I’ll use the Gspace one for this example) and choose ‘Save as…’ Once you have it, open up the commandline (or force WinZip to open it, unsure of how to do that) with unzip (the extensions end with .xpi, but are really a kind zipfile):

unzip gspace-0.5.92-fx+fl.xpi

After it unpacks things, you’ll have an install.rdf file in your current directory, it’s just an XML structure file so you can open it in vi:

vi install.rdf

Search for the following block for targetApplication and find the variable for maxVersion:

2.0.0.*

Then simply modify the maxVersion variable to something like:

3.*

Then save that file. Next rebuild the xpi file with all the same files it had before, but this time with the modified install.rdf (NOTE: I renamed the file from .xpi to -edited.xpi so I could tell it apart)

zip -r -D gspace-0.5.92-fx+fl-edited.xpi chrome defaults license.txt install.rdf gpl.txt chrome_jar.manifest chrome.manifest

And lastly open the newly created xpi with Firefox – it should now install no problem. If there’s an issue bail out by manually deleting the .jar (and any other chrome or default files you saw when you unpacked). Worse case you’ll need to build a new profile, but that’s hardly a price to pay for the thrill of the adventure! (blah, that sounds pretty geeky, eh?)


HOWTO: determine optimal fastcgi settings for Lighttpd

PHP Fast-cgiAnyone building a server with a LAMP stack today has tons of options, mine have evolved to using Varnish -> Lighttpd -> Xcache -> PHP5 -> MySQL. Once I had Lighttpd (aka Lighty) installed and running PHP pages I looked to optimize the configuration and push it as hard as possible for more speed. Of course lately I’ve been getting unexplained slowdowns, with many instances of php5-cgi appearing to be taking up almost all of my available CPU on `top`. Reading up on things it appears that I had max_procs, along with PHP_FCGI_CHILDREN, set far too high for the load I’m getting. When you start lighty it gives you the number of processes you’ve define, and then those in turn spawn the number of children you’ve specified. While my settings were too high, they were really overshooting things when you take into account that I’m using Xcache (which provides PHP pre-caching) and Varnish (for HTTP acceleration). So even though one of my dynamic sites that I’m working on to ‘monetize’ things is getting 700-800 hits each day, my caching strategy is taking the load away from the ever available Lighty. Because of this, lighty has much less to do, so giving it a ton of processes to just sit there and eat memory until they’re zombified is a waste. After reading the lighty FAQ and other posts specific to this, I’ve settled on the this for my fastcgi config block within my lighttpd.conf file. (more…)


HOWTO: sound after hibernate in Linux (Gusty/Lenny)

Ignignokt says - Using a key to gouge expletives on another’s vehicle is a sign of trust and friendshipWith all the tweaking to get my Dell Vostro 1500 working with Ubuntu, it’s still been an annoyance to get sound working evertime after hibernation. It goes to sleep fine, it wakes up fine, it obeys all of the power preferences I defined within Gnome fine too, it’s just that when it comes out of hibernation, the sound is usually off. It’s not muted, it’s off. Trying to restart alsa (the sound server) is a lession in frustration, so until now I’ve been ignoring it since it was rare that I would need it, but still…come on. This week I came across a solution in the Debian Forums that creates a new task for Linux to do before it shuts down and before it starts up.: “Create the file /etc/pm/sleep.d/49sound…

mkdir /etc/pm
vi /etc/pm/sleep.d49sound

with the following contents:

function kill_sound_apps() {
pidsnd=$(lsof | grep /dev/snd | awk '{ print $2 }')
pidmixer=$(lsof | grep /dev/mixer | awk '{ print $2 }')
piddsp=$(lsof | grep /dev/dsp | awk '{ print $2 }')
kill $pidsnd $pidmixer $piddsp
}

case "$1" in
hibernate|suspend)
kill_sound_apps
modprobe -r snd_hda_intel
;;
thaw|resume)
modprobe snd_hda_intel
;;
*)
;;
esac

exit $?

Then just make it executable:

# chmod +x /etc/pm/sleep.d/49sound

So before shutting down, Linux properly shuts down the sound, and when it comes back it, it properly starts the sound. As always, this *should* work, but the fact that it hasn’t been updated in Ubuntu Gusty is one of the reasons I’m shifting to Debian Lenny on this ‘top.

NOTE: yes, I am thinking of making Ignignokt my official HOWTO mascot. “Using a key to gouge expletives on another’s vehicle is a sign of trust and friendship


HOWTO: Speedup Firefox in 5 easy steps

Firefox logoThese are some basic tweaks to speed up Firefox that have been tried and true for some time now.  I haven’t seen these collected in one place recently, so if you have Firefox and want to improve its performance, try these steps.  If you have a broadband connection (who doesn’t?), you can speed up your page loads considerably using these steps.   Basically you’re allowing Firefox to load multiple things on a page instead of one at a time. By default, it’s optimized for dialup connections (lowest common denominator) so here’s what you need to do to fix that.

  1. Type “about:config” into the address bar and hit return. Type “network.http” in the filter field, and change the following settings (double-click on them to change them):
  2. Set “network.http.pipelining” to “true”
  3. Set “network.http.proxy.pipelining” to “true”
  4. Set “network.http.pipelining.maxrequests” to 8 (recommended by Firefox devs)
  5. Right-click anywhere and select New-> Integer.  Name it “nglayout.initialpaint.delay” and set its value to “0″. This value is the amount of time the browser waits before it acts on information it receives.  With it set to zero the page just pops up, it’s a dramatic change.

So give those a go, and don’t stop there, there’s plenty of Firefox add-ons to download and explore with; whatever you want to do online, there’s likely an add-on that’ll make it easier/better/faster.


HOWTO: use monit to keep Lighttpd and Varnish running

IgnignoktThanks to a post from Steve over at debian-administration.org, I finally got around to setting up monit, the little monitoring app we use at work to keep things sane.  I was getting around to installing it at home, but it became more urgent when Varnish went down last week; without it running there’s nothing to handle requests on :80, so as a webserver it’s dead. So here’s my monitrc for the webserver Lighttpd fronted by Varnish, acting in the reverse proxy/http accel role. Varn is listening on 80, then, if things aren’t cached, it forwards things on to Lighttpd listening on 82. Lighty also listens on the standard 443 for HTTPS requests, so we check that as well.

check process varnish with pidfile /var/run/varnishd.pid
start program = "/etc/init.d/varnish start"
stop program = "/etc/init.d/varnish stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 80 protocol http
then restart
if 3 restarts within 5 cycles then timeout

check process lighttpd with pidfile /var/run/lighttpd.pid
start program = "/etc/init.d/lighttpd start"
stop program = "/etc/init.d/lighttpd stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 82 protocol http
then restart
if failed host 127.0.0.1 port 443 type tcpssl protocol http
with timeout 15 seconds
then restart
if 3 restarts within 5 cycles then timeout

So now we have monit watching Lighttpd, Varnish, Postifx, MySQL and OpenSSH – restarting things if they fail, and emailing me the status when they do.  Next on to some long term trending with Cacti providing some rrd graphing and then we’ll really have an idea of what this box is doing and be able to tune it accordingly.


HOWTO: log the user’s IP, not the proxy’s, in Lighttpd access log

Lighttpd - fly lightWhen you run a webserver behind a reverse proxy or HTTP accelerator like Squid or Varnish, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP.  This not only breaks any kind of tracking or reporting you want to run against your webserver logs, but it also takes away a datapoint I’ve had use for in general server admin tasks. This server runs Varnish in front of Lighttpd, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making Lighttpd (lighty) use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple; in lighttpd.conf, enter this:

accesslog.format = "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b /
\"%{Referer}i\" \"%{User-Agent}i\""

For the definition of these variables, and plenty more, hit Lighty’s wiki. Props to the poster on the Varnish mailing list for bringing this up and reminding me to fix it! I’ve sent this link to the list so now it’s out there.