fak3r

I’m a big Bash fan, I know Perl is the more popular scripting language, and I’m slowly using it more, but hey, if I need something done, I can do it quicker in Bash (keeping in mind that I’m a systems guy, not a dev guy). While at work looking up Bash related syntax I came across a page describing how to run a webserver with 100 lines of Bash. It uses the old school GNU utility Netcat (nc) for communication between the pipes, and just a ton of basic logic and functions to pass it on to the user. It’s one of those things I look at and can’t believe it works, but it does. Of course security is unknown, as is the original author, but I consider this a reference on how to do networking things in Bash; who knows what I’ll use (parts) of it for. If anyone has details on who originally wrote this I’m all ears.

#!/bin/bash

function debug {
    local severity="$1"
    shift
    local message="$@"

    echo -n "`date -u`"    1>&2
    echo -ne '\t'        1>&2
    echo -n "$severity"    1>&2
    echo -ne '\t'        1>&2
    echo "$message"        1>&2
}

function fix_path {
    echo -n "$1" | head -n 1 | sed 's|^[/.-]*||' | sed 's|/\.*|/|g'
}

function serve_dir {
    local dir="`fix_path "$1"`"
    if [ "$dir" = "" ]; then
        dir="./"
    fi
    echo 'HTTP/1.1 200 OK'
    echo 'Content-type: text/html;charset=UTF-8'
    echo
    echo LISTING "$dir"
    echo '<br />'
    ls -p "$dir" | sed -e 's|^\(.*\)$|<a href="/'"$dir"'\1">\1</a><br />|'
}

function serve_file {
    echo 'HTTP/1.1 200 OK'
    echo 'Content-type: application/x-download-this'
    echo
    local file="`fix_path "$1"`"
    debug INFO serving file "$file"
    cat "$file"
}

function process {
    local url="`gawk '{print $2}' | head -n 1`"
    case "$url" in
        */)
            debug INFO Processing "$url" as dir
            serve_dir "$url"
            break
            ;;
        *)
            debug INFO Processing "$url" as file
            serve_file "$url"
            ;;
    esac
}

function serve {
    local port="$1"
    local sin="$2"
    local sout="$3"

    while debug INFO Running nc; do

        nc -l -p "$port" < "$sin" > "$sout" &amp;
        pid="$!"

        debug INFO Server PID: "$pid"

        trap cleanup SIGINT
        head -n 1 "$sout" | process > "$sin"
        trap - SIGINT

        debug INFO Killing nc

        kill "$pid"
    done

    debug INFO Quiting server
}

function cleanup {
    debug INFO Caught signal, quitting...
    rm -Rf "$tmp_dir"
    exit
}

tmp_dir="`mktemp -d -t http_server.XXXXXXXXXX`"
sin="$tmp_dir"/in
sout="$tmp_dir"/out
pid=0
port="$1"

mkfifo "$sin"
mkfifo "$sout"

debug INFO Starting server on port "$port"
serve "$port" "$sin" "$sout"
cleanup

I have some AVIs that I needed to get into the VOB format so I could burn them to DVD. I knew I could do this in Linux, but didn’t know how. Here is how I did it with Debian GNU/Linux (testing - Lenny). First I installed the GTK+ app, Avidemux (don’t worry, we’ll get back to the commandline soon). The I opened the AVI in Avidemux - after it imported it I clicked on FILE -> SAVE -> SAVE VIDEO - then choose where to save the file. I saved it as movie.mpg so it would work with my next step. This took some time for me, even with my duo-core 1.6Mhz 1Gig RAM laptop, but once it was complete I could play the mpg file in multimedia apps, so it worked. Now we need to make it into the format that you can burn to DVD. These are the weird looking folders named VIDEO_TS and AUDIO_TS that you’ll burn to the root of the DVD. To build this from an mpg is pretty easy, you need to install the commandline app dvdauthor. Once this is downloaded we need a simple XML file to tell dvdauthor what to do, so I created a base/simple one - open your text editor of choice (vim ftw!) and create a new file:

vi dvdauthor.xml

with the contents (NOTE: change all ( and )s to brackets, apparently neither the pre or code tag accepts brackets in them in wordpress - reminder to self, fix this):

(dvdauthor dest="DVD")
  (vmgm /)
   (titleset)
     (titles)
       (pgc)
         (vob file="movie.mpg" chapters="0,15:00,30:00,45:00,1:00:00"/)
       (/pgc)
      (/titles)
   (/titleset) 

Now run dvdauthor referring to new XML file:

dvdauthor -x dvdauthor.xml

When it’s done you’ll have a new directory called DVD, with the contents VIDEO_TS and AUDIO_TS. Open your favorite DVD burning app (I recommend GnomeBaker) and place those two direcotries in the root of the DVD and burn. Notice that in the XML file we called out chapters as 0, 15, 30, 45, 1 hour - obviously these can be further tweaked to be ‘real’ chapters, or left out all together. I’m sure there’s a GUI DVD authoring app that helps you do this, if you figure it out post below and let me know. HTH!

Have you ever found a batch of mp3s online on someone’s ‘Index of’ page?  I know you have (and if not, do a search for ‘google hacks’ in google to learn about the fun)  The issue always comes up that I find an album I want to grab, but the individual files are in a directory, so you have something like band_name-album_name/01-songone.mp3, and so on.  To grab all of them I used to issue a wget command, with the -r (recursive) switch like this:

wget -r http://www.someurl.com/band_name*

but then I’d end up with a ton of other files from the root directory that would take time and confuse the download so I’d have to search around for the mp3 payload.  I found a better way to do it, still using the -r for recursive search, but then only downloading the mp3s, forgoing any html pages or other directories in the root.  The command goes something like this:

wget -A mp3,mpg,mpeg,avi -r -l 3 http://www.someurl.com/band_name*

The curl command operates in a similar way. Its advantage is that it’s actively developed. Other similar commands that you can use are snarf, fget, and fetch, but I don’t see a direct advantage over wget with the proper filters.

This is a big deal for me, I played with Skype back in the day, but never really used it much since it required a second client, and I have always used Gaim (which is now Pidgin) to consolidate all of my accounts into one client and didn’t want to break out of that mold, but now I don’t have to.  Using the Skype API, Eion Robb has created a Pidgin plugin called Skype API plugin for Pidgin/libpurple/Adium.  Now I just add my user to the Pidgin accounts tab and I can now chat via Skype in Pidgin just like I chat with all my other contacts.  Note that you can’t do the video of Skype on Pidgin.  Mac users note that you can use this on Adium (my fav OS X chat client), which uses libpurple, which is the backend for Pidgin, on Mac.  So now I’m using Skype again, which is a propreitary app, thanks to them providing an API for the Open Source community to latch on to.  Ah, the circle of life…

If you’re like me you’re already running one of the Firefox 3 betas and loving the new features and stability lacking in the earlier series. The only issue now is that it’s taking extension writers time to update their extension to be compatible with the new Betas. Some of these extensions can almost hold me back to the 2.x series since they’re so useful, but 3 is just so much better in many ways. So, to install an old (2.x) extension into a new (3.x) Firefox just takes a little tweak to the install file. By default the extensions are hardcoded to something like 2.0.12 or the like, and *generally* nothing else needs to change to make it work with the 3.x series. (please re-read the *generally* part…done? ok). To do this, just right click on the extension (I’ll use the Gspace one for this example) and choose ‘Save as…’ Once you have it, open up the commandline (or force WinZip to open it, unsure of how to do that) with unzip (the extensions end with .xpi, but are really a kind zipfile):

unzip gspace-0.5.92-fx+fl.xpi

After it unpacks things, you’ll have an install.rdf file in your current directory, it’s just an XML structure file so you can open it in vi:

vi install.rdf

Search for the following block for targetApplication and find the variable for maxVersion:

2.0.0.*

Then simply modify the maxVersion variable to something like:

3.*

Then save that file. Next rebuild the xpi file with all the same files it had before, but this time with the modified install.rdf (NOTE: I renamed the file from .xpi to -edited.xpi so I could tell it apart)

zip -r -D gspace-0.5.92-fx+fl-edited.xpi chrome defaults license.txt install.rdf gpl.txt chrome_jar.manifest chrome.manifest

And lastly open the newly created xpi with Firefox - it should now install no problem. If there’s an issue bail out by manually deleting the .jar (and any other chrome or default files you saw when you unpacked). Worse case you’ll need to build a new profile, but that’s hardly a price to pay for the thrill of the adventure! (blah, that sounds pretty geeky, eh?)

Anyone building a server with a LAMP stack today has tons of options, mine have evolved to using Varnish -> Lighttpd -> Xcache -> PHP5 -> MySQL. Once I had Lighttpd (aka Lighty) installed and running PHP pages I looked to optimize the configuration and push it as hard as possible for more speed. Of course lately I’ve been getting unexplained slowdowns, with many instances of php5-cgi appearing to be taking up almost all of my available CPU on `top`. Reading up on things it appears that I had max_procs, along with PHP_FCGI_CHILDREN, set far too high for the load I’m getting. When you start lighty it gives you the number of processes you’ve define, and then those in turn spawn the number of children you’ve specified. While my settings were too high, they were really overshooting things when you take into account that I’m using Xcache (which provides PHP pre-caching) and Varnish (for HTTP acceleration). So even though one of my dynamic sites that I’m working on to ‘monetize’ things is getting 700-800 hits each day, my caching strategy is taking the load away from the ever available Lighty. Because of this, lighty has much less to do, so giving it a ton of processes to just sit there and eat memory until they’re zombified is a waste. After reading the lighty FAQ and other posts specific to this, I’ve settled on the this for my fastcgi config block within my lighttpd.conf file. (more…)

With all the tweaking to get my Dell Vostro 1500 working with Ubuntu, it’s still been an annoyance to get sound working evertime after hibernation. It goes to sleep fine, it wakes up fine, it obeys all of the power preferences I defined within Gnome fine too, it’s just that when it comes out of hibernation, the sound is usually off. It’s not muted, it’s off. Trying to restart alsa (the sound server) is a lession in frustration, so until now I’ve been ignoring it since it was rare that I would need it, but still…come on. This week I came across a solution in the Debian Forums that creates a new task for Linux to do before it shuts down and before it starts up.: “Create the file /etc/pm/sleep.d/49sound…

mkdir /etc/pm
vi /etc/pm/sleep.d49sound

with the following contents:

function kill_sound_apps() {
pidsnd=$(lsof | grep /dev/snd | awk '{ print $2 }')
pidmixer=$(lsof | grep /dev/mixer | awk '{ print $2 }')
piddsp=$(lsof | grep /dev/dsp | awk '{ print $2 }')
kill $pidsnd $pidmixer $piddsp
}

case "$1" in
hibernate|suspend)
kill_sound_apps
modprobe -r snd_hda_intel
;;
thaw|resume)
modprobe snd_hda_intel
;;
*)
;;
esac

exit $?

Then just make it executable:

# chmod +x /etc/pm/sleep.d/49sound

So before shutting down, Linux properly shuts down the sound, and when it comes back it, it properly starts the sound. As always, this *should* work, but the fact that it hasn’t been updated in Ubuntu Gusty is one of the reasons I’m shifting to Debian Lenny on this ‘top.

NOTE: yes, I am thinking of making Ignignokt my official HOWTO mascot. “Using a key to gouge expletives on another’s vehicle is a sign of trust and friendship“

These are some basic tweaks to speed up Firefox that have been tried and true for some time now.  I haven’t seen these collected in one place recently, so if you have Firefox and want to improve its performance, try these steps.  If you have a broadband connection (who doesn’t?), you can speed up your page loads considerably using these steps.   Basically you’re allowing Firefox to load multiple things on a page instead of one at a time. By default, it’s optimized for dialup connections (lowest common denominator) so here’s what you need to do to fix that.

1. Type “about:config” into the address bar and hit return. Type “network.http” in the filter field, and change the following settings (double-click on them to change them):
2. Set “network.http.pipelining” to “true”
3. Set “network.http.proxy.pipelining” to “true”
4. Set “network.http.pipelining.maxrequests” to 8 (recommended by Firefox devs)
5. Right-click anywhere and select New-> Integer.  Name it “nglayout.initialpaint.delay” and set its value to “0″. This value is the amount of time the browser waits before it acts on information it receives.  With it set to zero the page just pops up, it’s a dramatic change.

So give those a go, and don’t stop there, there’s plenty of Firefox add-ons to download and explore with; whatever you want to do online, there’s likely an add-on that’ll make it easier/better/faster.

Thanks to a post from Steve over at debian-administration.org, I finally got around to setting up monit, the little monitoring app we use at work to keep things sane.  I was getting around to installing it at home, but it became more urgent when Varnish went down last week; without it running there’s nothing to handle requests on :80, so as a webserver it’s dead. So here’s my monitrc for the webserver Lighttpd fronted by Varnish, acting in the reverse proxy/http accel role. Varn is listening on 80, then, if things aren’t cached, it forwards things on to Lighttpd listening on 82. Lighty also listens on the standard 443 for HTTPS requests, so we check that as well.

check process varnish with pidfile /var/run/varnishd.pid
start program = "/etc/init.d/varnish start"
stop program = "/etc/init.d/varnish stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 80 protocol http
then restart
if 3 restarts within 5 cycles then timeout

check process lighttpd with pidfile /var/run/lighttpd.pid
start program = "/etc/init.d/lighttpd start"
stop program = "/etc/init.d/lighttpd stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 82 protocol http
then restart
if failed host 127.0.0.1 port 443 type tcpssl protocol http
with timeout 15 seconds
then restart
if 3 restarts within 5 cycles then timeout

So now we have monit watching Lighttpd, Varnish, Postifx, MySQL and OpenSSH - restarting things if they fail, and emailing me the status when they do.  Next on to some long term trending with Cacti providing some rrd graphing and then we’ll really have an idea of what this box is doing and be able to tune it accordingly.

When you run a webserver behind a reverse proxy or HTTP accelerator like Squid or Varnish, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP.  This not only breaks any kind of tracking or reporting you want to run against your webserver logs, but it also takes away a datapoint I’ve had use for in general server admin tasks. This server runs Varnish in front of Lighttpd, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making Lighttpd (lighty) use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple; in lighttpd.conf, enter this:

accesslog.format = "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b /
\"%{Referer}i\" \"%{User-Agent}i\""

For the definition of these variables, and plenty more, hit Lighty’s wiki. Props to the poster on the Varnish mailing list for bringing this up and reminding me to fix it! I’ve sent this link to the list so now it’s out there.