Vote Obama/Biden 2008 - Together We Can!
 

fak3r

dim high beams for oncoming traffic

Archive for the ‘howto’


HOWTO: webserver in 100 lines of Bash

I’m a big Bash fan, I know Perl is the more popular scripting language, and I’m slowly using it more, but hey, if I need something done, I can do it quicker in (keeping in mind that I’m a systems guy, not a dev guy). While at work looking up related syntax I came across a page describing how to run a webserver with 100 lines of Bash. It uses the old school GNU utility Netcat (nc) for communication between the pipes, and just a ton of basic logic and functions to pass it on to the user. It’s one of those things I look at and can’t believe it works, but it does. Of course security is unknown, as is the original author, but I consider this a reference on how to do things in ; who knows what I’ll use (parts) of it for. If anyone has details on who originally wrote this I’m all ears.

#!/bin/

function debug {
    local severity="$1"
    shift
    local message="$@"

    echo -n "`date -u`"    1>&2
    echo -ne '\t'        1>&2
    echo -n "$severity"    1>&2
    echo -ne '\t'        1>&2
    echo "$message"        1>&2
}

function fix_path {
    echo -n "$1" | head -n 1 | sed 's|^[/.-]*||' | sed 's|/\.*|/|g'
}

function serve_dir {
    local dir="`fix_path "$1"`"
    if [ "$dir" = "" ]; then
        dir="./"
    fi
    echo 'HTTP/1.1 200 OK'
    echo 'Content-type: text/html;charset=UTF-8'
    echo
    echo LISTING "$dir"
    echo '<br />'
    ls -p "$dir" | sed -e 's|^\(.*\)$|<a href="/'"$dir"'\1">\1</a><br />|'
}

function serve_file {
    echo 'HTTP/1.1 200 OK'
    echo 'Content-type: application/x--this'
    echo
    local file="`fix_path "$1"`"
    debug INFO serving file "$file"
    cat "$file"
}

function process {
    local url="`gawk '{print $2}' | head -n 1`"
    case "$url" in
        */)
            debug INFO Processing "$url" as dir
            serve_dir "$url"
            break
            ;;
        *)
            debug INFO Processing "$url" as file
            serve_file "$url"
            ;;
    esac
}

function serve {
    local port="$1"
    local sin="$2"
    local sout="$3"

    while debug INFO Running nc; do

        nc -l -p "$port" < "$sin" > "$sout" &amp;
        pid="$!"

        debug INFO Server PID: "$pid"

        trap cleanup SIGINT
        head -n 1 "$sout" | process > "$sin"
        trap - SIGINT

        debug INFO Killing nc

        kill "$pid"
    done

    debug INFO Quiting server
}

function cleanup {
    debug INFO Caught signal, quitting...
    rm -Rf "$tmp_dir"
    exit
}

tmp_dir="`mktemp -d -t http_server.XXXXXXXXXX`"
sin="$tmp_dir"/in
sout="$tmp_dir"/out
pid=0
port="$1"

mkfifo "$sin"
mkfifo "$sout"

debug INFO Starting server on port "$port"
serve "$port" "$sin" "$sout"
cleanup

HOWTO: convert an AVI or to DVD (VOB) in Linux

I have some AVIs that I needed to get into the format so I could burn them to . I knew I could do this in , but didn’t know how. Here is how I did it with GNU/ (testing - ). First I installed the GTK+ app, Avidemux (don’t worry, we’ll get back to the commandline soon). The I opened the in Avidemux - after it imported it I clicked on FILE -> SAVE -> SAVE VIDEO - then choose where to save the file. I saved it as movie.mpg so it would work with my next step. This took some time for me, even with my duo-core 1.6Mhz 1Gig RAM laptop, but once it was complete I could play the mpg file in multimedia apps, so it worked. Now we need to make it into the format that you can burn to . These are the weird looking folders named VIDEO_TS and AUDIO_TS that you’ll burn to the root of the . To build this from an mpg is pretty easy, you need to install the commandline app dvdauthor. Once this is downloaded we need a simple XML file to tell dvdauthor what to do, so I created a base/simple one - open your text editor of choice (vim ftw!) and create a new file:

vi dvdauthor.xml

with the contents (NOTE: change all ( and )s to brackets, apparently neither the pre or tag accepts brackets in them in wordpress - reminder to self, fix this):

(dvdauthor dest="")
  (vmgm /)
   (titleset)
     (titles)
       (pgc)
         ( file="movie.mpg" chapters="0,15:00,30:00,45:00,1:00:00"/)
       (/pgc)
      (/titles)
   (/titleset) 

Now run dvdauthor to new XML file:

dvdauthor -x dvdauthor.xml

When it’s done you’ll have a new directory called , with the contents VIDEO_TS and AUDIO_TS. Open your favorite burning app (I recommend GnomeBaker) and place those two direcotries in the root of the and burn. Notice that in the XML file we called out chapters as 0, 15, 30, 45, 1 hour - obviously these can be further tweaked to be ‘real’ chapters, or left out all together. I’m sure there’s a GUI authoring app that helps you do this, if you figure it out post below and let me know. HTH!

HOWTO: recursively download only specific file types

Have you ever found a batch of mp3s online on someone’s ‘Index of’ page?  I know you have (and if not, do a search for ‘google hacks’ in google to learn about the fun)  The issue always comes up that I find an album I want to , but the individual files are in a directory, so you have something like band_name-album_name/01-songone., and so on.  To all of them I used to issue a wget command, with the -r (recursive) switch like this:

 -r http://.someurl.com/band_name*

but then I’d end up with a ton of other files from the root directory that would take time and confuse the so I’d have to search around for the payload.  I found a better way to do it, still using the -r for recursive search, but then only downloading the mp3s, forgoing any html pages or other directories in the root.  The command goes something like this:

 -A ,mpg,mpeg, -r -l 3 http://.someurl.com/band_name*

The curl command operates in a similar way. Its advantage is that it’s actively developed. Other similar commands that you can use are , fget, and fetch, but I don’t see a direct advantage over with the proper filters.

Chat on Skype via Pidgin on Linux (or Adium on Mac)

Skype on Linux - FTW!This is a big deal for me, I played with Skype back in the day, but never really used it much since it required a second client, and I have always used Gaim (which is now Pidgin) to consolidate all of my accounts into one client and didn’t want to break out of that mold, but now I don’t have to.  Using the API, Eion Robb has created a plugin called Skype API plugin for Pidgin/libpurple/Adium.  Now I just add my user to the accounts tab and I can now via in just like I with all my other contacts.  Note that you can’t do the video of on .  Mac users note that you can use this on (my fav OS X client), which uses , which is the backend for , on Mac.  So now I’m using again, which is a propreitary app, thanks to them providing an API for the community to latch on to.  Ah, the circle of life…

HOWTO: make old Firefox extensions install in 3.0 betas

If you’re like me you’re already running one of the 3 betas and loving the new features and stability lacking in the earlier series. The only issue now is that it’s taking extension writers time to update their extension to be compatible with the new Betas. Some of these extensions can almost hold me back to the 2.x series since they’re so useful, but 3 is just so much better in many ways. So, to install an old (2.x) extension into a new (3.x) just takes a little to the install file. By default the extensions are hardcoded to something like 2.0.12 or the like, and *generally* nothing else needs to change to make it work with the 3.x series. (please re-read the *generally* part…done? ok). To do this, just right click on the extension (I’ll use the Gspace one for this example) and choose ‘Save as…’ Once you have it, open up the commandline (or force WinZip to open it, unsure of how to do that) with unzip (the extensions end with .xpi, but are really a kind zipfile):

unzip gspace-0.5.92-fx+fl.xpi

After it unpacks things, you’ll have an install.rdf file in your current directory, it’s just an XML structure file so you can open it in vi:

vi install.rdf

Search for the following block for targetApplication and find the variable for maxVersion:

2.0.0.*

Then simply modify the maxVersion variable to something like:

3.*

Then save that file. Next rebuild the xpi file with all the same files it had before, but this time with the modified install.rdf (NOTE: I renamed the file from .xpi to -edited.xpi so I could tell it apart)

zip -r -D gspace-0.5.92-fx+fl-edited.xpi chrome defaults license.txt install.rdf gpl.txt chrome_jar.manifest chrome.manifest

And lastly open the newly created xpi with - it should now install no problem. If there’s an issue bail out by manually deleting the .jar (and any other chrome or default files you saw when you unpacked). Worse case you’ll need to build a new profile, but that’s hardly a price to pay for the thrill of the adventure! (blah, that sounds pretty geeky, eh?)

HOWTO: determine optimal fastcgi settings for Lighttpd

PHP Fast-cgiAnyone building a server with a LAMP stack today has tons of options, mine have evolved to using -> -> Xcache -> -> MySQL. Once I had Lighttpd (aka Lighty) installed and running pages I looked to optimize the configuration and push it as hard as possible for more . Of course lately I’ve been getting unexplained slowdowns, with many instances of -cgi appearing to be taking up almost all of my available CPU on `top`. Reading up on things it appears that I had , along with , set far too high for the load I’m getting. When you start it gives you the number of processes you’ve define, and then those in turn spawn the number of children you’ve specified. While my settings were too high, they were really overshooting things when you take into account that I’m using Xcache (which provides pre-caching) and (for HTTP acceleration). So even though one of my dynamic sites that I’m working on to ‘monetize’ things is getting 700-800 hits each day, my caching strategy is taking the load away from the ever available . Because of this, has much less to do, so giving it a ton of processes to just sit there and eat memory until they’re zombified is a waste. After reading the lighty FAQ and other posts specific to this, I’ve settled on the this for my fastcgi block within my .conf file. (more…)

HOWTO: sound after hibernate in Linux (Gusty/Lenny)

Ignignokt says - Using a key to gouge expletives on another’s vehicle is a sign of trust and friendshipWith all the tweaking to get my Dell Vostro 1500 working with Ubuntu, it’s still been an annoyance to get working evertime after hibernation. It goes to sleep fine, it wakes up fine, it obeys all of the power preferences I defined within fine too, it’s just that when it comes out of hibernation, the is usually off. It’s not muted, it’s off. Trying to restart alsa (the server) is a lession in frustration, so until now I’ve been ignoring it since it was rare that I would need it, but still…come on. This week I came across a solution in the Debian Forums that creates a new task for to do before it shuts down and before it starts up.: “Create the file /etc/pm/sleep.d/49sound…

mkdir /etc/pm
vi /etc/pm/sleep.d49sound

with the following contents:

function kill_sound_apps() {
pidsnd=$(lsof | grep /dev/snd | awk '{ print $2 }')
pidmixer=$(lsof | grep /dev/mixer | awk '{ print $2 }')
piddsp=$(lsof | grep /dev/dsp | awk '{ print $2 }')
kill $pidsnd $pidmixer $piddsp
}

case "$1" in
|suspend)
kill_sound_apps
modprobe -r snd_hda_intel
;;
thaw|resume)
modprobe snd_hda_intel
;;
*)
;;
esac

exit $?

Then just make it executable:

# chmod +x /etc/pm/sleep.d/49sound

So before shutting down, properly shuts down the , and when it comes back it, it properly starts the . As always, this *should* work, but the fact that it hasn’t been updated in Ubuntu Gusty is one of the reasons I’m shifting to on this ‘top.

NOTE: yes, I am thinking of making Ignignokt my official mascot. “Using a key to gouge expletives on another’s vehicle is a sign of trust and friendship

HOWTO: Speedup Firefox in 5 easy steps

Firefox logoThese are some basic tweaks to up that have been tried and true for some time now.  I haven’t seen these collected in one place recently, so if you have and want to improve its performance, try these steps.  If you have a broadband connection (who doesn’t?), you can up your page loads considerably using these steps.   Basically you’re allowing to load multiple things on a page instead of one at a time. By default, it’s optimized for dialup connections (lowest common denominator) so here’s what you need to do to fix that.

  1. Type “about:” into the address bar and hit return. Type “network.http” in the filter field, and change the following settings (double-click on them to change them):
  2. Set “network.http.” to “true”
  3. Set “network.http.proxy.” to “true”
  4. Set “network.http..” to 8 (recommended by devs)
  5. Right-click anywhere and select New-> Integer.  Name it “nglayout..delay” and set its value to “0″. This value is the amount of time the browser waits before it acts on information it receives.  With it set to zero the page just pops up, it’s a dramatic change.

So give those a go, and don’t stop there, there’s plenty of Firefox add-ons to and explore with; whatever you want to do online, there’s likely an add-on that’ll make it easier/better/faster.

HOWTO: use monit to keep Lighttpd and Varnish running

IgnignoktThanks to a post from Steve over at -administration.org, I finally got around to setting up monit, the little monitoring app we use at work to keep things sane.  I was getting around to installing it at home, but it became more urgent when went down last week; without it running there’s nothing to handle requests on :80, so as a it’s dead. So here’s my monitrc for the fronted by , acting in the /http accel role. Varn is listening on 80, then, if things aren’t cached, it forwards things on to listening on 82. also listens on the standard 443 for HTTPS requests, so we check that as well.

check process  with pidfile /var/run/varnishd.pid
start program = "/etc/init.d/ start"
stop program = "/etc/init.d/ stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 80 protocol http
then restart
if 3 restarts within 5 cycles then timeout

check process  with pidfile /var/run/.pid
start program = "/etc/init.d/ start"
stop program = "/etc/init.d/ stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 82 protocol http
then restart
if failed host 127.0.0.1 port 443 type tcpssl protocol http
with timeout 15 seconds
then restart
if 3 restarts within 5 cycles then timeout

So now we have monit watching , , Postifx, MySQL and OpenSSH - restarting things if they fail, and emailing me the status when they do.  Next on to some long term trending with Cacti providing some rrd graphing and then we’ll really have an idea of what this box is doing and be able to tune it accordingly.

HOWTO: log the user’s IP, not the proxy’s, in Lighttpd access log

Lighttpd - fly lightWhen you run a behind a or like Squid or Varnish, the access logs will display the of the proxy (generally 127.0.0.1) instead of the end user’s .  This not only breaks any kind of tracking or reporting you want to run against your logs, but it also takes away a datapoint I’ve had use for in general server admin tasks. This server runs in front of Lighttpd, and it reveals the end user’s in the header as , so it’s just a matter of making () use that variable in its access logs instead of the default variable defining the . Once we know that, the configuration is simple; in .conf, enter this:

accesslog.format = "%{}i %l %u %t \"%r\" %>s %b /
\"%{Referer}i\" \"%{User-Agent}i\""

For the definition of these variables, and plenty more, hit Lighty’s wiki. Props to the poster on the mailing list for bringing this up and reminding me to fix it! I’ve sent this link to the list so now it’s out there.


  • Recent Comment

    • MikeG: I do consulting and have no paid holidays etc. So I figure on 47 weeks a year. This allows for holidays I am...
    • sapien1980: good post! thanks :)
    • fak3r: @Shawn Certainly that would be more accurate, my orginal post was more of a use if you were trying to figure...
    • fak3r: –AZ-Sen: Jon Kyl –AZ-01: Rick Renzi –AZ-05: J.D. Hayworth –CA-04: John Doolittle...
    • assisted living: this is awesome…i will grow old and get drunk