fak3r

I’m a big Bash fan, I know Perl is the more popular scripting language, and I’m slowly using it more, but hey, if I need something done, I can do it quicker in Bash (keeping in mind that I’m a systems guy, not a dev guy). While at work looking up Bash related syntax I came across a page describing how to run a webserver with 100 lines of Bash. It uses the old school GNU utility Netcat (nc) for communication between the pipes, and just a ton of basic logic and functions to pass it on to the user. It’s one of those things I look at and can’t believe it works, but it does. Of course security is unknown, as is the original author, but I consider this a reference on how to do networking things in Bash; who knows what I’ll use (parts) of it for. If anyone has details on who originally wrote this I’m all ears.

#!/bin/bash

function debug {
    local severity="$1"
    shift
    local message="$@"

    echo -n "`date -u`"    1>&2
    echo -ne '\t'        1>&2
    echo -n "$severity"    1>&2
    echo -ne '\t'        1>&2
    echo "$message"        1>&2
}

function fix_path {
    echo -n "$1" | head -n 1 | sed 's|^[/.-]*||' | sed 's|/\.*|/|g'
}

function serve_dir {
    local dir="`fix_path "$1"`"
    if [ "$dir" = "" ]; then
        dir="./"
    fi
    echo 'HTTP/1.1 200 OK'
    echo 'Content-type: text/html;charset=UTF-8'
    echo
    echo LISTING "$dir"
    echo '<br />'
    ls -p "$dir" | sed -e 's|^\(.*\)$|<a href="/'"$dir"'\1">\1</a><br />|'
}

function serve_file {
    echo 'HTTP/1.1 200 OK'
    echo 'Content-type: application/x-download-this'
    echo
    local file="`fix_path "$1"`"
    debug INFO serving file "$file"
    cat "$file"
}

function process {
    local url="`gawk '{print $2}' | head -n 1`"
    case "$url" in
        */)
            debug INFO Processing "$url" as dir
            serve_dir "$url"
            break
            ;;
        *)
            debug INFO Processing "$url" as file
            serve_file "$url"
            ;;
    esac
}

function serve {
    local port="$1"
    local sin="$2"
    local sout="$3"

    while debug INFO Running nc; do

        nc -l -p "$port" < "$sin" > "$sout" &amp;
        pid="$!"

        debug INFO Server PID: "$pid"

        trap cleanup SIGINT
        head -n 1 "$sout" | process > "$sin"
        trap - SIGINT

        debug INFO Killing nc

        kill "$pid"
    done

    debug INFO Quiting server
}

function cleanup {
    debug INFO Caught signal, quitting...
    rm -Rf "$tmp_dir"
    exit
}

tmp_dir="`mktemp -d -t http_server.XXXXXXXXXX`"
sin="$tmp_dir"/in
sout="$tmp_dir"/out
pid=0
port="$1"

mkfifo "$sin"
mkfifo "$sout"

debug INFO Starting server on port "$port"
serve "$port" "$sin" "$sout"
cleanup

I changed around my Conky config, and it’s something you could do forever, but it’s great because it can be as heavy or light as you want it. Recently I dropped Gnome almost all together to run Openbox (full HOWTO on this forthcoming). I found a panel that will house things like nm-applet output, but was missing things like a simple clock, network activity, etc. So now, using most of the same code/look that I used here, I have a small, transparent strip at the bottom of the screen showing me time, date, proc, proc temp, network up, network down, and power status (battery, AC and the level of charge). It looks good, it’s light, it’s all I need. Nice to bring some of the memory requirements down from Gnome as well.

# Create own window instead of using desktop (required in nautilus)
own_window true
own_window_hints undecorated,below,skip_taskbar
background no
# Use double buffering (reduces flicker, may not work for everyone)
double_buffer true
# fiddle with window
use_spacer right
use_xft true
# Update interval in seconds
update_interval 3.0
# Minimum size of text area
minimum_size 10000 5
# Draw shades?
draw_shades yes
# Text stuff
draw_outline no # amplifies text if yes
draw_borders no
uppercase no # set to yes if you want all text to be in uppercase
# Stippled borders?
stippled_borders 8

# border margins
border_margin 1
# border width
border_width 1
# Default colors and also border colors, grey90 == #e5e5e5
default_color white
default_shade_color black
default_outline_color white
own_window_colour brown
own_window_transparent yes
# Text alignment, other possible values are commented
#alignment top_left
#alignment top_right
alignment bottom_left
#alignment bottom_right
# Gap between borders of screen and text
gap_x 10
gap_y 5
# stuff after 'TEXT' will be formatted on screen
override_utf8_locale no
#xftfont Terminus:size=8
xftfont Terminus:size=10
xftalpha 0.8
#Mail:${color}${execi 300 python ~/scripts/gmail.py}
TEXT

${offset 0}${color }${time %H:%M} ${color slate grey}${time %Z    }Date: ${color }${time %a, } ${time %e %B %G} ${offset 0} ${offset 0}   ${color slate grey}Proc:${color} $cpu%${offset 5}${acpitemp}C${offset 5}${cpugraph 16,100 000000 ffffff} ${offset 0}   ${color slate grey}Net:${offset 5}${color}Up:${upspeed wlan0}k/s${offset 5}${upspeedgraph wlan0 16,100 000000 ffffff}${offset 0}   ${color}Dn:${downspeed wlan0}k/s${color}${offset 5}${downspeedgraph wlan0 16,100 000000 ffffff}   ${color slate grey}    Power:${offset 5}${color}${battery}

Try it, you might like it - I’ll keep working on it, I’m sure I’ll find more things to add/improve. Conky rocks.

Have you ever found a batch of mp3s online on someone’s ‘Index of’ page?  I know you have (and if not, do a search for ‘google hacks’ in google to learn about the fun)  The issue always comes up that I find an album I want to grab, but the individual files are in a directory, so you have something like band_name-album_name/01-songone.mp3, and so on.  To grab all of them I used to issue a wget command, with the -r (recursive) switch like this:

wget -r http://www.someurl.com/band_name*

but then I’d end up with a ton of other files from the root directory that would take time and confuse the download so I’d have to search around for the mp3 payload.  I found a better way to do it, still using the -r for recursive search, but then only downloading the mp3s, forgoing any html pages or other directories in the root.  The command goes something like this:

wget -A mp3,mpg,mpeg,avi -r -l 3 http://www.someurl.com/band_name*

The curl command operates in a similar way. Its advantage is that it’s actively developed. Other similar commands that you can use are snarf, fget, and fetch, but I don’t see a direct advantage over wget with the proper filters.

If you run a Linux desktop you need to be using conky.  It compiles all those shiny gadget you see on other desktops eating system RAM, down to what you need; information on what your system is doing.  So try it out, install conky, and then drop this into your home directory as .conkyrc - then run conky.  The file is pretty self explanatory, enjoy!

(more…)

Thanks to a post from Steve over at debian-administration.org, I finally got around to setting up monit, the little monitoring app we use at work to keep things sane.  I was getting around to installing it at home, but it became more urgent when Varnish went down last week; without it running there’s nothing to handle requests on :80, so as a webserver it’s dead. So here’s my monitrc for the webserver Lighttpd fronted by Varnish, acting in the reverse proxy/http accel role. Varn is listening on 80, then, if things aren’t cached, it forwards things on to Lighttpd listening on 82. Lighty also listens on the standard 443 for HTTPS requests, so we check that as well.

check process varnish with pidfile /var/run/varnishd.pid
start program = "/etc/init.d/varnish start"
stop program = "/etc/init.d/varnish stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 80 protocol http
then restart
if 3 restarts within 5 cycles then timeout

check process lighttpd with pidfile /var/run/lighttpd.pid
start program = "/etc/init.d/lighttpd start"
stop program = "/etc/init.d/lighttpd stop"
if cpu > 60% for 2 cycles then alert
if cpu > 80% for 5 cycles then restart
if totalmem > 200.0 MB for 5 cycles then restart
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then stop
if failed host 127.0.0.1 port 82 protocol http
then restart
if failed host 127.0.0.1 port 443 type tcpssl protocol http
with timeout 15 seconds
then restart
if 3 restarts within 5 cycles then timeout

So now we have monit watching Lighttpd, Varnish, Postifx, MySQL and OpenSSH - restarting things if they fail, and emailing me the status when they do.  Next on to some long term trending with Cacti providing some rrd graphing and then we’ll really have an idea of what this box is doing and be able to tune it accordingly.

When you run a webserver behind a reverse proxy or HTTP accelerator like Squid or Varnish, the webserver access logs will display the IP of the proxy (generally 127.0.0.1) instead of the end user’s IP.  This not only breaks any kind of tracking or reporting you want to run against your webserver logs, but it also takes away a datapoint I’ve had use for in general server admin tasks. This server runs Varnish in front of Lighttpd, and it reveals the end user’s IP in the header as X-Forwarded-For, so it’s just a matter of making Lighttpd (lighty) use that variable in its access logs instead of the default variable defining the referring IP. Once we know that, the configuration is simple; in lighttpd.conf, enter this:

accesslog.format = "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b /
\"%{Referer}i\" \"%{User-Agent}i\""

For the definition of these variables, and plenty more, hit Lighty’s wiki. Props to the poster on the Varnish mailing list for bringing this up and reminding me to fix it! I’ve sent this link to the list so now it’s out there.

It’s time for a new laptop, as I’ve detailed, I’ve ripped apart, inserted coins and duct-taped  the old iBook back together again enough times, and it’s no longer viable. It’ll work fine on a flat surface, but if you try to use it as a laptop the minor flexing must loosen the video chip, because you quickly find your video locked, with a hard reboot the only fix. The wildcards are me as a buyer, since I’m hardly ordinary with my expectation that any laptop or desktop I’m going to buy is only going to run Linux, and the recent announcements by HP, IBM/Lenovo and Dell about their Linux support (some even pre-installed), I knew I’d finally have choices to consider. In the end I came up with a pretty current system, that Debian or Ubuntu will be 100% compatible with, and will be proud to call home. The detailed specs:

Intel Core 2 Duo T5470, 1.6GHz, 800Mhz FSB, 2M L2 Cache
15.4 inch Wide Screen XGA LCD display
1GB, DDR2, 667MHz 2 DIMM
128MB NVIDIA GeForce 8400M GS
120G 5400RPM SATA Hard Drive
Integrated 10/100 Network Cardand Modem
8X DVD+/-RW with double-layer DVD+R write capability
Integrated High Definition Audio 2.0
Intel 3945 WLAN (802.11a/g) Mini Card
Integrated 2.0 mega pixel webcam
Integrated Bluetooth
85 WHr 9-cell Lithium Ion Primary Battery

This is more system that I originally spec’d out, but the price was right, so I’m very happy.  Before I reveal which brand I picked, I’ll tell the interesting story of how I ended up with the ‘top I did, and how things compare for laptop Linux options these days, it’s an interesting ride.

NOTE: feel free to Digg this article if you like it.

(more…)

Looking over the Debian own harden-doc guide online, (which is a monster of a resource) as well as Debian Help’s security page gave me some excellent new ideas on how to secure Debian and Linux in general.   Also today i found a netstat command with some nice switches to help you figure out what is listening on each port in an easy to read layout, -plunt:

netstat -plunt

Plus it’s fun to say, ‘plunt’. Lastly there’s a good overview of deborphan (which assists you in keeping your system clear of unneeded packages) with coverage on how to use it at Debian Adminstrator.org. But in the comments a thread talks about how it’s better to use aptitude, as this does it automatically.

UPDATE: thanks to a comment below from Ted, we now have a solution to have this option persist across kernel updates.  In grub, “…at the end of this new menu item add it as an argument to the line:

defoptions=quiet splash irqpoll

I knew there had to be a way, thanks for the post Ted!

There’s a known bug in Ubuntu 7.04 (Feisty) with some ata detection routine that causes the system to take over 2 minutes to boot. Since this has happened to me more than once I’m documenting it here for me, and for other desperate souls that may find their way here. If your system is very slow to boot, and you see error messages in your dmesg (`dmesg | grep ata`) such as this:

[ 34.122465] ata1.00: qc timeout (cmd 0xef)
[ 34.122519] ata1.00: failed to set xfermode (err_mask=0x4)
[ 34.122565] ata1: failed to recover some devices, retrying in 5 secs
[ 46.260055] ata1: port is slow to respond, please be patient (Status 0x90)
[ 69.218482] ata1: port failed to respond (30 secs, Status 0x90)

You just need to ad `irqpoll` to your grub line. So in so in /boot/grub/menu.lst I added irqpoll to the kernel line:

kernel /boot/vmlinuz-2.6.20-15-generic root=UUID=48c5a348-eb39-4171-8531-671a49fdb75b ro quiet splash irqpoll

and it fixes the issue. Probably a work around, but since this resets every time you install a new kernel you’ll realize when it’s broken and when it’s fixed. Oh, and my system boots in 21 seconds now…is it geeky that I know that, and I tweaked the system to make it boot faster than the 27 seconds it was booting in? I guess we’ll never know!

So the only thing I don’t love about my new job is the same old thing; you have to run Windows XP on the desktop. Yeah, I’ll give it a bit more time before I really start pushing to run Linux on the desktop, so until then it’s my ongoing struggle to get XP to work the way I want it to (ie- more like Linux). One simple way is to install the excellent Jimmac mouse cursor theme that’s the default for the majority of Linux distributions. Jakub Steiner (aka Jimmac) is the famous designer of this set, and with a 3rd party app called CursorXP , it’s a snap to get them into XP. First grab the Jimmac theme created to work with CursorXP, then Download and install CursorXP and get into its config menu, which is a new tab under Settings > Mouse. From the drop down list choose <Broswe>, point it to the theme and you’re done. You wouldn’t think a change of mouse cursors would change the feel of a system so much, but these do; I feel more at home. Even if you’ve never used Linux you’ll still love this cursor set, try it, it’s all free.