Ruby on Rails: gem install versus apt-get
UPDATE: Thanks to Ryan, Ant and Fern for the tips. With that in mind I found an online Slicehost tutorial that contained the steps and explained how to install ruby via apt-get, then get the latest rubygems, install that manually, ran gem to update itself, then run gem to install rails – as suggested. The steps I took from that page:
(more…)
shutdown -h now
Just shut down the old server, chavez, that was running FreeBSD 6.1 – an awesome server. I’ve switched to Debian, and I’m really loving it, having used it in the past I really never dug into it as a server until recently. Administration is just easier, and it’s in line with me trying to cut back on things I have to do on the computer; gives me more time for other efforts. Anywho, it’s down now, for the record here’s the uname/uptime:
[23:45:42] [root@chavez /home]# uname -a
FreeBSD chavez 6.1-SECURITY FreeBSD 6.1-SECURITY #0: Wed Feb 14 15:33:28 UTC 2007 root@builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386
11:45PM up 237 days, 4:30, 1 user, load averages: 0.00, 0.00, 0.00
[23:45:45] [root@chavez /home]# shutdown -h now
Shutdown NOW!
shutdown: [pid 24665]
[23:55:00] [root@chavez /home]#
*** FINAL System shutdown message from root@chavez.cryer.us ***
System going down IMMEDIATELY
System shutdown time has arrived
Connection to chavez closed by remote host.
Defcon15
Oh yeah, I’m going to Defcon again this year, just found out this Friday for sure. It’s August 3rd – 5th, in Las Vegas, and this year it’s all paid for by my new consulting group; what a great thing. I argued that I would learn so much more there than any class, for a fraction of the cost; and I will. More on this later.
Apache server lockdown challenge
One of my favorite things about being a Linux admin is the ability to specify how things are going to be executed on the servers. I’ve been running the Apache web server for over 10 years now (1997), so setting up a new environment is no big deal, but I wanted to take it farther and cut as much out of a base install as possible, which still having it do what I need. I started with a Google search and a blank file for my httpd.conf, and went from there. Some background, since this is a work project I have a few restraints. First, we’re running on Red Hat Enterprise Server 4 with some pretty beefy hardware. Also, currently we ARE NOT building from source (something I usually do on my own Apache instances) since we’re still working out support options, which limits what we can do down to the almighty httpd.conf. I’ve trimmed down my conf at home, but since we have a smaller and more specific set of tasks for Apache here, I wanted to trim it down to the bone. So far I’ve gone through the Apache Security site, where I found their chapter on Installation and configuration especially helpful. I followed their suggestion of starting httpd.conf as a blank file. Later I ran my newly created conf through an Apache 2.0 Hardening Guide, and even combed through the Apache HTTP Server Module guide to be sure I wasn’t using anything extraneous. Now I’m being a bit idealistic with this config I know, but again, it’s for a specific purpose, and I don’t need to worry about many other factors that would cloud the waters as far as providing more options. I’ve taken out any specific modules that need to be loaded as part of my work so as not to confuse things, but I’ve left in our token variables (those that start with a T_) that get substituted just before install, so the question is, is there anything else I could cut back on? Also, is there anything missing that could lock things down further that don’t need to be installed separately? (ie- I’m not going to be installing mod_security…yet, but I’d like to). Read on to see my current ‘locked down’ config, all suggestions and (constructive?) criticisms appreciated.
HOWTO: populate your term’s title automatically
When you’re running a ton of termial windows or tabs, it helps to have the title of the box name, along with some environment values, easily available to keep you orientated. Here’s a quick script I created to do this automatically when called via your .profile file in your home directory.
#!/bin/bash
HOST_NAME=`hostname -f`
if [ `id -u` = 0 ]; then
OPT="`uname` (`uname -a | cut -f12 -d' ' -`) - ROOT USER"
else
OPT="`uname` (`uname -a | cut -f12 -d' ' -`)"
fi
REPLACE="${HOST_NAME} - ${OPT}"
echo -n -e "\033]0; $REPLACE \007 "
echo "${REPLACE}"
exit 0When I run this script in my term here at work, the title or tab becomes:
nldg-8 (Linux / x86_64)
Drop this into a bin directory your user can hit – I always put on in my home directory and append ~/bin to my PATH in my .profile. For Solaris fans/users, it needs to be done a bit differently:
#!/usr/local/bin/bash
HOST_NAME=`uname -a | cut -f2 -d' ' -`
OPT="(`uname -a | cut -f1 -d' ' -` / `uname -a | cut -f6 -d' '`)"
REPLACE="${HOST_NAME} - ${OPT}"
echo -n -e "\033]0; $REPLACE \007 "
echo "${REPLACE}"
exit 0foo
HOWTO: ssh tunneling for fun and profit
Recently I had an issue at work; while trying to transfer files between Unix hosts we were unable to hit the known scp port, but we could still hit the ssh port. All of this was occurring from home, late at night on a Saturday where I was the main technical point man to move/install these files. In the past I had done ssh tunneling, but never on the fly to fix something like this, so I cracked open my notes and did a quick Google search for a refresher.
The first we’ll look at the basic syntax of the command to setup the SSH tunnel:
ssh -L <local free port>:localhost:<local sshd port> -p <remote host sshd port> <remote host name>
Where:
- <local free port> is an unused high-number port on the local host
- <local sshd port> is the ssh port on the local host
- <remote host sshd port> is the remote host’s ssh port
- <remote host name> is the remote host you want to tunnel to
So, for example, if I wanted to copy files from work to my homeserver (but scp/sftp wasn’t running there) I could still scp the file via the ssh tunnel to home. Here’s how I’d do it:
ssh -L 5555:localhost:22 -p 2222 fak3r.com
Then I’d point to the tunnel while I issue a command I’d like to direct to it, and give it a username that is valid on the remote host:
scp –P 5555 fiile.txt bob@localhost:~
The file would then be in the home directory for bob’s account on fak3r.com. So anything directed at my local port of 5555 would be tunneled via ssh to the remote host’s sshd port of 2222 all via the tunnel I setup on my localhost, whose sshd is running on the default port of 22.
FreeNAS: network backup system
Yesterday NewsForge had an excellent article called, “A look at the FreeNAS server“. Basically FreeNAS is a small operating system based on FreeBSD 6 that provides NAS, or network-attached storage, (Wikipedia page) services like NFS (Network File System), CIFS (Microsoft’s Common Internet File System aka Samba) as well as tried and true Unix utilities like ftp, rsync, unison, ssh, scp, etc. The short explanation, this will take an old/unused PC and turn it into a true network accessible backup system that all of my home clients (Linux, Mac OS X and Windows) can talk to. My current backup strategy consists of my FreeBSD server running RAID1 to provide mirroring for redundancy over two drives that the clients rsync over ssh to. A standalone solution would be a better option as it would backup everything to the NAS, which I would also run in a RAID1 mirrored mode, giving the same amount of protection to all of the client data, but doubling up the backup of the server (server has it’s data already mirrored over two drives, plus the data copied to the NAS is mirrored over two drives for a total of four copies). There are plenty of cool features of FreeNAS including the fact that since it’s FreeBSD based it supports the same gmirror setup I’ve worked with before for RAID, the whole thing is bootable from a USB drive, compact flash (since the whole OS weighs in at only 32Meg!), or a regular harddrive, all of the administration can be done via the WebGUI, and the base distro is based off the one used for m0n0wall, a similar project that just handles firewall duties. So they’ve made a smart move using an existing framework, and then just building the backup control the web employs. The project is under active development, and looks like a winner for anyone needing a network backup system for home to small office. I have an old machine picked out at home, I need to find two drives for the storage and then I’ll build this out. Shouldn’t take long at all, and will give my USB drive something to do besides hang on my keychain! Their install docs seem pretty complete…stay tuned.
HOWTO: have vim create backup and tmp directories
This may only apply to those of us geeks that use vim to admin servers daily, but today I needed a way to backup, and automate the creation of backup and tmp directories to house those ever annoying ~ and .swp files from showing up in my working directory ($PWD). I didn’t want to lose them, just move them somewhere so they don’t clutter up the directory I’m working in. The solution was a function I found on the vim forums. Basically it uses directories it creates in your home directory, so you’ll have something like ~/.vim/backup and ~/.vim/tmp which is perfect; files are moved out of the way, but still backed up in a place you can rely on. I slightly reworked this, you can try it out by opening your ~/.vimrc file, and find the line:
set backup " keep a backup fileThen after that cut/paste the following (if you don’t have the set backup line, add it first):
function InitBackupDir()
let separator = "."
let parent = $HOME .'/' . separator . 'vim/'
let backup = parent . 'backup/'
let tmp = parent . 'tmp/'
if exists("*mkdir")
if !isdirectory(parent)
call mkdir(parent)
endif
if !isdirectory(backup)
call mkdir(backup)
endif
if !isdirectory(tmp)
call mkdir(tmp)
endifendif
let missing_dir = 0
if isdirectory(tmp)
execute 'set backupdir=' . escape(backup, " ") . "/,."
else
let missing_dir = 1
endif
if isdirectory(backup)
execute 'set directory=' . escape(tmp, " ") . "/,."
else
let missing_dir = 1
endif
if missing_dir
echo "Warning: Unable to create backup directories: " . backup ." and " . tmp
echo "Try: mkdir -p " . backupecho "and: mkdir -p " . tmp
set backupdir=. set directory=.
endifendfunction call InitBackupDir()
Back in black
Well, to make a long story short, we’re back. Installed a fresh version of FreeBSD 6.1 on my old(er) server that I built a few years back, and it’s humming along nicely. Going to try and make things simpler here, add a little more focus and less razzle dazzle – which seems to be something like feature creep here in the blog world. For now, sit back, I’ll have some new music coming up, geek talk and later maybe some beer. You can keep up by subscribing to the feed. I’ll update it to point to feedburner and put up the sub by email option again soon to get this show back on the road.
HOWTO: Passwordless ssh logins
Having the ability to do passwordless ssh logins helps in so many ways when automating tasks via scripts in Unix/Linux/BSD. Thankfully the great OpenSSH can take care of that for you, allowing you to do ssh, scp, sftp from and to multiple hosts. I always have to look up how to do it, so this time I found a good, consise way, and am setting it here for reference. For more detailed instructions check out the page this is taken from, otherwise fire up yr term and do it, “…for great justice!”
UPDATE2: Recently, while building a proof of concept computer cluster, I came across a much simpler way to do this. If you have ssh-keygen and ssh-copy-id installed, it’s a two step process.
First, create a password-less ssh rsa key:
ssh-keygen -b 2048 -f ~/.ssh/id_rsa -P ''
Second, copy the key to your remote host:
ssh-copy-id user@remote.host
And that’s it, easy cheesy. Might as well test it to make sure it worked:
ssh user@remote.host
It should drop you to a prompt on the remote box without asking for a password.
UPDATE: Apparently this has changed slightly, instead of writing to autorized_keys, you should use authorized_keys2 so any updates to the core OpenSSH won’t mess up your ‘local’ keyfile (verus the system one). Here are the correct (and more complete) directions:
ssh-keygen -t rsa (Enter)
You shouldn’t have a key stored there yet, but if you do it will prompt you now; make sure you overwrite it.
Enter passphrase (empty for no passphrase): (Enter)
Enter same passphrase again: (Enter)
We’re not using passphrases so logins can be automated, this should only be done for scripts or applications that need this functionality, it’s not for logging into servers lazily!
Now, replace REMOTE_SERVER with the hostname or IP that you’re going to call when you SSH to it, and copy the key over to the server:
cat ~/.ssh/id_rsa.pub | ssh REMOTE_SERVER 'cat - >> ~/.ssh/authorized_keys2'
Set the permissions to a sane level:
ssh REMOTE_SERVER 'chmod 700 .ssh'
Lastly, give it a go to see if it worked:
ssh REMOTE_SERVER
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub | ssh REMOTE_SERVER 'cat - >> ~/.ssh/authorized_keys'
ssh REMOTE_SERVER 'chmod 700 .ssh'
ssh REMOTE_SERVERHello (again) world
Are you like me, do you like to roll with the changes? Well I have, as you can see, with a move to Wordpress from the cool, but unstable Typo. My Typo experience was a blast, but honestly I just don’t have the time to babysit a webapp running on my server (which I’m already looking after) that seemed to prefer to bomb out than work more often than not. So, welcome to fak3r.com – the WordPress edition. Migration of articles was a snap, I just had to set the number of articles the RSS feed would generate in Typo to 750 (funny thing was when I tried to do this in the Admin section it bombed with ‘Application Error (Rails)’ so I had to go into the DB by hand and tune it), save the feed.xml file and then import it via WP’s RSS importer. All of the articles, along with the categories, came right over. The only thing is the comments, importing the comments RSS feed from Typo just puts them all in as other articles, not good. So, for now, no comments were transfered, but I’ll keep working on it. I could tell you more but hey, let’s just move on; I’m happy enough to be able to post again!
Speed up Ruby-on-Rails with memcached
Today I learned about memcached, which I’d heard of before, but never really investigated. From the project’s site, ”memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.” So, even though I don’t have a huge amount of traffic, I still have dynamic sites, and I’m always looking at ways to speed up my Typo blog (this site not anymore). So, using memcached, you can get a big performance boost in databases calls, which sold me on giving it a go. I read two posts today, one about howto set this up in Freebsd, along with howto modify the source code for a boost over the default setting, and how to make Ruby-on-Rails take advantage of it. Below are steps compiled from both sites, and used on my FreeBSD 6.0 server, but most of the steps should work as well in Linux. Read more for the steps.
First let’s get memcached installed
cd /usr/ports/databases/memcached/We only want it to get past the configure step before we modify code
make configureNow it’s time to modify the code (NOTE: the howto linked to above was specific to a FreeBSD issue, if using Linux you may not need to make this modification)
vi work/memcached-1.1.12/memcached.cFind this line
#include "memcached.h"Add the undef line below it and save
#include "memcached.h"
#undef TCP_NOPUSHNow we want it to compile and install
make installOnce that’s complete, we want to enable memcached in rc.conf
echo "memcached_enable="YES"" >> /etc/rc.confThen we’ll start memcached
/usr/local/etc/rc.d/memcached.sh startNext we’ll install the ruby-memcache client
cd ../ruby-memcache/
make installFinally we’ll modify our Ruby-on-Rails app’s environment to use memcache as its session store (make a backup first!)
cp config/environment.rb config/environment.rb.dist
vi config/environment.rbFind the line that tells session_store to use the database instead of the file system
#config.action_controller.session_store = :active_record_storeModify it so it tells it to use memcached, and save
config.action_controller.session_store = :mem_cache_storeStop Typo, and then manually clear the cache
rake sweep_cacheNow restart your Typo server, and you’re done! It should now be storing all session data via memcached instead of your database.
HOWTO: usable xterm configuration
- UPDATE: if you’re using xterm in place of gnome-terminal due to speed, you aren’t any longer. The 2.14 version of Gnome sports a much faster gnome-terminal; it beats xterm for display by allot, and log startup is 20x faster than before! Wow, they did some work on tuning there!
Ok, this is a cheap HOWTO seeing as how I just found it, cut/pasted it and restarted X – but hey, it rocks. I’ve always used Gnome-terminal when in Gnome, but I also tend to when I’m in Openbox or Xfce I use it since I can control the fonts to make it usable, unlike xterm. Last night after installing Xfce4 (recommended) on my iBook I hit their FAQ to do some tweaking where I came across this, and now I can use xterm! It’s right in time to since Gnome-terminal loads very slowly when you’re not in Gnome, and that’s especially annoying when you’re playing in a ‘lighter’ window manager/desktop environment. Xterm, on the other hand, just pops up like it was waiting for you. So, to try out this config yourself, hit this page on their documentaion page, or read more on this post for the quick HOWTO and code.
Edit or create an .Xsessions file in your user’s home directory:
vi ~/.XsessionsCopy code from the URL above, or the code below and paste it in there. Now save:
:wq!Have xrdb source the .Xresources file:
xrdb -merge .Xresourcesand finally, launch xterm:
xtermThat’s it! Now xterm will look like that anytime you launch it, and this even works in Windows when using Cgywin, same steps as above. Sure as heck beats Cgywin’s shell…don’t get me wrong, I appreciate Cgywin, but installing X just to get a useable term is a necessary when you’re forced to use Windows.
Here is the code to copy:
! this are Xresources to make xterm look good
! put into ~/.Xresources
! after changing contents, run xrdb -merge .Xresources
! gentoo has a bug so that it doesnt read it when X starts, so add above
! command to /etc/xfce4/xinitrc (top) and be happy.
!xterm*background: Black
!xterm*foreground: Grey
xterm*font: -Misc-Fixed-Medium-R-Normal--20-200-75-75-C-100-ISO10646-1
!xterm*font: -misc-fixed-medium-r-normal--18-*-*-*-*-*-iso10646-1
!xterm*iconPixmap: ...
xterm*iconPixmap: /usr/share/pixmaps/gnome-gemvt.xbm
xterm*iconMask: /usr/share/pixmaps/gnome-gemvt-mask.xbm
!XTerm*iconName: terminal
!Mwm*xterm*iconImage: /home/a/a1111aa/xterm.icon
XTerm*loginShell: true
XTerm*foreground: gray90
XTerm*background: black
XTerm*cursorColor: rgb:00/80/00
XTerm*borderColor: white
XTerm*scrollColor: black
XTerm*visualBell: true
XTerm*saveLines: 1000
!! XTerm.VT100.allowSendEvents: True
XTerm*allowSendEvents: True
XTerm*sessionMgt: false
!XTerm*eightBitInput: false
!XTerm*metaSendsEscape: true
!XTerm*internalBorder: 10
!XTerm*highlightSelection: true
!XTerm*VT100*colorBDMode: on
!XTerm*VT100*colorBD: blue
!XTerm.VT100.eightBitOutput: true
!XTerm.VT100.titeInhibit: false
XTerm*color0: black
XTerm*color1: red3
XTerm*color2: green3
XTerm*color3: yellow3
XTerm*color4: DodgerBlue1
XTerm*color5: magenta3
XTerm*color6: cyan3
XTerm*color7: gray90
XTerm*color8: gray50
XTerm*color9: red
XTerm*color10: green
XTerm*color11: yellow
XTerm*color12: blue
XTerm*color13: magenta
XTerm*color14: cyan
XTerm*color15: white
XTerm*colorUL: yellow
XTerm*colorBD: white
!XTerm*mainMenu*backgroundPixmap: gradient:vertical?dimension=400&start=gray10&end=gray40
!XTerm*mainMenu*foreground: white
!XTerm*vtMenu*backgroundPixmap: gradient:vertical?dimension=550&start=gray10&end=gray40
!XTerm*vtMenu*foreground: white
!XTerm*fontMenu*backgroundPixmap: gradient:vertical?dimension=300&start=gray10&end=gray40
!XTerm*fontMenu*foreground: white
!XTerm*tekMenu*backgroundPixmap: gradient:vertical?dimension=300&start=gray10&end=gray40
!XTerm*tekMenu*foreground: white
!XTerm Profiles (idea from dag wieers)
XTerm*rightScrollBar: trueHOWTO: Install Roundcube Webmail from SVN (was CVS) on FreeBSD
UPDATE3: The FreeBSD port is keeping up with this project very well, the current one is only 5 days old! I highly recommend going this route unless you’re a developer or someone who likes to get the bugs before anyone else! 
Thanks to Bernard for bringing this up.
UPDATE2: Roundcube now uses SVN (Subversion) for version control instead of CVS, I’ve updated all effected instructions.
UPDATE: Looking at the download page for Roundcube, I see that There’s also a Spanish version of Phil’s guide written by Daniel A. Rodriguez. ¡se ha traducido eso es lejano hacia fuera, yo! You can find the link here , I hope that helps. I would like to see more translations of HOWTOs in general, I wish there was a good Firefox plugin to translate pages (well) on the fly.
In all my years of running a mailserver at home, I’ve used quite a few different webmail apps to find the balance of functionality and style that I’ve been craving. This includes Horde/IMP, Squirrelmail, IlohaMail, OpenExchange, Hula (as well as a couple of others) but all were either lacking functionality, style, or readiness (Hula I’m looking at you). A few months ago I stubled across Roundcubemail, a webmail app which aims to provide an application-like user interface, which it does. When you can drag and drop mail to a folder, you know you’re dealing with something pretty new, and AJAX related. The smoothness and speed of refreshes when you switch folders (along with the ‘spinner’ at the top of the screen to give you needed feedback) is just a breath of fresh air.
Try out the demo to see what all the fuss is about if you’re looking for a LAMP based IMAP webmail solution. Time will tell, but this project’s out of the gate showing makes me think it’s going to be a great app. Let’s hope they keep the configuration and options to a sane amount, unlike other webmail apps (Horde/IMP, I’m looking at you), so they can keep their clean, non-bloated feel. Since I’ve installed it I’ve moved on to nightly rebuilds from CVS, so I thought I’d document the steps needed to run the latest CVS version of Roundcubemail on FreeBSD. While there is a FreeBSD port in the ports tree (mail/roundcube) it is far out of date, so the CVS version is recommended since there are so many changes happening day-to-day with it. Once running you should also join the dev@lists.roundcube.net mailing list to keep abreast of, and report any, bugs that need ironing out. This HOWTO assumes you have a webserver (www/apache2), a mail transport agent (mail/postfix), and an IMAP server (mail/dovecot) installed and working. I brought them up and tested them with Squirrelmail (mail/squirrelmail), since it’s almost as easy to setup and always “just works”. SO, without further babble, here’s my first (of many) HOWTOs on fak3r.com.
HOWTO: Install Roundcubemail from CVS SVN on FreeBSD
- NOTICE: This document assumes you already have a webserver running with PHP support, a mySQL database a Mail transport agent, an IMAP server and the Subversion client to check out the code.. My configuration consists of Apache2 (www/apache2), PHP 4 (lang/php-4), mySQL 4.x (database/mysql40-server), Postfix (mail/postfix) and Dovecot (mail/dovecot). Additionally this install was done on FreeBSD 6.0, but I see nothing specific that would stop the same procedure to allow Roundcubemail on 5.x, 4.x or even any Linux distribution. Feedback on this is welcome.
1) As root, change to the webroot of Apache
cd /usr/local/www/data-dist2) Login to the Sourceforge Roundcubemail CVS server (when prompted for a password, just press ENTER)
cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/roundcubemail login
2) Checkout Roundcubemail from the Sourceforge SVN server (when prompted, choose ‘p’ to accept the encrypted key permanantly)
svn checkout https://svn.roundcube.net/trunk3) Change into the roundcubemail directory
cd roundcubemail
3) Move the roundcubemail directory to your webroot, remove the ‘trunk’ directory, and then change into the roundcubemail directory
mv trunk/roundcubemail .
rm -rf trunk
cd roundcubemail4) Set permissions of the temp and logs dir so that the web user can read/write to them
chown -R www:www temp logs5) Create a database for storage of Roundcubemail data, replace $PASSWORD with the password you want the roundcube user to use to access mySQL
# mysql
> create database 'roundcubemail';
> GRANT ALL PRIVILEGES ON roundcubemail.* TO roundcube@localhost
IDENTIFIED BY '$PASSWORD';
> quit6) Import the inital Roundcubemail SQL
# mysql roundcubemail < SQL/mysql.initial.sql7) Change into the config directory
cd config 
Copy the config *php.dist files to *.php
cp db.inc.php.dist db.inc.php
cp main.inc.php.dist main.inc.php9) Modify the config files to suit your environment. In db.inc.php you only need to change the database definition line, add your password in place of $PASSWORD
$rcmail_config['db_dsnw'] = 'mysql://roundcube:PASSWORD@localhost/roundcubemail';Assuming your mailserver is running on the same physical box as the webserver, disable database caching
$rcmail_config['enable_caching'] = FALSE;define the host as localhost
$rcmail_config['default_host'] = 'localhost';define smtp as localhost
$rcmail_config['smtp_server'] = 'localhost';and increase the session lifetime from 5 to something more reasonable (optional)
$rcmail_config['session_lifetime'] = 30;Launch a web browser and point it to
http://some.url/roundcubemailThen login with a valid/existing IMAP username and password.
To debug problems just tail -f (or multitail if you’re cool like me) /var/log/maillog to see what’s happening behind the scenes. Consult the mailing lists for issues and feel free to give feedback below. Since this app is under heavy development I expect this HOWTO to change as the app does.
Hula progress on FreeBSD
I’ve been pretty quiet about Hula since I’ve been unable to successfully build *and* run it since r370 (currently Hula is at r609). While I’ve solved and committed all the autogen build issues on FreeBSD, it still won’t run; the controlling hulamanager process just hangs, with no errors or output to help out. On the mailing list this behavior is reportedly due to the (hardlinked) renaming of ’server messaging server’ to ‘hula messaging’ server, which bombs if you use the filesystem based mdb. Alex sent me this patch:
diff -urNad --exclude=CVS --exclude=.svn ./src/libs/mdb-file/mdbfile.c
/tmp/dpep-work.Qpsn4d/hula-0.1.0+svn472/src/libs/mdb-file/mdbfile.c
--- ./src/libs/mdb-file/mdbfile.c 2005-09-16 12:19:45.000000000 +0100
+++ /tmp/dpep-work.Qpsn4d/hula-0.1.0+svn472/src/libs/mdb-file/mdbfile.c 2005-09-20 20:45:41.000000000 +0100
@@ -3207,8 +3207,8 @@
MDBFile.unload = FALSE;
strcpy(MDBFile.localTree, "\Tree");
- strcpy(MDBFile.serverDN, "\Tree\Context\Hula");
- strcpy(MDBFile.replicaDN, "Hula");
+ strcpy(MDBFile.serverDN, "\Tree\Context\Server");
+ strcpy(MDBFile.replicaDN, "Server");
strcpy(MDBFile.base64Chars, "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=");
#if defined(LINUX)and it applies cleanly, but yet the same issue occurs. I have a bug written on this, and will try to get back on IRC tomorrow and work on it. Also, due to a request on the hula-dev list, I have added some logic to my gethula script; version 1.7 should now support NetBSD, but I’m waiting for feedback on that. Again, it *should* compile, but I don’t expect it to run…yet.
95,899 hits in one day
I’m still posting on my Slashdot thread about Friday’s slashdotting of fak3r.com as well as learning what worked, and where the bottleneck occurred. First of all the all important numbers; visits, pages, hits and transferred data for 2005-10-15, as reported by Awstats:
Date Pages Hits Bandwidth
10-14-2005 18092 95899 644.47 MBHoly smokes, 95,899 hits for the day while transferring almost a cd’s worth of data. Again, not a huge number for a colo’d webserver with a big audience, but for a home built rig behind a 384/1.5 ADSL line, pretty cool. Other interesting data gathered was:
Operating Systems (Top 10)
Operating Systems Hits Percent
Windows 80112 63.6 %
Linux 25319 20.1 %
Macintosh 14305 11.3 %
Unknown 5034 4 %
FreeBSD 741 0.5 %
Sun Solaris 278 0.2 %
OpenBSD 37 0 %
NetBSD 20 0 %
WebTV 1 0 %This was expected since it’s a tech site, but it was still nice to see Linux so well represented (when I hit the site from work I’m coming in via XP unfortunately)
Browsers (Top 10)
Browsers Grabber Hits Percent
Firefox No 84168 66.8 %
MS Internet Explorer No 17268 13.7 %
Safari No 9715 7.7 %
Mozilla No 4509 3.5 %
Opera No 4011 3.1 %
Unknown ? 2867 2.2 %
Konqueror No 1278 1 %
Camino No 555 0.4 %
Galeon No 405 0.3 %
Netscape No 307 0.2 %
Others 764 0.6 %Again, same disclaimer, but it would be nice if Firefox were the rule, and not the exception for the general public.
So what did I learn? I learned that my FreeBSD 6.0 box is setup well enough to handle *at least* 100,000 hits a day. I learned that using Apache2 -> mod_proxy -> lighttpd -> fastcgi powered by Typo for blogging is a good enough combination to easily handle the traffic thrown it’s way. Still, while my server was only running the Ruby process around 35%:
51469 fak3r 1 99 0 38632K 32832K RUN 14:50 34.47% rubystatic pages were still taking ~60 seconds to refresh during peak load. Thankfully my ssh tunnel held up, so I was watching top and tailing the logs in real time. From this I could see that everything was setup to handle the traffic, with headroom to spare, expect for my ADSL line, which still performed as expected. It was indeed the bottleneck, but the fact that pages were still being severed (albeit slowly) showed it handled the traffic and served pages to all that would wait for them. In the future I will likely use my OpenBSD firewall running pf to limit the traffic to the websever to still allow internal clients the bandwidth to surf, but with the same configuration this would only decrease our numbers. So, better test would have this box on a bigger pipe (Speakeasy has a 1.5/6.0 line, as well as T1 options) which probably won’t happen in the confines of my home network, but would likely really push the server to its limits. Perhaps one day, in a colo’d location, my new FreeBSD powered 4U server on a T1 will notice a spike in traffic; seconds after my recent post to Slashdot…
A minor Slashdoting!
This morning on Slashdot there was a story about Ruby on Rails and my comment turned out to the the second post. I took the opportunity to plug this site…err…I mean used this site as an example of Ruby on Rails via Typo and suggested people take a look and try out the ‘live search’ to give the database a workout, and did they ever. Logfiles were just scrolling along, httpd was throwing up pages, Ruby was driving all database queries via fastcgi. Top showed Ruby pushing upwards of 18%, so I killed Hulaweb, which was eating more, and watched Ruby take over, running at 34% at one point:
51469 fak3r 1 99 0 38632K 32832K RUN 14:50 34.47% rubyDuring one of the peak load times a full page reload from my client took just over 60 seconds, which is a very long time, but the DSL was the bottleneck, not the server or any of it’s processes. Tailing the logs I watched it continuously spitting out pages to other clients in the queue before me, so Lighttpd was doing its job as it should, and it had plenty of RAM/proc overhead (even though I saw it peak around 34%! at times), so the delay was simply my home DSL (1.5/384 down/up); which performed adminrable condisering the abuse. Thanks Speakeasy! Once things calmed down a bit three hours later (ruby was still using ~18%) I could hit pages and have them reload as if nothing was happening, database searches as well.
So, for some very *rough* numbers; my ’second post’ to Slashdot occured at 9:15AM, and at roughly 12:15AM /var/log/http-access.log showed 50,000 mod_proxy requests. So, requests that Apache handled via mod_proxy to Lighttpd:
50,000 / 03 = 16666.6666 pageviews/hour16,666 / 60 = 277.7777- pageviews/minute277 / 60 = 4.629- pageviews/secondSo my home server that I built by hand, running behind a standard ADSL line, served up an average of ~5 pages per second for over three hours. Fuckin’ A! Later, at 6 hours (3:15PM):
mod_proxy calls (page requests)
grep 14/Oct /var/log/httpd-access.log | wc -l
78741DB Calls
awk '/^"action"=>"search"/' /usr/local/typo/log/production.log
grep Oct 14 production.log | wc -l
13478Successful DB Results
awk '/^Completed/' /usr/local/typo/log/production.log
| grep Oct 14 production.log | wc -l
13478I’ll do more numbers once I learn if the way I pulled them were accurate, but as it stands the box handled almost 80,000 page requests today, and the logs are still rolling. Stay tuned.
Windows shows its BSD heritage
It’s well known that MS utilized BSD code (which is allowed under the BSD License) in various places in Windows, but it’s still fun to see it in there. Unhappily I’m using XP at my current consulting gig, but I’ll fix that soon. If you are as well, drop to the cmd.exe window and do the following:
c:> strings.exe c:\\WINDOWS\\system32\\ftp.exe | grep CopyrightYou’ll get back the following:
@(#) Copyright (c) 1983 The Regents of the University of California.Incidentally, on my FreeBSD server at home it shows a bit more up to date code:
[pepe:/usr/bin]$ strings /usr/bin/ftp | grep Copyright;
strings /usr/bin/ftp | grep California
@(#) Copyright (c) 1985, 1989, 1993, 1994
The Regents of the University of California. All rights reserved.And yes, you’d expect it to be a bit out of date; there’s no reason to use ftp nowadays, as it’s completely insecure. OpenSSH provides scp and sftp for secure transfers, and you can tunnel almost anything else through it (I do rsync over ssh for backups) so there’s no reason not to use it.
Hi, I'm fak3r and I've been online for over 15 years, blogging for 10. I work with open source software, various hardware and whatever else that I can hack on. I enjoy learning by doing, painting infrequently and listening to more music than the law allows. Feel free to follow me and comment on my thoughts and ideas; we're all in it together kid.
