I’m still posting on my Slashdot thread about Friday’s slashdotting of fak3r.com as well as learning what worked, and where the bottleneck occurred. First of all the all important numbers; visits, pages, hits and transferred data for 2005-10-15, as reported by Awstats:

Date		Pages		Hits		Bandwidth
10-14-2005	18092 		95899 		644.47 MB

Holy smokes, 95,899 hits for the day while transferring almost a cd’s worth of data. Again, not a huge number for a colo’d webserver with a big audience, but for a home built rig behind a 384/1.5 ADSL line, pretty cool. Other interesting data gathered was:

Operating Systems (Top 10)
Operating Systems Hits Percent
Windows 80112 63.6 %
Linux 25319 20.1 %
Macintosh 14305 11.3 %
Unknown 5034 4 %
FreeBSD 741 0.5 %
Sun Solaris 278 0.2 %
OpenBSD 37 0 %
NetBSD 20 0 %
WebTV 1 0 %

This was expected since it’s a tech site, but it was still nice to see Linux so well represented (when I hit the site from work I’m coming in via XP unfortunately)

Browsers (Top 10)
Browsers Grabber Hits Percent
Firefox No 84168 66.8 %
MS Internet Explorer No 17268 13.7 %
Safari No 9715 7.7 %
Mozilla No 4509 3.5 %
Opera No 4011 3.1 %
Unknown ? 2867 2.2 %
Konqueror No 1278 1 %
Camino No 555 0.4 %
Galeon No 405 0.3 %
Netscape No 307 0.2 %
Others 764 0.6 %

Again, same disclaimer, but it would be nice if Firefox were the rule, and not the exception for the general public.

So what did I learn? I learned that my FreeBSD 6.0 box is setup well enough to handle *at least* 100,000 hits a day. I learned that using Apache2 -> mod_proxy -> lighttpd -> fastcgi powered by Typo for blogging is a good enough combination to easily handle the traffic thrown it’s way. Still, while my server was only running the Ruby process around 35%:

51469 fak3r         1  99    0 38632K 32832K RUN  14:50 34.47% ruby

static pages were still taking ~60 seconds to refresh during peak load. Thankfully my ssh tunnel held up, so I was watching top and tailing the logs in real time. From this I could see that everything was setup to handle the traffic, with headroom to spare, expect for my ADSL line, which still performed as expected. It was indeed the bottleneck, but the fact that pages were still being severed (albeit slowly) showed it handled the traffic and served pages to all that would wait for them. In the future I will likely use my OpenBSD firewall running pf to limit the traffic to the websever to still allow internal clients the bandwidth to surf, but with the same configuration this would only decrease our numbers. So, better test would have this box on a bigger pipe (Speakeasy has a 1.5/6.0 line, as well as T1 options) which probably won’t happen in the confines of my home network, but would likely really push the server to its limits. Perhaps one day, in a colo’d location, my new FreeBSD powered 4U server on a T1 will notice a spike in traffic; seconds after my recent post to Slashdot…

This morning on Slashdot there was a story about Ruby on Rails and my comment turned out to the the second post. I took the opportunity to plug this site…err…I mean used this site as an example of Ruby on Rails via Typo and suggested people take a look and try out the ‘live search’ to give the database a workout, and did they ever. Logfiles were just scrolling along, httpd was throwing up pages, Ruby was driving all database queries via fastcgi. Top showed Ruby pushing upwards of 18%, so I killed Hulaweb, which was eating more, and watched Ruby take over, running at 34% at one point:

51469 fak3r   1  99    0 38632K 32832K RUN  14:50 34.47% ruby

During one of the peak load times a full page reload from my client took just over 60 seconds, which is a very long time, but the DSL was the bottleneck, not the server or any of it’s processes. Tailing the logs I watched it continuously spitting out pages to other clients in the queue before me, so Lighttpd was doing its job as it should, and it had plenty of RAM/proc overhead (even though I saw it peak around 34%! at times), so the delay was simply my home DSL (1.5/384 down/up); which performed adminrable condisering the abuse. Thanks Speakeasy! Once things calmed down a bit three hours later (ruby was still using ~18%) I could hit pages and have them reload as if nothing was happening, database searches as well.

So, for some very *rough* numbers; my ‘second post’ to Slashdot occured at 9:15AM, and at roughly 12:15AM /var/log/http-access.log showed 50,000 mod_proxy requests. So, requests that Apache handled via mod_proxy to Lighttpd:

50,000 / 03 = 16666.6666 pageviews/hour

16,666 / 60 = 277.7777- pageviews/minute

277    / 60 = 4.629- pageviews/second

So my home server that I built by hand, running behind a standard ADSL line, served up an average of ~5 pages per second for over three hours. Fuckin’ A!  Later, at 6 hours (3:15PM):

mod_proxy calls (page requests)

grep 14/Oct /var/log/httpd-access.log | wc -l
78741

DB Calls

awk '/^"action"=>"search"/' /usr/local/typo/log/production.log
grep Oct 14 production.log | wc -l
13478

Successful DB Results

awk '/^Completed/' /usr/local/typo/log/production.log
| grep Oct 14 production.log | wc -l
13478

I’ll do more numbers once I learn if the way I pulled them were accurate, but as it stands the box handled almost 80,000 page requests today, and the logs are still rolling. Stay tuned.