HOWTO monitor your servers via Twitter

Alert: your server has failed!

UPDATE: thanks to a reader’s comment I looked into what it would take to get this working again since Twitter has completely disabled the old style of authentication in favor of full on OAuth. Basically a lot. To just post messages now it seems far more complex than it once was.My original idea with this was to do it as low tech as possible so users wouldn’t have to install a ton of stuff and configure it – I wanted it to ‘just work’ easily. Now with OAuth it seems this will never work easily, first of all you have to ‘register an application‘ for it to have access to OAuth – which seems crazy to me, we don’t want it to have access, just the ability to push a message to an account. Then if you look there are libraries out there that *can* post, but look at the directions for one of the libraries, it involves not only building the app and getting temporary access to the twitter API, then you also have to get the two keys from that and bake them into the app by recompiling it, and then more configuration, etc. So for now I am MARKING THIS IDEA AS DEAD. If I figure out a new way to do it that I can sketch out I will, or if anyone else has a simple way post it in the comments and I’ll update it here. Thanks for your interest and good luck!

The other day I got inspired to write a script that would allow me to monitor my servers via Twitter. The idea of having a column in Tweetdeck set aside to inform me of my servers’ statuses’ sounded cool, plus, it’s quicker than checking email.  I know sending tweets from the command-line had been done before, but after seeing briealeida’s post titled Tweeting Cron Jobs I really got inspired. While hers was written in Perl, I didn’t want to go that route since I had a few, self imposed, restrictions I wanted to stick to. One, I only wanted to use standard shell commands, the ones you get by default in Linux, so you would have absolutely no dependencies to install for this to work. Two, I wanted to see how much info I could stuff into a 140 character tweet, and still have it make sense. While I’m still working on adding more info, the current state of the script gives me a quick snapshot of seven specifics metrics on a selected server, which I’m quite happy with. To try it yourself only takes a few minutes.

First you’ll need a Twitter account, either the one you already use, or a dedicated one that only your servers post to (this is what I’ve done). For a tad bit more security I protected my tweets and made the account private – I don’t think I’ll get hacked by anyone that sees my system load is too high, but I’d rather not hear about it from security experts that I’m leaving myself open, so I’ll continue to keep things blocked by default. Also, I’ve setup the update method to use SSL to login and transmit, so you can rest a little easier knowing that things are staying encrypted while they move over the wire.  The only option I’ve coded in is that you can use wget (by default) or curl (not installed by default in any Linux distro I’m running, but is on OS X if someone wants it for that), so you have a little bit of flexibility if you need it.  Otherwise it’s pretty much ready to go out of the box, define your username and password at the top of the script, chmod it, and away we go.  Slap it in your crontab  for daily/hourly updates, and the rest, as they say, will be handled by simple, beautiful, BASH.

[codesyntax lang="bash" lines="no" container="pre_valid" capitalize="no"]

#!/bin/bash
#
###############################
# twitter username/password #
###############################
user="1user"
pass="sekrit"
#
###############################
# run tasks for the report #
###############################
HOST=`hostname -s`
UP=`uptime | cut -d" " -f4,5 | cut -d"," -f1`
LOAD=`uptime | cut -d":" -f5,6`
PING=`ping -q -c 3 google.com | tail -n1 | cut -d"/" -f5 | cut -d"." -f1`
MEM=`ps aux | awk '{ sum += $4 }; END { print sum }'`
CPU=`ps aux | awk '{ sum += $3 }; END { print sum }'`
if [ -x "/usr/bin/lsb_release" ]; then
DIST="`lsb_release -s -i`/`lsb_release -s -c` on `uname -m`"
else
DIST="`uname -o` on `uname -m`"
fi
#
###############################
# build the report for post #
###############################
tweet="(HOST) ${HOST} (UP) ${UP} (CPU) ${CPU}% (MEM) ${MEM}% (LOAD) ${LOAD} (PING) ${PING}ms (DIST) ${DIST}"
#
################################
# check that post is <140 char #
################################
if [ $(echo "${tweet}" | wc -c) -gt 140 ]; then
echo "FATAL: The tweet is longer than 140 characters!"
exit 1
fi
#
################################
# post the report to twitter #
################################
### via wget (default)
wget -q --user="${user}" --password="${pass}" --post-data=status="${tweet}" https://twitter.com/statuses/update.xml
### via curl
#curl -k -u ${user}:${pass} -d status="${tweet}" https://twitter.com/statuses/update.xml >/dev/null 2>&1
#
################################
# if no errors, successful #
################################
if [ $? -eq '0' ]; then
echo "Tweet sent using `echo ${tweet} | wc -c`/140 characters."
fi
rm update.xml
#
################################
exit 0

[/codesyntax]

Told you it was pretty simple, now I get updates to my ‘secret’ server account on Twitter that look like this:

(HOST) mookie (UP) 8 days (CPU) 6% (MEM) 44.1% (LOAD) 0.07, 0.06, 0.07 (PING) 29ms (DIST) Debian/squeeze on i686

So what other statistics would be helpful to gather that we can tack on to what we already have? Heck, we could even drop the (DIST) section, that just gives me a reminder if what distro each box is running; it could be shortened to just say Debian or Slackware. One thing I’m working on, I would like a monitor to report when any partitions get over 90% full, I haven’t been able to figure that one out yet without using an external app not installed by default, like monit.  So, is this helpful? Can you think of better way to do any/all of the above, while fulfilling my basic requirements? Does it blend?




  • http://www.unixsysadmin.org briealeida

    Hey,

    Way cool! I was talking with my dad about my post and he suggested using SNMP to accomplish this. I’m currently working on a way to do this with Python (which is installed in most distros and wouldn’t require the Net::Twitter module I had to install with mine). So yeah, but nicely done. I’m going to implement this as well. :) !


    Brie

    • http://fak3r.com fak3r

      thanks, glad you like it. I’m not adverse to using outside apps to get more info, I just wanted to constraint it to see how much I could do with just basic commands. I’d like to add things, expand it, let me know if you have more ideas, and I’ll keep checking your site, some really nice howtos for inspriation!

  • http://www.unixsysadmin.org briealeida

    Hey,Way cool! I was talking with my dad about my post and he suggested using SNMP to accomplish this. I'm currently working on a way to do this with Python (which is installed in most distros and wouldn't require the Net::Twitter module I had to install with mine). So yeah, but nicely done. I'm going to implement this as well. :) !–Brie

  • http://www.unixsysadmin.org briealeida

    Hey,Way cool! I was talking with my dad about my post and he suggested using SNMP to accomplish this. I'm currently working on a way to do this with Python (which is installed in most distros and wouldn't require the Net::Twitter module I had to install with mine). So yeah, but nicely done. I'm going to implement this as well. :) !–Brie

  • Jim

    Personally, I’d skip the OS declaration and processor type and instead report temperatures. Perhaps hddtemp for /dev/sda and lm-sensors to read the temp for CPU0.

    • http://fak3r.com fak3r

      yep, that would be more useful, good ideas

  • Jim

    Personally, I'd skip the OS declaration and processor type and instead report temperatures. Perhaps hddtemp for /dev/sda and lm-sensors to read the temp for CPU0.

  • http://fak3r.com fak3r

    thanks, glad you like it. I'm not adverse to using outside apps to get more info, I just wanted to constraint it to see how much I could do with just basic commands. I'd like to add things, expand it, let me know if you have more ideas, and I'll keep checking your site, some really nice howtos for inspriation!

  • http://fak3r.com fak3r

    yep, that would be more useful, good ideas

  • http://fak3r.com fak3r

    thanks to a tip by @rosy1280, I now have a way to get the % usage of my /root partition:
    df -h | grep -v “% /[a-z]” | tail -n1 | awk ‘{print $5}’

    it also showed me that I should probably be using awk ‘{print $x}’` instead of my old/hack-y tail | cut | cut

    Thanks, this is helping!

  • http://fak3r.com fak3r

    thanks to a tip by @rosy1280, I now have a way to get the % usage of my /root partition: df -h | grep -v “% /[a-z]” | tail -n1 | awk '{print $5}'it also showed me that I should probably be using awk '{print $x}'` instead of my old/hack-y tail | cut | cutThanks, this is helping!

  • http://www.air-jordan-19.com air jordan 19

    Leisure is more and more important in the life. Here I know a lot of leisure that is good at me. Adjust our life interest, bring us the enthusiasm of exercise. The post is very well. By the way I know some websites is very well such as XXXXX. Disscount fashionable item is hotting on sale! news-need.com

  • http://www.bootoutlet.co.uk/ ugg boots

    I am agree with landlord.Here provide many game for everyone . Wonderful artical excelling nature images. Let you have unexpected surprise. After you play game tired ,you can come to these websites to relax your mood. http://www.air-jordan-22.com

  • Mineo

    A cosmetician wholesale mac cosmetics is a professional wholesale mac who provides facial wholesale mac makeup and body treatments for clients. The term cosmetologist is wholesale mac products sometimes used interchangeably with this term, but most commonly refers to a certified professional. A freelance makeup wholesale mac makeup products artist provides clients wholesale mac makeup products with mac cosmetics beauty advice and cosmetics assistance—usually paid by the cosmetic company by the mac makeup hour.

  • http://twitter.com/mlv Michael Vezie

    For those of us who’ve already taken the plunge with OAUTH and scripts, it’s not a big deal, and this script is still cool.

    One thing I changed: if there’s a serious problem, I prepend @myNormalTwitterAcct to it, so I get an alert as a mention.

    • http://fak3r.com fak3r

      Do you have steps too setup OAUTH? I just haven’t taken the time to really look into it, but I’d like to create a 1, 2, 3 step document to make this work.

      That’s a great idea of including your normal Twitter account if something serious is happening, thanks for sharing it!

      • http://twitter.com/mlv Michael Vezie

        I used a python Twitter library, tweepy and wrote my own wrappers for it. I still had to register my own application with Twitter, and connect my accounts, but that’s a one-time task. If you want a simple post script, probably the best place to look is http://jeffmiller.github.com/2010/05/31/twitter-from-the-command-line-in-python-using-oauth

        If you prefer perl, then I recommend http://www.floodgap.com/software/ttytter/ You won’t have to register it. Look at twitter.com/big_ben_clock for a client example.

        Another thing I did was run it from $reliablehost getting data from $unreliablehost. I then can also tweet if $unreliablehost is down.

        • http://fak3r.com fak3r

          Wow, thanks very much for all the detail, I’m going to try this, I also want to try and tie in monit to it (I’m a big fan of monit btw http://fak3r.com/tag/monit/) so if monit wanted to reach me it could Tweet me as well as the normal email…cool. Look for a follow-up post with docs using OAUTH as you suggested, thanks again!

          • http://twitter.com/mlv Michael Vezie

            You’re welcome!

Read previous post:
HOWTO use monit to monitor sites and alert users

Ok, I've used the process management software, monit, since at least 2004, and it is simply an indespensible tool in [...]

Close