fak3r » geek

HOWTO securely delete files in OS X on the commandline

fak3r — Wed, 13 Jan 2010 02:51:44 +0000

So I’ve had my MacBook Pro for a few months now, and since I have a 500 Gig harddrive, I haven’t bothered to empty my trash yet. Now I’m on a work trip in China, and it makes me think about the privacy (internet and otherwise) that I have in the US, that I don’t expect here. In fact, since we’re blocked from posting to either Facebook or Twitter, I know this post will only make it there because this site will post if for me after I post it to my site (again, not something you’d think about just living in most other parts of the world). So what a good time to learn how to securely emptying my trash! The first thing I did was use the ‘Secure delete’ feature of the OS X trash folder, but with over 190,000 files to remove, it sat there at 0% while the fan spun up for about 15 minutes. That was it for me, it was clear it was going to take years for this to happen, so canceled that and hit Google to learn the right way to do it via the commandline. One of the best pages talks about srm a secure file deletion for posix systems that is installed by default on OS X. I’ve crafted my srm command to use the nice command to reduce the amount of overhead the process causes (again, the GUI version was taking over the system and heating things up quickly) and the sudo command to ensure all files would be deleted regardless of permission/ownership. In the end in looks like this:

nice -19 srm -rfv ~/.Trash/*

Yeah, while the -v flag will slow things down slightly, I prefer to have ‘verbose’ output from the command to understand exactly what it’s doing. Does anyone have better/more secure way to do this? Leave a message in the comments if you do, I’d love to learn more about this.

HOWTO run Chromium OS on a Dell Mini 9 with wifi

fak3r — Tue, 12 Jan 2010 09:48:23 +0000

While I still really dig my Dell Mini 9, even with 2Gig of RAM it feels kinda sluggish when I have my normal 50 tabs open, and I’ve always known someone could do better (since I’m too lazy to recompile a kernel for it like I would have in the past). With all the focus on netbooks it was bound to be addressed, and while Android looks promising, it’s currently still more of a phone OS than something you’d be able to use on your netbook. I’ve run it off a USB drive on the Mini 9 just to check it out, it was cool, but again, not really usable enough for a ‘top – maybe that’s not the target. Another I want to check is Moblin, Intel’s effort using Ubuntu as a base, but I haven’t seen a Mini 9 HOWTO (maybe I’ll have to write my own…) for that. So, enter Google Chrome OS, Google’s idea of how to not only address this problem, but perhaps lay out how we will use these computers in the future. It’s always funny when I start talking about cloud and thin clients, it takes me back to dumb terminals talking to mainframes, but I digress. The point is, thanks to great posts at jasongriffey.net and Lifehacker, it’s really easy to install Google’s Chrome OS on a Dell Mini 9, the only thing I really have to add is that you have to use ChromeOS Zero from the hexxeh.net site. After all, this is an open source project, so folks are going to make changes/fix things and share with everyone. Looking at the site they had a new release, yesterday (gotta love it!) The last time I tried a build the wifi on my Mini just worked, so it looks like those problems are a thing of the past.

HOWTO defend databases from SQL attacks with GreenSQL

fak3r — Thu, 10 Dec 2009 23:26:16 +0000

UPDATE: as if to underscore the importance of this tool and approach, yesterday a story hit about a SQL Injection attack infecting over 132,000 systems in short order. Net-Security have the full details on this attack, including how it probes the host via JavaScript to check for known vulnerabilities, how it exploits them, and how it ultimately downloads a back-door trojan to get the game going. It’s really amazing to see how complicated and professional these things have gotten, and just adds to the reasoning that we have to step up to the plate and learn how to better defend against them.

I’ve been privy to some log dumps showing real, and successful, SQL attacks on some MSSQL servers before, and they weren’t pretty. Of course a SQL injection attack has little to do with the database (well, as long as it’s still SQL based at least (nod to CouchDB and MongoDB)), and more with the code that calls it, and how that code deals with sanitizing inputs. For this reason MySQL is just as vulnerable, after all, bad code is bad code. While a client of mine opted for a firewall ‘module’ they had to buy an additional licence for, that set them back many thousands of dollars, I knew there had to be cheaper/better ways to address this kind of vulnerability. One way of course is to fix the code, but with legacy sites that no one has touched for years, this may be impractcal (I didn’t say this, I only heard it), and the other idea is to proxy the SQL and ‘clean’ it before it hits the database. The advantage of this approach is that it protects against known attacks, as well as unknown attacks, since it limits so much of what an attack is allowed to accomplish when trying to get its’ foot in the door. This approach is what the folks over at GreenSQL have done, and it’s very impressive. They sum things up nice and sweet with, “GreenSQL is an Open Source database firewall used to protect databases from SQL injection attacks. GreenSQL works as a proxy for SQL commands and has built in support for MySQL & PostgreSQL . The logic is based on evaluation of SQL commands using a risk scoring matrix as well as blocking known db administrative commands (DROP, CREATE, etc). GreenSQL is distributed under the GPL license.” A high-level view shows GreenSQL acting as the proxy from the frontend to the database:

This sounds and looks ideal, so with that in mind I installed GreenSQL with the default options on a MySQL server, and setup the included web-based Management Console to kick the tires and see what it does. To show that it’s working I logged into a MySQL database, first directly, on port 3306:

# mysql -h 127.0.0.1 -P 3306 -u dbadmin -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 24768
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.mysql> show databases;
+-------------------------+
| Database                |
+-------------------------+
|database      |
| names were listed here> |
+-------------------------+
32 rows in set (0.00 sec)
mysql> quit

And that’s what you’d expect, you login as a privledged user and you can see all of your databases, simple. Now I tried it going through GreenSQL, so essentially a proxy to the database server, on port 3305.

# mysql -h 127.0.0.1 -P 3305 -u dbadmin -p
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 24763
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> show databases;
Query OK, 0 rows affected (0.00 sec)
mysql> quit

Ah, ok, I can see how this would make things more secure! I then logged into the Management console to see what it had to say about the incident, and it told me what had happened:

Matching queries:
Query:    show databases
Time:    2009-12-09 22:02:26
DB User:    dbadmin
Risk:    31 blocked
Reason:    Detected attempt to discover db internal information.
ID:    1

After this I went into my website’s config file, (wp-config.php in Wordpress) and instructed it to connect to port 3305, instead of 3306. A bit of a note, this wasn’t directly documented in my wp-config.php file, so to change the port, you just add a colon and the port number at then end of localhost, or your DB hostname, so it looks like this:

define('DB_HOST', 'localhost:3305');

I bring this up because most other configs will include a seperate line for the port, but hey, this works too, it was just a little ‘gotcha’ that I had to Google. Now GreenSQL is protecting my websites, including fak3r, and I expect to only expand upon this as I learn more ways to protect my servers from the wilds of the Internet.

Conclusion: While some will whine that GreenSQL “only” supports MySQL and Postgresql, when I first looked into this they were only supporting MySQL – and that was last month, so I suspect they’ll cover things like MSSQL and even Oracle in the future. This would be huge for businesses and corportations of all sizes to protect their data, something sites like the Chronology of Data Breaches are showing us are STILL NOT HAPPENING. Plus, coming from a firewall background, this seems like an obvious way to protect things. While there’s a performance trade off, their tests show it to be very minimal, but I’ll try to run my own tests and report back. So, make no doubt about it, the GreenSQL folks have put together an enterprise ready product that they are actively developing to address the latest database threats. Highly recommended.

Please post questions or comments below, I’m always learning and and chances are you know something I don’t!

EFF’s SSD (Surveillance Self-Defense) Project

fak3r — Wed, 02 Dec 2009 18:25:16 +0000

EFF has a page covering what they call The SSD Project (Surveillance Self-Defense) which they provide, “…to educate the American public about the law and technology of government surveillance in the United States, providing the information and tools necessary to evaluate the threat of surveillance and take appropriate steps to defend against it.“ This is important stuff, and what I wish others would know, so I’m posting links to the source in the hope it will get more exposure and results in the search engines of the Internet. I will contact EFF and see if we can formulate a better method to disseminate and distribute this text, allowing for updates and annotations going forward. Also, I aggregate news that cover these kind of issues over on Left to chance, take a look, then follow @lefttochance and @eff on Twitter to stay informed, and consider joining the LinkedIn EFF Group I run to join in the conversation. In other words, get involved and …

Know your rights!

Ruby on Rails: gem install versus apt-get

fak3r — Wed, 18 Nov 2009 17:09:35 +0000

UPDATE: Thanks to Ryan, Ant and Fern for the tips. With that in mind I found an online Slicehost tutorial that contained the steps and explained how to install ruby via apt-get, then get the latest rubygems, install that manually, ran gem to update itself, then run gem to install rails – as suggested. The steps I took from that page:

On a Debian Lenny system that does not have ruby, rubygems or rails installed on it yet:

apt-get update apt-get upgrade apt-get install ruby-dev ruby ri rdoc irb libreadline-ruby libruby libopenssl-ruby sqlite3 libsqlite3-ruby libsqlite-dev libsqlite3-dev

Once that completes without errors, make sure ruby is installed and ok:

ruby -v

Now download the latest rubygem (1.3.5 as of this post) from RubyForge http://rubyforge.org/frs/?group_id=126:

wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz

Unpack it, change into the directory, run setup:

tar xzvf rubygems-1.3.5.tgz cd rubygems-1.3.5 ruby setup.rb

After that you’ll see:

RubyGems 1.3.5 installed

Then it’s suggested that you make a symlink to gem1.8 so you can run it as gem:

ln -s /usr/bin/gem1.8 /usr/bin/gem

Now make sure everything is up to date (even though we just installed the latest):

gem update gem update --system

And finally – install rails:

gem install rails

After this you can check what gem has installed, and their version numbers:

gem list

And there you have it, more steps than I wanted, but now I know how to have a Debian system up to date, with Ruby, and then having rubygems handling all of the other ruby things that are better dealt with as gems. As for systems I already have running in production in mixed enviroments? I’ll look to migrate those to properly configured installs in the future. I guess for extra credit I should contact the maintainer of rubygems, and the associated gems, for Debian to get their side of the story, or maybe a solution they could put in place moving forward.

Original post:

I’ve been using Ruby on Rails on and off for many years now, and friends are always showing me new RoR apps to try out that look fly. I can get things up and running fine, but it’s when the time comes to update an app that I have issues; I seem to come to the fork in the road where apt-get doesn’t have the latest version of Rails or some dendancy, and gem install is the proposed solution. I worry that mixing the two updating procedures will mess things up, since I have seen this before in Debian GNU/Linux, as well as FreeBSD (I suspect it’s me, and there’s a right way to do it). So, for example, today I noticed there was a new version Redmine a few days ago, so I update to the latest via SVN (the suggested way of updating Redmine):

# cd /opt/redmine-svn # svn up At revision 3076.

Now I copy in the email.yml and database.yml from my working instance so this will use the same config:

# cp /opt/redmine/config/database.yml /opt/redmine/config/email.yml config/
So far so good, let’s rake it up:

# RAILS_ENV=production rake db:migrate (in /opt/redmine-svn) Missing the Rails 2.3.4 gem. Please `gem install -v=2.3.4 rails`, update your RAILS_GEM_VERSION setting in config/environment.rb for the Rails version you do have installed, or comment out RAILS_GEM_VERSION to use the latest version installed.

So here we are, crap, what version of Rails do I have installed via apt-get?

# apt-cache showpkg rails | head -n3 Package: rails Versions: 2.2.3-1 (/var/lib/apt/lists/ftp.debian.org_debian_dists_squeeze_main_binary-i386_Packages) (/var/lib/dpkg/status)

Damn, so what version of Debian am I running?

# cat /etc/issue.net Debian GNU/Linux squeeze/sid

Yep, the latest, testing branch. So here I am, do I leave the apt-get world and start up gem install or what? My hesitation is that this is my ‘production’ version of Redmine, and I don’t really want to build out a sep install just to test my Rails updating, and if I do that, will the gem Rails install hose my current apt-get installed Rails anyway? So this is the problem I’ve had since I started playing with Rails apps, and it’s been about 3 years now (fak3r.com was on Typo for almost a year). I’m open to suggestions as to how others handle this, do you just install Debian and then not even use apt-get for Rails/Ruby stuff? It seems that gem install always have the most up to date stuff, I’m just concerned that updating things that way will interfere with an apt-get update; apt-get upgrade of the main system later, particularly now that I’m already in the apt-get side. Do I reinstall and go all gem install for just Ruby stuff, and apt-get just for the system? How do people segment this? There has to be a proper way that I’m missing.

Comments?

HOWTO build your own open source Dropbox clone

fak3r — Tue, 15 Sep 2009 04:21:03 +0000

UPDATE: Thanks to everyone who has contributed to this, and the Reddit thread, as it has provided some great ideas building off of my concept. I’m starting to rethink about how we could have version control on top of things, and I’ll update things when I have more to share. Also, does anyone have iFolder (thanks for the proper linksalubrium) working? It looks like you need SUSE Linux, which I don’t have access to, plus I know most Novell projects need a *ton* of Mono dependencies installed to have any of their stuff working, at least on the server side; but it sounds like they have Mac, Linux and Windows clients, which is encouraging. While for my needs something a bit more ‘close to the bone’ (as below) might be better for the server side, having it be inter-operable with something like iFolder could provide a lot more functionality for others.

First off, if you haven’t tried Dropbox, you should check it out; sync all of your computers via the Dropbox servers, their basic free service gives you 2Gigs of space and works cross-platform (Windows, Mac, Linux). I use it daily at home and work, and just having a live backup of my main data for my work workstation, my home netbook, and any other computer I need to login to is a huge win. Plus, I have various ’shared’ folders that distribute certain data to certain users that I’ve granted access to, this means work details can be updated and automatically distributed to the folks I want to review/use the data. I recommend everyone try it out, and see how useful it is, it’s turned into a game changer for me. So a few months ago they made headlines on supporting Linux as they released the client as open source. While this got hopes up for many, it was only the client that was open source, the server is still proprietary. While slightly disappointing, this is fine, they’re a company trying to make money. I don’t fault them for this, it’s just that a free, portable service like that would be a killer app.

Meanwhile at work I’m working on a solution to sync large data clusters online and the project manager described it as the need for ‘Dropbox on steroids’. Before I had thought it was more complicated, but after thinking about it, I realized he was right. Look, Dropbox is a great idea, but it obviously is just a melding of rsync, with something watching for file changes to initiate the sync, along with an easy to use front end. From there I just started looking at ways this could work, and there are more than a few; here’s how I made it work.

Linux now includes inotify, which is a kernel subsystem that provides file system event notification. From there all it took was to find an application that listens to inotify and then kicks off a command when it hears of a change. I tried a few different applications like inocron, inosync and iwatch, before going with lsyncd. While all of them could work, lsyncd seemed to be the most mature, simple to configure and fast. Lsyncd uses inotify to watch a specified directory for any new, edited or removed files or directories, and then calls rsync to take care of business. So let’s get started in making our own open source Dropbox clone with Debian GNU/Linux (lenny)

Ladies and gentlemen, start your engines servers!

First, you need 2 severs; one being the server and the other the client. (you could do this on one host if you wanted to see how it works for a proof of concept)

Install OpenSSH server

First you’ll need to install OpenSSH Server on the remote system:
apt-get install openssh-server

Configure SSH for Passwordless Logins

You’ll need to configure passwordless logins between the two hosts you want to use, this is how rsync will pass the files back and forth. I’ve previously written a HOWTO on this topic, so we’ll crib from there.

First, generate a key:

ssh-keygen -t rsa

(Enter)

You shouldn’t have a key stored there yet, but if you do it will prompt you now; make sure you overwrite it.

Enter passphrase (empty for no passphrase):

(Enter)

Enter same passphrase again:

(Enter)

We’re not using passphrases so logins can be automated, this should only be done for scripts or applications that need this functionality, it’s not for logging into servers lazily, and it should not be done as root!

Now, replace REMOTE_SERVER with the hostname or IP that you’re going to call when you SSH to it, and copy the key over to the server:

cat ~/.ssh/id_rsa.pub | ssh REMOTE_SERVER 'cat - >> ~/.ssh/authorized_keys2'

Set the permissions to a sane level:

ssh REMOTE_SERVER 'chmod 700 .ssh'

Lastly, give it a go to see if it worked:

ssh REMOTE_SERVER

You should be dropped to a prompt on the remote server. If not you may need to redo your .ssh directory, so on both servers:

`mv ~/.ssh ~/.ssh-old`

and goto 10

Install rsync and lsyncd

Next up is to install rsync and lsyncd. First, rsync is simple, and could already be installed (you don’t need to run it as a server, just the client), make sure you have it with:

apt-get install rsync

Next is lsyncd. There is no official Debian package yet, but it’s simple to build from source and install. First off, if you don’t have build essentials you’ll need them, as well as libxml2-dev to build the lsyncd source. Installing those is as simple as:

apt-get install libxml2-dev build-essentials

Now we’ll get the lsyncd code (you can check for a newer version at http://lsyncd.googlecode.com) and build that:

wget http://lsyncd.googlecode.com/files/lsyncd-1.26.tar.gz
tar -zxf lsyncd-1.26.tar.gz
cd lsyncd-1.26
./configure
make; make install

This install does not install the configuration file, so we’ll do that manually now:

cp lsyncd.conf.xml /etc/

Configure lsyncd

Next up, we’ll edit the configuration file now located in /etc The file is a simple, well documented XML file, and mine ended up like so – just be sure to change the source and target hosts and paths to work with your systems:

[sourcecode language='xml']

[/sourcecode]

Launch lsyncd in debug for testing

We’re ready to give it a go, may as well run it in debug for fun and to learn how lsyncd does what it does:

lsyncd --conf /etc/lsyncd.conf.xml --debug

Watch for errors, if none are found, continue.

Add files and watch them sync

Now we just need to copy some files into this directory on the source box:

/var/www/sync_test

And again, watch for any errors on the screen, if these come back as a failed connection it’ll be an SSH/key issue, common, and not too difficult to solve. From here add some directories and watch how they’re queued up, and then take a look at them on the remote box: from this point out it “just works”. Now give it more to do by adding files and directories, and then the logging for errors while they sync. As it stands the system uses the source system as the preferred environment, so any files that change, or are added or removed, will be processed on the remote system. This is analogous to how Dropbox works, you can use multiple sources (your laptop, your desktop, etc) and their server serves as the remote system, keeping all the clients in line.

Conclusion

You should now have a basic, working Dropbox style setup for your own personal use. I had this running and used it to sync my netbook back to my home server, and then have my work desktop sync to my home server, so both the netbook and the desktop would stay in sync without me doing anything besides putting files in the specfied folder. For my week long test I ran a directory alongside my Dropbox directory just to see how they both acted, and I didn’t have any failures along the way.

Now we have is a simple Dropbox style app that is lightweight, with a functional back-end running rsync, which is a known stable app that will scale, and while it doesn’t provide the front-end and web view that Dropbox does, that could be an easy part for a UX developer to tackle. The cool thing is, we have a solution that works, and other options like the apps I described in the beginning, can be dropped in and replace the functionality lsyncd provides in case they can do something better. For now, I’m playing around with it to learn the ins and outs of the system to see how it will behave long term under a much larger store (50Gig to start) to keep in check. I will also work on better integrating this solution it into a working system, and update this tread with init scripts, reports, or maybe even a web view beyond just an index view from Apache or nginx. Ideally we could have a web front end that would intelligently report if a file is complete on the server, and if the file is completely mirrored on another server or client. P2P or Bitorrent would also be really cool to consider with this, and I’m sure there will be more applications for a setup like this once we’ve it around as a resource for a time. Can you think of more applications for this? Did you get it to work? Can you think of a better way to do this?

Architecture for future LSID resolvers

fak3r — Tue, 18 Aug 2009 20:32:20 +0000

NOTE: the following is my generalized overview of some thoughts I came up with months ago in regards to LSID resolvers, and how to architect a fault tolerant solution (LINK). I missed the meeting in Denmark last week (I was on a family vacation for once) where they were discussing this, and wrote the following for another attendee to submit on my behalf. I’m posting it here for further exposure and discussion of the merits and shortcomings of these ideas.

“In thinking about the architecture for future LSID resolvers we need to remember that a single point of failure will fail. This has been proven true too many times, and it’s clearly not the way to proceed if we want to build a system the community can rely on. To succeed we should follow successful implementations of distributed software services such as the domain name server (DNS), and the network time protocol (NTP). These succeed because the protocols are standardized and the software to connect to, and utilize, these services are simple to deploy. If we had a package-able server that would connect to these services and be easy to deploy, an institution could ultimately count on its own server (with other servers providing automatic fail over) to serve as a resolver that would be updated against lead node servers regularly.

Using the idea of DNS replication, we can understand how something like this could function. A primary server would serve servers below it passing out changes (deltas) such as adds and edits occur. An institution wouldn’t have to check with a primary server to get resolution, it could talk to any of the servers that are in line for updates. Meanwhile having a batch of main servers behind a URL would allow pooling of resources just as the pool of Unix networked time servers all resolve around a single URL (http://www.pool.ntp.org). The network time protocol (NTP) allows many nodes to resolve to the same address, so that if one server is unavailable, another one can provide automatic failover, without the need to change the URL.

After setting up this type of architecture all that would be left to do would be to expose the data on a server to others. Once other nodes were brought online and replicated against lead nodes, this would form the basis for a permanent and sustainable resolver system. If a main node ever died, it could be resurrected from other lead nodes, or even from an institution’s copy of the current database, so that no data (original or accumulated) would be lost. Using a distributed, fault-tolerant architecture with replication and protocols designed to handle these issues would allow LSIDs to become the true resource the community needs, and be sustainable for the long term with minimal administration overhead, and a low barrier for entry for would be contributors.”

File system full, but why?

fak3r — Wed, 27 May 2009 04:23:35 +0000

UPDATE: posted my workaround code below, good feedback already from Ryan (djatoka dev) and I’ll be testing the proper fix on the server soon.
I’ve got a server that keeps filling up its disk space and failing to serve images after it gets to the file system full error message. First of all let me say, I don’t blame it in the least, if the admin (aka me) doesn’t do enough to secure the server enough disk space to do its job, I say, let me have it. But after I’ve set the suspect daemon to use a *reasonable* amount of space I stopped thinking of it as the culprit, so when this issue arose again, I looked elsewhere for the cause. Fast forward to today, the server’s file system filled up again, and refused to serve any more data, again, I totally understand where the server is coming from, if it doesn’t have enough disk space to do its job, it shouldn’t have to apologize to anyone; it’s all on the admin (again, aka, me), but what was going on? So, after I finally figured things out, I was/am still a bit confused here, but to my defense, when I did an ‘ls -ltrs /tmp’ to look at directories of old cached files left over by adore-djatoka (which is the JPEG2000 (J2K) image server that I suspected of taking up all the disk space) :

# ls -ltrs /tmp | grep temp-20090519-*
5.7M drwxr-xr-x 2 tomcat55 root 5.7M 2009-05-19 13:08 temp-20090519.130841

I concluded that the adore-djatoka server was innocent, and I felt bad for accusing it of the infraction since the largest directory I’d come across that it was responsible for was a paltry 5.7M. Immediately I thought to myself, “…these are not the droids you’re looking for“, after all, this couldn’t be responsible for a directory that was taking up almost 100G and making me look like a sophomore taking the “Intro To Computers” class for the second time, right? But finally I got an unbiased opinion when I found a Unix utility called ncdu, which is an NCurses Disk Utility (du is an old school Unix utility which displays ‘disk usage’ of a selected directory), that, when run against the same directory as I scanned before, told me:

# ncdu /tmp | grep temp-20090519-*
70.5GB [##########] /temp-20090519.130841

Now look, I’m no math wiz, but come on, WTF am I seeing wrong here?

[...]

So, long story short, the ncdu utility is able to delete the Gigs worth of files much quicker than my script, so the server now has plenty of disk space, and I now have a rotate script that I humorously call tomcat_turnover that will:

rotate out the old Tomcat temp directory (which is where the adore-djatoka server stores its cached images)
create a new Tomcat temp directory
set the proper permissions on the new Tomcat temp directory
restart Tomcat (and thus the adore-djatoka server)
and finally, delete the contents of the old Tomcat temp directory, thus returning that used disk space to free disk space once again

So, now tell me why is this “my problem” and not instead handled by the sever, and why doesn’t the adore-djatoka server respect the settings I set in djatoka.properties?

# grep cache /var/lib/tomcat5.5/webapps/adore-djatoka/WEB-INF/classes/djatoka.properties
OpenURLJP2KService.cacheEnabled=true
#OpenURLJP2KService.cacheTmpDir=
OpenURLJP2KService.cacheSize=1000
OpenURLJP2KService.cacheImageMaxPixels=100000

Now, assumming cacheSize=1000 doesn’t stand for 1000Gig, I’ve either found a bug in the djatoka software, which I’ll post a bug report to the project to determine, or this is an error by the well meaning admin (aka me). More info when I learn about it, and yes, I’ll post my tomcat_turnover script here for extra credit next
The tomcat_turnover.sh workaround code:

# wrapper to recycle tomcat, while taking care to clean the old
# temp directory independantly of restarting tomcat.
APP=”tomcat5.5″
GROUP=”root”
DIR=”/var/lib/${APP}/temp”
DATE=.`date +%Y%m%d.%N`
# stop tomcat
/etc/init.d/${APPILCATION} stop
# shuffle directories
mv ${DIR} /tmp/temp${DATE}
mkdir ${DIR}
chown -R ${APP}:${GROUP} ${DIR}
# restart tomcat
/etc/init.d/${APPILCATION} start
# clean old temp
cd /tmp
nice -n 19 find temp${DATE} -atime 1 | while read x
do
rm $x
done
rm -rf temp${DATE}
# done
exit 0

Four free Linux eBooks

fak3r — Wed, 29 Apr 2009 23:47:13 +0000

While looking for something else, (which is mainly when I find *other* interesting things) I found an article which included links for four free Linux eBooks. This is a great resource for anyone with some Linux experience, back to others who may be looking to get started with tux, and I would have loved to have this when I started, but that was before the Internet was available to most people. So, if you’re new to Linux, or want to get started (I used Red Hat Unleashed in 1996, here it is online!), here’s some great downloads to learn from:

Linux Starter Pack

By TuxRadar

Here’s your complete guide to using Linux, taking you from beginner level to an intermediate user with ease. This ebook will show you how to install Linux, navigate around the desktop, use common software, add more programs and fix any problems you may encounter. It’s based around a slightly older version of Ubuntu Linux so there may be some minor differences with newer releases, but you can always download the latest version from www.ubuntu.com or get a CD sent by post at shipit.ubuntu.com.

The Linux Starter Pack (PDF) 11.5MB

The Easiest Linux Guide You’ll Ever Read

by Scott Morris
After several months of writing and revising, Scott has made available the “Easiest Linux Guide You’ll Ever Read”. It is a 160-page book geared towards people who are competent with using Windows, who have never attempted to use Linux but are interested in giving it a try. This ebook seems to be around for quite some time, so I wouldn’t be surprised if a lot of you have already went through this copy. For a more thorough review of this ebook, click here, or click on the link below to grab it.

The Easiest Linux Guide You’ll Ever Read (PDF) 5.66MB

SUSE Linux Administration Guide

Too many authors to be listed here
The SUSE LINUX Administration Guide provides background information about the way your SUSE LINUX operates. This book introduces Linux system administration basics such as file systems, kernels, boot processes, an Apache web server, and secure authentication.

SUSE Linux Administration Guide (HTML)

Ubuntu: Pocket Guide and Reference

A concise companion for day-to-day Ubuntu use
by Keir Thomas
A comprehensive, 170 page ebook which will get you started with Ubuntu. I believe a lot of computer users have intentions of getting a Linux system up just to test and experience the power behind this free operating system. There are various types of Linux, but so far, and at least for me, Ubuntu seems the most popular so far. I’m no means an experienced user of Ubuntu, so I can’t really speak on why it’s popular, but I think the simplicity, compatibility and user friendliness should do the trick. Some of the chapters in this ebook include:-

Introduction
Chapter One: Installing Ubuntu
Chapter Two: Configuring Ubuntu
Chapter Three: Getting to grips with the desktop
Chapter Four: Users and the filesystem
Chapter Five: Hands-on at the command-line
Chapter Six: Software management
Chapter Seven: Securing the system

Ubuntu: Pocket Guide and Reference (PDF) 2MB

Resolving LSIDs with URL resolvers and CouchDB

fak3r — Wed, 29 Apr 2009 14:29:04 +0000

Recently I’ve been looking at ways to solve some of biodiversities’ long standing issues with LSIDs, which are, “Life Science Identifiers are a way to name and locate pieces of information on the web. Essentially, an LSID is a unique identifier for some data, and the LSID protocol specifies a standard way to locate the data (as well as a standard way of describing that data). They are a little like DOIs used by many publishers.“ I posted my thoughts to the TDWG discussion mailing list on the topic, and am reprinting it here to allow for further community commentary; Code4lib, I’m looking at you. While much of it is theoretical, it is doable, and if it covers all that needs to be addressed, would be a cool, sustainable way forward for link resolvers for all kinds of usage.

I’m with Tim on this one, and taking one of Rod’s other posts (“LSIDs, disaster or opportunity“) a bit further, I think coming up with a simple, extend-able URL resolver would give us many benefits and allow LSIDs with extra, added information around them for all to use. Looking at his example, a URL would get permanent tracking that would also post referrers, location and traffic. A summary of the link could even be a page in itself, a cached version, a screenshot, or just a scrape of the code – pulling out the HTML tags, for future reference in case the real link goes down. We could use the ability to create a customizable prefix (ie- http://someresolvr.com/bhl/SDFoijF), to somewhat follow DOI conventions, but could even save old DOIs or handles for historical purposes in a field attached to the new URL, or for reuse, making the new URL resolve to a current DOI with a simple post at the end of the new URL (ie- http://someresolvr.com/bhl/SDFoijF/DOI). In the same way we could use user input, data pulled about the URL semantically to generate RDFa (by using pyRdfa), then exposing that for all newly created URLS, and coming up with a standard to make it predictable (ie- http://someresolvr.com/bhl/SDFoijF/RDF). The example at bit.ly shows the use of Open Calais to get more background information on the original link to provide more information, but it could also be pointed to other services we provide/use in biodiversity to provide a snapshot across the board of more context/content. Users of the service could login to examine/add/edit the data by hand if desired, so they would still retain ultimate control over how their record is presented. Thus, from a simple URL, we could build a complete summary that would build on what we’re given while sharing it all back out.
Then the architecture (aka, the fun part) would be simple and distributed. A webserver able to process PHP, running the database CouchDB would be all that is needed to run the resolver. CouchDB is schema-less, so the way it handles replication is very simple, and is built to be distributed, only handing out the bits that have changed during replication, as well as scale in this manner. Having a batch of main servers behind a URL in a pooled setup (think of a simplified/smaller version of the Pool of Unix networked time servers) would allow a round-robin DNS, or a ucarp setup (“urcarp allows a couple of hosts to share common virtual IP addresses in order to provide automatic failover“), so if one main server went down, another would automatically take over, without the user needing to change the URL. Plus, if we wanted to, to battle heavy usage of the main servers we could use the idea of Primary and Secondary servers as outlined in the pool.ntp.org model, so an institution with heavy usage could become a Secondary host and run their own resolver simply, with almost no maintenance. They would just need the PHP files, which would be a versioned project, and then have a cron task to replicate the database from a pool of the main servers. The institution’s resolver could be customized to appear as their own, (ie- http://someresolvr.bhl.org/bhl/SDFoijF) and for simplicity could be read-only. This way a link like http://someresolvr.com/bhl/SDFoijF could be resolvable against any institution’s server, like http://someresolvr.bhl.org/bhl/SDFoijF or http://someresolvr.ebio.org/bhl/SDFoijF – as all of the databases would be the same, although maybe a day behind, depending on the replication schedule. New entries would only be entered on a main server, or in ‘the pool’ (ie- http://pool.someresolvr.com/), then those changes would be in the database to be handed out to all on the next replication (I won’t add my P2P ideas in this email – it may not be needed for the deltas that would need to be transfered daily or weekly). Add to all of this that CouchDB is designed as “…a distributed, fault-tolerant and schema-free document-oriented database” which would fit into what we want to do; build a store of documents (data) about a URL that we can serve, while being a permanent, sustainable resolver to the original document. If the service ever died, it could be resurrected from anyone’s copy of the database (think LOCKSS (Lots of Copies Keep Stuff Safe)), so that no data (original or accumulated) would be lost. The data could be exported from the database in XML, and then migrated from that to a desired platform.
I have not been dealing with LSIDs as long as most on this list so I expect I’m glossing over (or missing) some of the concepts, so please let me know what I am lacking. This is a needed service, and is a project I’d like to be involved in building.

fak3r » geek

HOWTO securely delete files in OS X on the commandline

Related posts

HOWTO run Chromium OS on a Dell Mini 9 with wifi

Related posts

HOWTO defend databases from SQL attacks with GreenSQL

Related posts

EFF’s SSD (Surveillance Self-Defense) Project

Know your rights!

Related posts

Ruby on Rails: gem install versus apt-get

Related posts

HOWTO build your own open source Dropbox clone

Ladies and gentlemen, start your engines servers!

Install OpenSSH server

Configure SSH for Passwordless Logins

Install rsync and lsyncd

Configure lsyncd

Launch lsyncd in debug for testing

Add files and watch them sync

Conclusion

Related posts

Architecture for future LSID resolvers

Related posts

File system full, but why?

Related posts

Four free Linux eBooks

Linux Starter Pack

The Easiest Linux Guide You’ll Ever Read

SUSE Linux Administration Guide

Ubuntu: Pocket Guide and Reference

Related posts

Resolving LSIDs with URL resolvers and CouchDB

Related posts