HOWTO build your own open source Dropbox clone

Posted by & filed under geek, howto, linux.

I KAN HAZ OPEN-SRC DROPBX?

UPDATE #4 It’s 2012, and this project is still alive, although I haven’t worked on lipsync as much as I should.  I want to, and have new ideas to implement and try out in the next few months. The two way sharing is a bit hacky, and I don’t like it, the installer creates a cronjob:  that checks for server changes to sync back every minute – and it tries to avoid conflicts by not running if a sync the other way is happening. Yes, if you’re using 2 computers at once it could get confused, but so far, it’s pretty good – but something I want to improve. I’m also very interested in ownCloud  and using remote storage auth protocol like Unhosted proposes – these are two things I’d love to integrate into lipsync over the next few months. I really think having something that is all owned by the user, and in full control of the user, is still the ultimate way. Watch the lipsync.info site for more details, thanks.

UPDATE #3: Ok, a long, overdue update on this project. I’ve worked on the next version of this ideal that I encourage everyone to checkout and try for themselves. You can get it on Github, and the project’s name is lipsync. My goal is to make something that is trivial for anyone to setup and use, providing them a ‘Dropbox-like’ experience. This process will also help send large files and overcome file size limitations. As before I’ve focused on the backend, server side, part of the game to get that working, but would be happy to work with anyone that wanted to work on a GUI, or integrate this with existing projects, such as Sparkleshare, which seems to have a great GUI, but a backend that relies on things like Github for storage. So give it a look and remember, the more feedback the better; and as always don’t worry about offending me! Thanks.

UPDATE #2: There was a big influx of new hits/posts on this article last week thanks to Lifehacker Australia linking to it, plus they even came up with a pretty sweet logo. It’s very cool that so many are (still) interested in this project – and that’s what it has become; a project. I’ll be releasing code to setup a complete command-line Dropbox like implementation on Linux in about a week. Code will be hosted on github.com and I’m hoping it will spur others to work on cross platform front-ends to talk to it. So far the technology is there, I’m just using what others have built, it’s just a matter of hooking it all up! After all, why reinvent the wheel? (not that I could ;) ) Thanks again for all the comments and support!

UPDATE: Thanks to everyone who has contributed to this, and the Reddit thread, as it has provided some great ideas building off of my concept.  I’m starting to rethink about how we could have version control on top of things, and I’ll update things when I have more to share.  Also, does anyone have iFolder (thanks for the proper link salubrium) working?  It looks like you need SUSE Linux, which I don’t have access to, plus I know most Novell projects need a *ton* of Mono dependencies installed to have any of their stuff working, at least on the server side; but it sounds like they have Mac, Linux and Windows clients, which is encouraging.  While for my needs something a bit more ‘close to the bone’ (as below) might be better for the server side, having it be inter-operable with something like iFolder could provide a lot more functionality for others.

First off, if you haven’t tried Dropbox, you should check it out; sync all of your computers via the Dropbox servers, their basic free service gives you 2Gigs of space and works cross-platform (Windows, Mac, Linux).  I use it daily at home and work, and just having a live backup of my main data for my work workstation, my home netbook, and any other computer I need to login to is a huge win.  Plus, I have various ‘shared’ folders that distribute certain data to certain users that I’ve granted access to, this means work details can be updated and automatically distributed to the folks I want to review/use the data.  I recommend everyone try it out, and see how useful it is, it’s turned into a game changer for me.  So a few months ago they made headlines on supporting Linux as they released the client as open source. While this got hopes up for many, it was only the client that was open source, the server is still proprietary.  While slightly disappointing, this is fine, they’re a company trying to make money.  I don’t fault them for this, it’s just that a free, portable service like that would be a killer app.

Meanwhile at work I’m working on a solution to sync large data clusters online and the project manager described it as the need for ‘Dropbox on steroids’.  Before I had thought it was more complicated, but after thinking about it, I realized he was right.  Look, Dropbox is a great idea, but it obviously is just a melding of rsync, with something watching for file changes to initiate the sync, along with an easy to use front end.  From there I just started looking at ways this could work, and there are more than a few; here’s how I made it work.

Linux now includes inotify, which is a kernel subsystem that provides file system event notification.  From there all it took was to find an application that listens to inotify and then kicks off a command when it hears of a change.  I tried a few different applications like inocron, inosync and iwatch, before going with lsyncd.   While all of them could work, lsyncd seemed to be the most mature, simple to configure and fast.  Lsyncd uses inotify to watch a specified directory for any new, edited or removed files or directories, and then calls rsync to take care of business.  So let’s get started in making our own open source Dropbox clone with Debian GNU/Linux (lenny)

Ladies and gentlemen, start your engines servers!

First, you need 2 severs; one being the server and the other the client. (you could do this on one host if you wanted to see how it works for a proof of concept)

Install OpenSSH server

First you’ll need to install OpenSSH Server on the remote system:
apt-get install openssh-server

Configure SSH for Passwordless Logins

You’ll need to configure passwordless logins between the two hosts you want to use, this is how rsync will pass the files back and forth.  I’ve previously written a HOWTO on this topic, so we’ll crib from there.

First, generate a key:

ssh-keygen -t rsa

UPDATE: actually, it’s easier to do it this way

ssh-keygen -N '' -f ~/.ssh/id_dsa

(Enter)

You shouldn’t have a key stored there yet, but if you do it will prompt you now; make sure you overwrite it.

Enter passphrase (empty for no passphrase):

(Enter)

Enter same passphrase again:

(Enter)

We’re not using passphrases so logins can be automated, this should only be done for scripts or applications that need this functionality, it’s not for logging into servers lazily, and it should not be done as root!

Now, replace REMOTE_SERVER with the hostname or IP that you’re going to call when you SSH to it, and copy the key over to the server:

cat ~/.ssh/id_rsa.pub | ssh REMOTE_SERVER 'cat - >> ~/.ssh/authorized_keys2'

UPDATE: now you can use ssh-copy-id for this instead (hat tip briealeida)

ssh-copy-id REMOTE_SERVER

Set the permissions to a sane level:

ssh REMOTE_SERVER 'chmod 700 .ssh'

Lastly, give it a go to see if it worked:

ssh REMOTE_SERVER

You should be dropped to a prompt on the remote server. If not you may need to redo your .ssh directory, so on both servers:

`mv ~/.ssh ~/.ssh-old`

and goto 10

Install rsync and lsyncd

Next up is to install rsync and lsyncd.  First, rsync is simple, and could already be installed (you don’t need to run it as a server, just the client), make sure you have it with:

apt-get install rsync

Next is lsyncd.  There is no official Debian package yet, but it’s simple to build from source and install.  First off, if you don’t have build essentials you’ll need them, as well as libxml2-dev to build the lsyncd source.  Installing those is as simple as:

apt-get install libxml2-dev build-essential

Now we’ll get the lsyncd code (you can check for a newer version at http://lsyncd.googlecode.com) and build that:

wget http://lsyncd.googlecode.com/files/lsyncd-1.26.tar.gz
tar -zxf lsyncd-1.26.tar.gz
cd lsyncd-1.26
./configure
make; make install

This install does not install the configuration file, so we’ll do that manually now:

cp lsyncd.conf.xml /etc/

Configure lsyncd

Next up, we’ll edit the configuration file now located in /etc  The file is a simple, well documented XML file, and mine ended up like so – just be sure to change the source and target hosts and paths to work with your systems:

<lsyncd version="1.25">  
 <settings> 
 <logfile filename="/var/log/lsyncd"/> 
 <!--Specify the rsync (or other) binary to call--> 
 <binary filename="/usr/bin/rsync"/> 
 <!--uncomment to create a file containing pid of the daemon--> 
 <!--pidfile filename="/tmp/pid"/--> 
 <!--this specifies the arguments handled to the rsync (or other) binary. 
 option is the default literal. only '%r' will be replaced with r when recursive
 operation is wanted, d when not. exclude file will be replaced with -exclude-from FILE 
 source will be the source path to sync from destination will be the
 destination path to sync to --> 
 <callopts> 
 <option text="-lt%r"/> 
 <option text="--delete"/> 
 <exclude -file/> 
 <source /> 
 <destination /> 
 </callopts>  
 </settings> 
 <directory> 
 <source path="/var/www/sync_test"/> 
 <target path="desthost::module/"/> 
 <!-- or it can also be an absolute path for localhost 
 <target path="/absolute/path/to/target"> --> 
 </directory> 
</lsyncd> 

Launch lsyncd in debug for testing

We’re ready to give it a go, may as well run it in debug for fun and to learn how lsyncd does what it does:

lsyncd --conf /etc/lsyncd.conf.xml --debug

Watch for errors, if none are found, continue.

Add files and watch them sync

Now we just need to copy some files into this directory on the source box:

/var/www/sync_test

And again, watch for any errors on the screen, if these come back as a failed connection it’ll be an SSH/key issue, common, and not too difficult to solve. From here add some directories and watch how they’re queued up, and then take a look at them on the remote box: from this point out it “just works”. Now give it more to do by adding files and directories, and then the logging for errors while they sync. As it stands the system uses the source system as the preferred environment, so any files that change, or are added or removed, will be processed on the remote system. This is analogous to how Dropbox works, you can use multiple sources (your laptop, your desktop, etc) and their server serves as the remote system, keeping all the clients in line.

Conclusion

You should now have a basic, working Dropbox style setup for your own personal use. I had this running and used it to sync my netbook back to my home server, and then have my work desktop sync to my home server, so both the netbook and the desktop would stay in sync without me doing anything besides putting files in the specfied folder. For my week long test I ran a directory alongside my Dropbox directory just to see how they both acted, and I didn’t have any failures along the way.

Now we have is a simple Dropbox style app that is lightweight, with a functional back-end running rsync, which is a known stable app that will scale, and while it doesn’t provide the front-end and web view that Dropbox does, that could be an easy part for a UX developer to tackle. The cool thing is, we have a solution that works, and other options like the apps I described in the beginning, can be dropped in and replace the functionality lsyncd provides in case they can do something better. For now, I’m playing around with it to learn the ins and outs of the system to see how it will behave long term under a much larger store (50Gig to start) to keep in check. I will also work on better integrating this solution it into a working system, and update this tread with init scripts, reports, or maybe even a web view beyond just an index view from Apache or nginx. Ideally we could have a web front end that would intelligently report if a file is complete on the server, and if the file is completely mirrored on another server or client. P2P or Bitorrent would also be really cool to consider with this, and I’m sure there will be more applications for a setup like this once we’ve it around as a resource for a time. Can you think of more applications for this? Did you get it to work? Can you think of a better way to do this?

suggest
  • Anonymous
  • DamienMcKenna

    OSX has a similar file system change notification engine:http://developer.apple.com/mac/library/document…

  • http://fak3r.com fak3r

    Right you are, kqueue does the same job as inotify, and I thought about that when I was writing this; how this would work crossplatform. Not that I’m trying to reverse engineer Dropbox ;) but it’s just interesting to see how this could be handled on the different platforms. So we know how to do it in Linux, then both OS X (http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/kqueue.2.html) and FreeBSD (http://www.madison-gurkha.com/publications/kqueue/) can use kqueue to provide the same function as inotify does here. Then it’s just a matter of finding an interface to interact with it, and almost without looking I’ve found some python bindings out there (http://www.freebsdsoftware.org/devel/py-kqueue.html). So, I suspect making this work in a similar way on OS X and FreeBSD wouldn’t be hard at all, if someone wants to give it a go, let me know, I’m happy to help and “play along at home”. I actually miss working in FreeBSD, it’s a great OS.

    • http://fak3r.com fak3r

      Also, and even though I said cross-platform I failed to mention Windows (a theme of mine) but I have no idea how this could be done in Windows, though I’m sure there’s a way. Anyone want to give it a stab I’d be willing to help the best I could (which would probably involve using Cygwin which would sort of be cheating ;) )

      • Anonymous

        There’s a Windows API to do the same thing, I’d just rather use DropBox than deal with it ;-)

  • http://fak3r.com fak3r

    Right you are, kqueue does the same job as inotify, and I thought about that when I was writing this; how this would work crossplatform. Not that I'm trying to reverse engineer Dropbox ;) but it's just interesting to see how this could be handled on the different platforms. So we know how to do it in Linux, then both OS X (http://developer.apple.com/mac/library/document…) and FreeBSD (http://www.madison-gurkha.com/publications/kqueue/) can use kqueue to provide the same function as inotify does here. Then it's just a matter of finding an interface to interact with it, and almost without looking I've found some python bindings out there (http://www.freebsdsoftware.org/devel/py-kqueue….). So, I suspect making this work in a similar way on OS X and FreeBSD wouldn't be hard at all, if someone wants to give it a go, let me know, I'm happy to help and “play along at home”. I actually miss working in FreeBSD, it's a great OS.

  • http://fak3r.com fak3r

    Also, and even though I said cross-platform I failed to mention Windows (a theme of mine) but I have no idea how this could be done in Windows, though I'm sure there's a way. Anyone want to give it a stab I'd be willing to help the best I could (which would probably involve using Cygwin which would sort of be cheating ;) )

  • DamienMcKenna

    There's a Windows API to do the same thing, I'd just rather use DropBox than deal with it ;-)

  • David Graves

    This is very interesting, but what about previous file versioning like what dropbox has?

    • http://fak3r.com fak3r

      Good point, this (sourcebox?) does not address that at all, I was talking last week about the idea of making the mirrored directory be a git repository, and have any revisions handled this way along with the adds and deletes, but I haven’t worked any more on it.

      • http://mikechelen.com Mike Chelen

        Git is slow with large files, instead use filesystem support for versioning or copy-on-write with ZFS, Btrfs, or ext3cow.

        • http://fak3r.com fak3r

          Whoa, I had never heard of ext3cow, that’s pretty cool, thanks for the tip!

        • Egbert Pot

          Hi fak3r and Mike Chelen. Thanks for this post and all the usefull information in the replies!

          I’ve used lsyncd and ZFS-FUSE together to create my own dropbox clone on Ubuntu 10.04. I’ve writen a quick how-to bellow. All the configuration files have been posted to pastebin for easy copy-paste

          /etc/lsyncd.conf.xml : http://pastebin.com/4z0Xg2Pk
          /usr/local/sbin/lsyncd-execute : http://pastebin.com/y6jQ3EVH

          Install ZFS-FUSE
          # aptitude install zfs-fuse

          Create a file that can be used as a ZFS pool

          # dd if=/dev/zero of=/opt/zfsbackupstorage/zfsbackupstorage001 bs=1k count=1000000

          This wil make a 100mb empty file

          Create a ZFS pool with the name zfsbackup

          # zpool create zfsbackup /opt/zfsbackupstorage/zfsbackupstorage001

          Set the mountpoint

          # zfs set mountpoint=/opt/backup

          Check if the ZFS filesystem has been mounted

          # df -h

          Install lsyncd

          First make sure rsync is installed:

          # apt-get install rsync

          Install build dependencies

          # apt-get install libxml2-dev build-essential

          Download the source code

          # cd /usr/local/src

          # wget http://lsyncd.googlecode.com/files/lsyncd-1.37.tar.gz

          # tar -xzvf lsyncd-1.37.tar.gz

          # cd lsyncd-1.37

          # ./configure

          # make

          # make install

          Note: It’s a very small programm, it might look like nothing is happening

          This install does not install the configuration file, so we’ll do that manually now:

          #cp lsyncd.conf.xml /etc/

          Edit /etc/lsyncd.conf.xml

          See http://pastebin.com/4z0Xg2Pk

          Create the /usr/local/sbin/lsyncd-execute file

          # nano /usr/local/sbin/lsyncd-execute

          See http://pastebin.com/y6jQ3EVH

          Give the /usr/local/sbin/lsyncd-execute script permissions to be executed

          # chmod +x /usr/local/sbin/lsyncd-execute

          Launch lsyncd in debug for testing

          We’re ready to give it a go, may as well run it in debug for fun and to learn how lsyncd does what it does:

          # lsyncd –conf /etc/lsyncd.conf.xml –debug

          Watch the log for errors, if none are found, launch lsyncd. You can also add lsyncd to /etc/rc.local so it will be started every time your server / desktop starts

          Good luck!

          Best, Egbert

          • http://fak3r.com fak3r

            Great writeup, I’ve been meaning to try out ZFS on Linux via FUSE, formerly ran ZFS on FreeNAS (FreeBSD). I’ve built a new RAID1 array from 2 1TB drives with ext4 on them. They’re the ‘green’ drives that use less power, and they do run cooler than the others, so having redundant storage using less power (considering I was running them in a separate server before it’s really less) was my goal, but having the advantages of ZFS would be the next step. Thanks for the detail!

          • Mike

            May I suggest NILFS for continuous snapshotting? May be simpler to setup than ZFS.

            http://www.nilfs.org/en/

  • David Graves

    This is very interesting, but what about previous file versioning like what dropbox has?

  • http://fak3r.com fak3r

    Good point, this (sourcebox?) does not address that at all, I was talking last week about the idea of making the mirrored directory be a git repository, and have any revisions handled this way along with the adds and deletes, but I haven't worked any more on it.

    • http://mikechelen.com Mike Chelen

      Git is slow with large files, instead use filesystem support for versioning or copy-on-write with ZFS, Btrfs, or ext3cow.

      • http://fak3r.com fak3r

        Whoa, I had never heard of ext3cow, that’s pretty cool, thanks for the tip!

  • http://www.facebook.com/profile.php?id=657835806 Anonymous

    File versioning could be done with rdiff-backup or even git. A better way however would be to use an opensolaris server with ZFS and create a snapshot of the filesystem with each revision. This moves all the processing to the server and the client can simply upload the files.

    • http://fak3r.com fak3r

      Funny, I looked at rdiff-backup first, but using git seems like a great idea. As for ZFS, heck ya, I know FreeBSD supports it too, either of those would just rock – I need to play around with it to better unstand it, and OpenSolaris may be the best/next path.. I’m still bummed they dropped ZFS support from Snow Leopard, that will be a killer feature if OS X gets that. Great suggestions, thanks!

  • http://www.facebook.com/profile.php?id=657835806 facebook-657835806

    File versioning could be done with rdiff-backup or even git. A better way however would be to use an opensolaris server with ZFS and create a snapshot of the filesystem with each revision. This moves all the processing to the server and the client can simply upload the files.

  • furicle

    But one of the big benefits to Dropbox is multiple clients. You can make a change on any client and it propagates out to all the clients – not just master slave like this is…

    • http://fak3r.com fak3r

      True, I haven’t tested it this way yet, but with Rsync you can set it up to function as a ‘hub’ – with the center being the server, and the clients all preferring whatever new things the server has. Someone tell me if I’m right on this, but yeah, it’s definitely a good use case that has to work. I’ll look at it closer.

  • furicle

    But one of the big benefits to Dropbox is multiple clients. You can make a change on any client and it propagates out to all the clients – not just master slave like this is…

  • Anonymous

    Novell had a product called ifolder, which they open sourced and I used it for about 2 years but it was a badly managed project and clients ended up not building on it. When Dropbox came out, I swapped to that but it seems that somebody has picked up on ifolder again and it’s an active project. Ifolder does exactly what dropbox does including versioning etc. It was a mono-based project and written in C#, so that might turn a few people off it but check it out. http://www.kablink.org/

    • http://fak3r.com fak3r

      I did find this while I was doing research, but it looked partially abandoned. Additionally I’ve worked on a Novell sponored project before, Hula, and was not happy with how the project was run, and eventually dropped/sold. Still, if this code works and is being developed I think it looks very promising, plus with the web interface would be a boon to others who don’t want to get their hands dirty ;) Will be installing this soon and trying it out.

      • Anonymous

        You’re right about the whole heap of mono dependencies for ifolder. It was a pain in the ass to build and get working and it got to the stage that it became all too hard for me to work with. One thing I really wish Dropbox would do is to detect other Dropbox clients on the local network and synch directly to them and one of them synch to the server. As it is now, I think Dropbox synchs to the server and then syncs the same set of files back to each of the clients in the same network. This is unnecessary web traffic.

        I forgot that I actually went from using Ifolder to Mindquarry, which was a Java based oss project using SVN as it’s source control. It also had a few other bits attached ie: A wiki & tasks. Check-ins were manual but the timeline versioning system was great. The company couldn’t get it’s 2nd round of funing and the founders had to go and get jobs. I haven’t used it or try and build it since then. http://code.google.com/p/mindquarry/

        • http://www.stoeck.it/ Jakob Stoeck

          “I really wish Dropbox would do is to detect other Dropbox clients on the local network and synch directly”

          Dropbox actually does this (I think two months ago it was in the beta version)

          • http://fak3r.com fak3r

            I believe this is a coming feature, I could really see it being useful to share documents in an office; we already do this, but don’t have anything larger than word/powerpoint docs, so those transfer up and back pretty quickly. Still, a bon-jour type service would be very cool, make it a ‘read-only’ server for local clients that aren’t hooked into Dropbox for example

          • http://fak3r.com fak3r

            Check it out, new version of Dropbox box includes:- LAN syncI think that’s what we’re talking about here.http://forums.dropbox.com/topic.php?id=15519&re…

  • salubrium

    Novell had a product called ifolder, which they open sourced and I used it for about 2 years but it was a badly managed project and clients ended up not building on it. When Dropbox came out, I swapped to that but it seems that somebody has picked up on ifolder again and it's an active project. Ifolder does exactly what dropbox does including versioning etc. It was a mono-based project and written in C#, so that might turn a few people off it but check it out. http://www.kablink.org/

  • http://blog.itsmine.co.uk/ Rob

    In theory this looks good and I haven’t tried it out yet. Would be nice to have version control in there too, so adding an SVN commit command onto the hook wouldn’t be a bad idea if you wanted it – this also might help solve the problem of conflict resolution.
    Also, there’s no Windows client which is a slight shame – although I’m sure some stuff could be done with cygwin etc.

  • http://rosensharma.wordpress.com/ Rosen Sharma

    Do any of the linux tools do the equivalent of dedup while transferring the blocks? I think dropbox does: http://sharevm.wordpress.com/2009/09/17/dropbox-dedup-only-in-the-cloud/

  • http://blog.itsmine.co.uk/ Rob

    In theory this looks good and I haven't tried it out yet. Would be nice to have version control in there too, so adding an SVN commit command onto the hook wouldn't be a bad idea if you wanted it – this also might help solve the problem of conflict resolution.Also, there's no Windows client which is a slight shame – although I'm sure some stuff could be done with cygwin etc.

  • http://rosensharma.wordpress.com/ Rosen Sharma

    Do any of the linux tools do the equivalent of dedup while transferring the blocks? I think dropbox does: http://sharevm.wordpress.com/2009/09/17/dropbox…

  • http://www.gilestro.tk/ gg

    This is pretty neat. Good idea, really. I hope this is going to grow and I’d be happy to help out in case you want to start an official project out of it.

  • http://www.gilestro.tk/ gg

    This is pretty neat. Good idea, really. I hope this is going to grow and I'd be happy to help out in case you want to start an official project out of it.

  • http://fak3r.com fak3r

    Funny, I looked at rdiff-backup first, but using git seems like a great idea. As for ZFS, heck ya, I know FreeBSD supports it too, either of those would just rock – I need to play around with it to better unstand it, and OpenSolaris may be the best/next path.. I'm still bummed they dropped ZFS support from Snow Leopard, that will be a killer feature if OS X gets that. Great suggestions, thanks!

  • http://fak3r.com fak3r

    True, I haven't tested it this way yet, but with Rsync you can set it up to function as a 'hub' – wit the center being the server, and the clients all preferring whatever new things the server has. Someone tell me if I'm right on this, but yeah, it's definitely a good use case that has to work. I'll look at it closer.

  • http://fak3r.com fak3r

    I did find this while I was doing research, but it looked partially abandoned. Additionally I've worked on a Novell sponored project before, Hula, and was not happy with how the project was run, and eventually dropped/sold. Still, if this code works and is being developed I think it looks very promising, plus with the web interface would be a boon to others who don't want to get their hands dirty ;) Will be installing this soon and trying it out.

  • salubrium

    You're right about the whole heap of mono dependencies for ifolder. It was a pain in the ass to build and get working and it got to the stage that it became all too hard for me to work with. One thing I really wish Dropbox would do is to detect other Dropbox clients on the local network and synch directly to them and one of them synch to the server. As it is now, I think Dropbox synchs to the server and then syncs the same set of files back to each of the clients in the same network. This is unnecessary web traffic.I forgot that I actually went from using Ifolder to Mindquarry, which was a Java based oss project using SVN as it's source control. It also had a few other bits attached ie: A wiki & tasks. Check-ins were manual but the timeline versioning system was great. The company couldn't get it's 2nd round of funing and the founders had to go and get jobs. I haven't used it or try and build it since then. http://code.google.com/p/mindquarry/

  • Jordan

    Hey there,

    just wanted to let you know there is an ubuntu installation guide for iFolder which may be of help. There’s quite a few mono dependancies though!

    https://help.ubuntu.com/community/iFolderEnterpriseServer

  • Jordan

    Hey there,just wanted to let you know there is an ubuntu installation guide for iFolder which may be of help. There's quite a few mono dependancies though!https://help.ubuntu.com/community/iFolderEnterp…

  • http://www.zencollegelife.com Ibrahim | ZenCollegeLife

    Cool Idea, although I didn’t see if this was possible to set up using windows on the server.

    • http://fak3r.com fak3r

      No, since my way is using inotify which is built into the Linux kernel. However, as mentioned above, kqueue provides the same functionality under OS X and FreeBSD, so it’s likely there’s a similar utility in Windows. After you discover that, maybe we can figure out a way to utilize rsync in a similar fashion. I suspect installing cygwin (http://sources.redhat.com/cygwin/) would make this task far easier, but I’m sure a Windows guru could do this within Windows only, I don’t really know.

    • http://poojanwagh.pip.verisignlabs.com/ Poojan Wagh

      See FreeFileSync: http://freefilesync.sourceforge.net/screenshots.php
      It won’t do remote sync directly (AFAIK) but it includes a Windows utility call RealTimeSync which can watch folders and execute a command when something changes.

      • http://fak3r.com fak3r

        While it looks pretty GUI heavy, that looks like a good Windows solution, thanks for posting! Trying to see how this could work (directly) with a setup on Linux, certainly it could sync to it…

  • http://www.zencollegelife.com Ibrahim | ZenCollegeLife

    Cool Idea, although I didn't see if this was possible to set up using windows on the server.

    • http://poojanwagh.pip.verisignlabs.com/ Poojan Wagh

      See FreeFileSync: http://freefilesync.sourceforge.net/screenshots.php
      It won’t do remote sync directly (AFAIK) but it includes a Windows utility call RealTimeSync which can watch folders and execute a command when something changes.

      • http://fak3r.com fak3r

        While it looks pretty GUI heavy, that looks like a good Windows solution, thanks for posting! Trying to see how this could work (directly) with a setup on Linux, certainly it could sync to it…

  • Pingback: Nicolas (nambuls) 's status on Tuesday, 06-Oct-09 04:32:50 UTC - Identi.ca

  • http://fak3r.com fak3r

    No, since my way is using inotify which is built into the Linux kernel. However, as mentioned above, kqueue provides the same functionality under OS X and FreeBSD, so it's likely there's a similar utility in Windows. After you discover that, maybe we can figure out a way to utilize rsync in a similar fashion. I suspect installing cygwin (http://sources.redhat.com/cygwin/) would make this task far easier, but I'm sure a Windows guru could do this within Windows only, I don't really know.

  • Jordan
  • Jordan

  • Pingback: Carles Bellver (carles) 's status on Saturday, 24-Oct-09 10:26:37 UTC - Identi.ca

  • Pingback: Markus Merz (m1) 's status on Tuesday, 27-Oct-09 16:40:18 UTC - Identi.ca

  • http://sankt-georg.info/ Markus Merz

    Didn’t check the whole comment thread but versions could be realised by taking a look at rsnapshot. Just an idea…

    • http://fak3r.com fak3r

      Yes, I still need to investigate how rsnapshot could fit into this, thanks for reminding me.

      • http://sankt-georg.info/ Markus Merz

        As git seems like a complicated canon to me rsnapshot can simply be used as an rsync shell/replacement whenever you fire off your sync. rsnapshot simply wraps around rsync to create snapshots instead of simple syncs. Every snapshot is complete via hard links.

        • http://fak3r.com fak3r

          Cool, this sounds ideal, plus lsyncd is sort of a wrapper, or at least a watcher, that calls rsync anytime it sees action in a watched directory. We’ll now have the disk space to store versioning info, so this seems like a good fit. Thanks

          • http://sankt-georg.info/ Markus Merz

            Push a link to this article into the rsnapshot mailinglist and ask for the best integration. Or better: Find out how to replace the rsync calls in lsyncd with the appropriate rsnapshot calls which then will fire the rsync calls.

  • http://sankt-georg.info/ Markus Merz

    Didn't check the whole comment thread but versions could be realised by taking a look at rsnapshot. Just an idea…

  • Pingback: Aktuelle Links (gespeichert vom 26.10.2009 bis zum 28.10.2009) « Der Webanhalter

  • Pingback: links for 2009-10-27 « Donghai Ma

  • georgygoshin

    Didn't you looked to Novell iFolder? It's cross platform and make the synchronization but does not plays with versions.

  • Anonymous

    Didn’t you looked to Novell iFolder? It’s cross platform and make the synchronization but does not plays with versions.

  • Athas

    Is there a WebDAV client that handles caching and synchronization of mounted WebDAV servers? If so, setting up subversion as an HTTP server and enabling autoversioning would seem to fix almost all of the problems.It wouldn't be true version control, because the last change would be committed without reporting potential conflicts, but it's pretty close.

  • Athas

    Is there a WebDAV client that handles caching and synchronization of mounted WebDAV servers? If so, setting up subversion as an HTTP server and enabling autoversioning would seem to fix almost all of the problems.

    It wouldn’t be true version control, because the last change would be committed without reporting potential conflicts, but it’s pretty close.

  • Jakob Stoeck

    “I really wish Dropbox would do is to detect other Dropbox clients on the local network and synch directly”Dropbox actually does this (I think two months ago it was in the beta version)

  • Adam

    Did you look into the multiple client idea? If I made a change on one client and it gets synced to the server, how would my other client know about it? I'd like to use this to sync some files between my iMac with my MacBook, so any ideas of how to set it up would be great!

  • Adam

    Did you look into the multiple client idea? If I made a change on one client and it gets synced to the server, how would my other client know about it? I’d like to use this to sync some files between my iMac with my MacBook, so any ideas of how to set it up would be great!

    • http://fak3r.com fak3r

      The other client would know about it when it rsync’d to the server and found newer things that it didn’t have. While this wouldn’t happen automatically, it would happen the next time the other client had a file change locally, or if it called it via cron every so often. Not ideal, but it would work. The rsyc command just always targets the “server” as the authoritative copy unless something on the client has changed earlier.

  • http://fak3r.com fak3r

    The other client would know about it when it rsync'd to the server and found newer things that it didn't have. While this wouldn't happen automatically, it would happen the next time the other client had a file change locally, or if it called it via cron every so often. Not ideal, but it would work. The rsyc command just always targets the “server” as the authoritative copy unless something on the client has changed earlier.

  • http://fak3r.com fak3r

    I believe this is a coming feature, I could really see it being useful to share documents in an office; we already do this, but don't have anything larger than word/powerpoint docs, so those transfer up and back pretty quickly. Still, a bon-jour type service would be very cool, make it a 'read-only' server for local clients that aren't hooked into Dropbox for example

  • http://fak3r.com fak3r

    Check it out, new version of Dropbox box includes:- LAN syncI think that's what we're talking about here.http://forums.dropbox.com/topic.php?id=15519&re…

  • http://fak3r.com fak3r

    Got some coverage at Linux Magazine: http://www.linux-magazine.com/Online/Blogs/Prod…

  • http://fak3r.com fak3r
  • akaihola

    A few notes:- Unison provides two-way synchronization and has both a command-line and a GTK user interface- there are issues with rsync on Cygwin; it's difficult to find exact versions which work flawlessly- git won't be good for very large datasets

    • cr4ckwh0re

      I also think that unison does exactly fullfill the requirements …

  • Anonymous

    A few notes:

    - Unison provides two-way synchronization and has both a command-line and a GTK user interface

    - there are issues with rsync on Cygwin; it’s difficult to find exact versions which work flawlessly

    - git won’t be good for very large datasets

    • cr4ckwh0re

      I also think that unison does exactly fullfill the requirements …

  • http://fak3r.com fak3r

    Yes, I still need to investigate how rsnapshot could fit into this, thanks for reminding me.

  • http://sankt-georg.info/ Markus Merz

    As git seems like a complicated canon to me rsnapshot can simply be used as an rsync shell/replacement whenever you fire off your sync. rsnapshot simply wraps around rsync to create snapshots instead of simple syncs. Every snapshot is complete via hard links.

  • http://fak3r.com fak3r

    Cool, this sounds ideal, plus lsyncd is sort of a wrapper, or at least a watcher, that calls rsync anytime it sees action in a watched directory. We'll now have the disk space to store versioning info, so this seems like a good fit. Thanks

  • http://sankt-georg.info/ Markus Merz

    Push a link to this article into the rsnapshot mailinglist and ask for the best integration. Or better: Find out how to replace the rsync calls in lsyncd with the appropriate rsnapshot calls which then will fire the rsync calls.

  • Pingback: RT @chrisfreeland @rdmpage @Jim_Croft @anthonygoddard … — biotweets.org

  • Pingback: RT @rdmpage: @Jim_Croft it’s what … — biotweets.org

  • gramps

    Thanks for your research on this. I have your Dropbox Clone now working with the data from my laptop going to my server. I was this data to also exist on my netbook. So if I set the netbook to use the same directories on the server as laptop when run will it pickup the data on the server and send it to my netbook.

  • gramps

    Thanks for your research on this. I have your Dropbox Clone now working with the data from my laptop going to my server. I was this data to also exist on my netbook. So if I set the netbook to use the same directories on the server as laptop when run will it pickup the data on the server and send it to my netbook.

  • gramps

    Thanks for your research on this. I have your Dropbox Clone now working with the data from my laptop going to my server. I was this data to also exist on my netbook. So if I set the netbook to use the same directories on the server as laptop when run will it pickup the data on the server and send it to my netbook.

  • Pingback: Cloud based backup choices | Nick Hammond

  • Pingback: Meanwhile in Gotham City ← pseudopost.org

  • Pingback: robert denton – a day » Blog Archive » fak3r – HOWTO build your own open source Dropbox clone

  • http://twitter.com/jandd Jan Dittberner

    I just ITPed lsyncd, see http://bugs.debian.org/cgi-bin/bugreport.cgi?bu… that will make your setup even easier with Debian.

  • http://twitter.com/jandd Jan Dittberner

    git and incron on the server side may be an option for your versioning idea, see http://andrew.mcmillan.net.nz/blog/using_incron…

  • http://twitter.com/jandd Jan Dittberner

    I just ITPed lsyncd, see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=570296 that will make your setup even easier with Debian.

  • http://www.microstockil.com/ yaniv hanina

    Hii am looking for someone to install and configure, i saw you wrote about this solution. we are a company based on remote working we would like to have Isyncd or any kind of “dropbox” clone installed and configured on our servers to be used by our workers and out side people related to us. i would appriciate if someone could get back to me regard a price quote and terms best regards, Yaniv yanivhanina@gmail.com

  • http://www.microstockil.com/ yaniv hanina

    Hi
    i am looking for someone to install and configure, i saw you wrote about this solution.

    we are a company based on remote working
    we would like to have Isyncd or any kind of “dropbox” clone installed and configured on our servers
    to be used by our workers and out side people related to us.

    i would appriciate if someone could get back to me
    regard a price quote and terms

    best regards,
    Yaniv
    yanivhanina@gmail.com

  • pantro

    apt-get install build-essential not build-essentials

  • Anonymous

    apt-get install build-essential not build-essentials

  • Arthur Czuma

    Sorry to leave an unrelated comment, but I couldn't figure out your email; I'm embarrassed to say I spent a good twenty minutes on it and barely got anywhere.I'm interested in advertising on your site, specifically in this post. Let me know if you're interested.

  • Arthur Czuma

    Sorry to leave an unrelated comment, but I couldn’t figure out your email; I’m embarrassed to say I spent a good twenty minutes on it and barely got anywhere.

    I’m interested in advertising on your site, specifically in this post. Let me know if you’re interested.

    • http://fak3r.com fak3r

      @Aurthur – you can reach me at fak3r at fak3r dot com – sorry for the trouble, it was more for fun than anything else.

  • chongopants

    Hey man awesome how – to, I am trying to do the same thing but on a remote server to win/mac. Most of us are on mac, I dunno what your using, but once you set this up how do you setup the clients? Right now I am rsyncing everything up, but once done I wanna put it all on there. I have found the problem with other services like webdav, etc, they tend to freak out with large files, or large amounts of files…. and everything else is lacking in speed. All help is appreciated.

  • chongopants

    oh yeah lsync not available in url given btw…..

  • Anonymous

    Hey man awesome how – to, I am trying to do the same thing but on a remote server to win/mac. Most of us are on mac, I dunno what your using, but once you set this up how do you setup the clients? Right now I am rsyncing everything up, but once done I wanna put it all on there. I have found the problem with other services like webdav, etc, they tend to freak out with large files, or large amounts of files…. and everything else is lacking in speed. All help is appreciated.

  • Anonymous

    oh yeah lsync not available in url given btw…..

  • http://oliver.treend.myopenid.com/ Ollie Treend

    I too would be very interested in some form of a Windows compatible client.Have there been any advances in this area?

    • http://fak3r.com fak3r

      @Ollie – No, sorry to say there has not been, however I’m almost ready to release my formalized setup script that works with Linux. My hope is that after that folks will think of ways Windows apps can talk to/sync with it. The rsync part is no problem in Windows, there’s just a need for some sort of ‘watcher’ that can tell it when to kick off the sync…outside of that there’s not a lot of difference. Of course a GUI would be another step.

      • julien23

        Open source FreeFileSync comes with a watcher called RealTimeSync that can launch what ever command-line … might be a trick

  • http://oliver.treend.myopenid.com/ Ollie Treend

    I too would be very interested in some form of a Windows compatible client.
    Have there been any advances in this area?

    • http://fak3r.com fak3r

      @Ollie – No, sorry to say there has not been, however I’m almost ready to release my formalized setup script that works with Linux. My hope is that after that folks will think of ways Windows apps can talk to/sync with it. The rsync part is no problem in Windows, there’s just a need for some sort of ‘watcher’ that can tell it when to kick off the sync…outside of that there’s not a lot of difference. Of course a GUI would be another step.

      • julien23

        Open source FreeFileSync comes with a watcher called RealTimeSync that can launch what ever command-line … might be a trick

      • John

        any status on this setup script?

        • http://fak3r.com fak3r

          Yes, I’ve posted an update at the top of this thread, we’re working on lipsync over on github, give it a look, I want to make it something more general that can be a ‘drop in’ app to make this kind of thing work. https://github.com/philcryer/lipsync

      • Johngillow

        How can I sync my files with an NFS server???

        • http://fak3r.com fak3r

          Just mount the NFS share, then instead of using SSH just use rsync by itself to sync your normal filesystem to the NFS mounted one. (so look at the directions above, but just ignore any SSH related steps (which are most of them!))

      • Johngillow

        How can I sync my files with an NFS server???

      • dawmail333

        There’s definitely a way to do file watching: .Net has a component to watch a directory.

        • http://fak3r.com fak3r

          Interesting, so if we could get lsyncd2 running under cgywin, could this .NET library watch for changes and issue an rsync command?

          • CenterOrbit

            I believe rsync has been ported to windows, in this case if one was to port lsync/rsync to .NET and integrate the watch changes, that would most likely be the strongest solution.

          • http://www.ldngames.co.uk/ LDN Games

            You could just use Cygwin … that would give you the full OpenSSH suite.

  • Pingback: How to build your own DropBox personal clone | Ask Superuser

  • tomi

    my ssh key is still asked by lsyncd (?) even if i ssh XXX@YYYY works without any keys

    • http://fak3r.com fak3r

      run `hostname` and see what that says; make sure you use the same hostname when you generate the key, the one that ssh-keygen sees/uses needs to match. If that fails, try again, but have it create a new file like known_hosts3 (or `mv ~/.ssh ~/.ssh-old` and start again. Let me know, sharing SSH keys is tricky but once it works, it makes a lot of things like this possible.

  • tomi

    my ssh key is still asked by lsyncd (?) even if i ssh XXX@YYYY works without any keys

  • J Withers

    The tweets widget on the left side of your site actually blocks text while you are trying to read it. At least on chrome. Bad usability. Makes article unreadable. Seems like you might have had something interesting to say here. Too bad.

    • http://fak3r.com fak3r

      Can you tell me what resolution you’re running? For me it’s way over to the side, no where near the text, but I guess on lower resolution it could be an issue. Thanks for letting me know!

      • J Withers

        1024 x 768, ubuntu, chrome 5.0.x, firefox 3.0.x

        Good luck with it.

      • http://www.unixsysadmin.org briealeida

        To that, I’m using Firefox 3.6, Ubuntu, resolution 1600×1200 and the tweets widgets pops right over the content. If it was to the right, instead, I don’t think it’d still be an issue (but I didn’t even think to open it until I read this so…).

  • J Withers

    The tweets widget on the left side of your site actually blocks text while you are trying to read it. At least on chrome. Bad usability. Makes article unreadable. Seems like you might have had something interesting to say here. Too bad.

  • Pingback: KafeKafe » Build Your Own Dropbox With lsyncd

  • http://fak3r.com fak3r

    Can you tell me what resolution you're running? For me it's way over to the side, no where near the text, but I guess on lower resolution it could be an issue. Thanks for letting me know!

  • http://fak3r.com fak3r

    run `hostname` and see what that says; make sure you use the same hostname when you generate the key, the one that ssh-keygen sees/uses needs to match. If that fails, try again, but have it create a new file like known_hosts3 (or `mv ~/.ssh ~/.ssh-old` and start again. Let me know, sharing SSH keys is tricky but once it works, it makes a lot of things like this possible.

  • http://fak3r.com fak3r

    @Aurthur – you can reach me at fak3r at fak3r dot com – sorry for the trouble, it was more for fun than anything else.

  • J Withers

    1024 x 768, ubuntu, chrome 5.0.x, firefox 3.0.xGood luck with it.

  • http://www.unixsysadmin.org briealeida

    You can use the ‘ssh-copy-id’ command to transfer the necessary files for passwordless login instead.

    –
    Brie

    • http://fak3r.com fak3r

      Thanks, I just recently came across this, and if that fails to find an id (an error that I got when I tried with a non-privileged user) you can use the -i switch, so `ssh-copy-id -i ~/.ssh/id_rsa.pub REMOTE_HOST`

      And by the way Brie, you have a great site, was checking out many of the articles today, I’m writing up a little BASH script to post system stats to Twitter from some of your ideas. Will post it here in a few…

      • http://www.unixsysadmin.org briealeida

        Thanks! I really appreciate that and I’ve added your feed to Akregator so I’ll be keeping an eye out for that. :) !

        –
        Brie

  • http://www.unixsysadmin.org briealeida

    You can use the 'ssh-copy-id' command to transfer the necessary files for passwordless login instead. –Brie

  • http://fak3r.com fak3r

    Thanks, I just recently came across this, and if that fails to find an id (an error that I got when I tried with a non-privileged user) you can use the -i switch, so `ssh-copy-id -i ~/.ssh/id_rsa.pub REMOTE_HOST`And by the way Brie, you have a great site, was checking out many of the articles today, I'm writing up a little BASH script to post system stats to Twitter from some of your ideas. Will post it here in a few…

  • http://www.unixsysadmin.org briealeida

    To that, I'm using Firefox 3.6, Ubuntu, resolution 1600×1200 and the tweets widgets pops right over the content. If it was to the right, instead, I don't think it'd still be an issue (but I didn't even think to open it until I read this so…).

  • http://www.unixsysadmin.org briealeida

    Thanks! I really appreciate that and I've added your feed to Akregator so I'll be keeping an eye out for that. :) !–Brie

  • http://twitter.com/reneknuvers Rene Knuvers

    Hi, I like your idea. I’m however trying something different myself: I’ll setup a WEBDAV shared folder (hence, available online), connect to that using a standard method (windows: mount the share), and use a tool like ‘FreeFileSync’ to actively monitor a folder outside the webdav to sync that with the online folder. When an internet connection is not available, I hope FreeFileSync will be smart enough to stop replicating. Basically this works with your implentation of inotify just the same, but then on Linux. The only thing is that I mount the remote folder rather than rsyncing into it.

    just my 5ct

  • http://twitter.com/reneknuvers Rene Knuvers

    Hi, I like your idea. I'm however trying something different myself: I'll setup a WEBDAV shared folder (hence, available online), connect to that using a standard method (windows: mount the share), and use a tool like 'FreeFileSync' to actively monitor a folder outside the webdav to sync that with the online folder. When an internet connection is not available, I hope FreeFileSync will be smart enough to stop replicating. Basically this works with your implentation of inotify just the same, but then on Linux. The only thing is that I mount the remote folder rather than rsyncing into it.just my 5ct

  • http://sanjayayogi.myopenid.com/ Sanjaya Yogi

    I have working iFolder and SImias scripts that build working client and servers for Ubuntu. The server is working on Ubuntu 9.04 and the iFolder client on Ubuntu 9.10. I tried your approach with the scripts running on the server to do backups for the Simias server, and the scripts on the client to keep moving changes into the iFolder for backups to the server end. I am also thinking about the version control issues. I think another layer beyond the Simias server…
    2010-04-02
    Check out the scripts and join the group at:
    http://groups.google.com/group/ifolder-ubuntu-debian-dev

  • http://sanjayayogi.myopenid.com/ Sanjaya Yogi

    I have working iFolder and SImias scripts that build working client and servers for Ubuntu. The server is working on Ubuntu 9.04 and the iFolder client on Ubuntu 9.10. I tried your approach with the scripts running on the server to do backups for the Simias server, and the scripts on the client to keep moving changes into the iFolder for backups to the server end. I am also thinking about the version control issues. I think another layer beyond the Simias server…2010-04-02Check out the scripts and join the group at:http://groups.google.com/group/ifolder-ubuntu-d…

  • phil_s

    For version control, you could play around with rsnapshot in place of rsync. I believe* rsnapshot uses rsync as its underlying engine so you don’t lose any of rsync’s proven goodness.

  • phil_s

    For version control, you could play around with rsnapshot in place of rsync. I believe* rsnapshot uses rsync as its underlying engine so you don't lose any of rsync's proven goodness.

  • Pingback: 3komma14 Notes » Blog Archive » build your own dropbox clone

  • http://fak3r.com fak3r

    I just came across pylsyncd, a python activated version of lsyncd, which brags that,

    “Pylsyncd is a python implementation similar to lsyncd that uses rsync to synchronize local directories with several remote machines running rsyncd. Pylsyncd monitors recursively a set of directories using pyinotify, a pure Python module used for monitoring filesystems changes that relies on inotify.

    The main advantage of pylsyncd against lsyncd is that it uses message queues in order to synchronize in a parallel way several destination servers, saving up time when it is required to have more than one destination. It has been tested in heavy loaded environments.”
    http://iaslanidis.github.com/pylsyncd/

    Worth testing, sounds like they’ve thought up some nice additions.

  • http://fak3r.com fak3r

    I just came across pylsyncd, a python activated version of lsyncd, which brags that,”Pylsyncd is a python implementation similar to lsyncd that uses rsync to synchronize local directories with several remote machines running rsyncd. Pylsyncd monitors recursively a set of directories using pyinotify, a pure Python module used for monitoring filesystems changes that relies on inotify.The main advantage of pylsyncd against lsyncd is that it uses message queues in order to synchronize in a parallel way several destination servers, saving up time when it is required to have more than one destination. It has been tested in heavy loaded environments.”http://iaslanidis.github.com/pylsyncd/Worth testing, sounds like they've thought up some nice additions.

  • http://fak3r.com fak3r

    I just came across pylsyncd, a python activated version of lsyncd, which brags that,”Pylsyncd is a python implementation similar to lsyncd that uses rsync to synchronize local directories with several remote machines running rsyncd. Pylsyncd monitors recursively a set of directories using pyinotify, a pure Python module used for monitoring filesystems changes that relies on inotify.The main advantage of pylsyncd against lsyncd is that it uses message queues in order to synchronize in a parallel way several destination servers, saving up time when it is required to have more than one destination. It has been tested in heavy loaded environments.”http://iaslanidis.github.com/pylsyncd/Worth testing, sounds like they've thought up some nice additions.

  • Pingback: Re: Afternoon sellout notes

  • Pingback: SparkleShare, cómo no se me ocurrió antes. O sí. | Incognitosis

  • Jaykumar1

    Hi Fak3r,

    Thank you for posting an excellent guide on creating your own dropbox clone. Though most parts are way over my head, I think i get the general idea.

    Would you be interested in taking on some freelance work and putting this system up on my server. We can figure out a price and start. Please let me know what is the best way to contact you because I could not find a contact form or your email address anywhere.

  • Jaykumar1

    Hi Fak3r,Thank you for posting an excellent guide on creating your own dropbox clone. Though most parts are way over my head, I think i get the general idea. Would you be interested in taking on some freelance work and putting this system up on my server. We can figure out a price and start. Please let me know what is the best way to contact you because I could not find a contact form or your email address anywhere.

  • Eric

    Just got iFolder running recently on Opensuse 11.3. Official support is for Opensuse 11.1 but there are various builds out there. If you ever do get access to Opensuse, you don’t actually have to deal with mono dependencies. Just go to software.opensuse.org, select version of Opensuse or even SLES and search for iFolder and install it. This guide is mostly on the ball with what you need: http://www.diwi.nl/node/50

  • Eric

    Just got iFolder running recently on Opensuse 11.3. Official support is for Opensuse 11.1 but there are various builds out there. If you ever do get access to Opensuse, you don't actually have to deal with mono dependencies. Just go to software.opensuse.org, select version of Opensuse or even SLES and search for iFolder and install it. This guide is mostly on the ball with what you need: http://www.diwi.nl/node/50

  • http://fak3r.com fak3r

    Hmmm…and then I heard about ‘MagicFolder’ http://pypi.python.org/pypi/MagicFolder

    Annoying name, but interesting project…could kick that via inotify, or just via cron for eventual consistency.

  • http://fak3r.com fak3r

    Hmmm…and then I heard about 'MagicFolder' http://pypi.python.org/pypi/MagicFolderAnnoying name, but interesting project…could kick that via inotify, or just via cron for eventual consistency.

  • http://www.google.com/profiles/109710032589912618388 gorostas

    Great tutorial fak3r!

    But I have problem with one thing only. I have managed to sync 2 local folder.. but no luck with network ..
    This part cofuse me the most, the notation of the destination inside lsyncd.conf.xml file:

    THIS GOES!

    lsyncd ~/WELLBIS/IMAGES/a ~/WELLBIS/IMAGES/aa –debug –no-daemon

    THIS NOT!

    lsyncd /home/kreso/WELLBIS/IMAGES mk-lap::~/WELLBIS/IMAGES –debug –no-daemon

    cheers

  • http://www.google.com/profiles/109710032589912618388 gorostas

    Great tutorial fak3r!But I have problem with one thing only. I have managed to sync 2 local folder.. but no luck with network ..This part cofuse me the most, the notation of the destination inside lsyncd.conf.xml file:<target path=”desthost::module/”/>THIS GOES!lsyncd ~/WELLBIS/IMAGES/a ~/WELLBIS/IMAGES/aa –debug –no-daemonTHIS NOT!lsyncd /home/kreso/WELLBIS/IMAGES mk-lap::~/WELLBIS/IMAGES –debug –no-daemoncheers

  • Egbert Pot

    Hi fak3r and Mike Chelen. Thanks for this post and all the usefull information in the replies!

    I’ve used lsyncd and ZFS-FUSE together to create my own dropbox clone on Ubuntu 10.04. I’ve writen a quick how-to bellow. All the configuration files have been posted to pastebin for easy copy-paste

    /etc/lsyncd.conf.xml : http://pastebin.com/4z0Xg2Pk
    /usr/local/sbin/lsyncd-execute : http://pastebin.com/y6jQ3EVH

    Install ZFS-FUSE
    # aptitude install zfs-fuse

    Create a file that can be used as a ZFS pool

    # dd if=/dev/zero of=/opt/zfsbackupstorage/zfsbackupstorage001 bs=1k count=1000000

    This wil make a 100mb empty file

    Create a ZFS pool with the name zfsbackup

    # zpool create zfsbackup /opt/zfsbackupstorage/zfsbackupstorage001

    Set the mountpoint

    # zfs set mountpoint=/opt/backup

    Check if the ZFS filesystem has been mounted

    # df -h

    Install lsyncd

    First make sure rsync is installed:

    # apt-get install rsync

    Install build dependencies

    # apt-get install libxml2-dev build-essential

    Download the source code

    # cd /usr/local/src

    # wget http://lsyncd.googlecode.com/files/lsyncd-1.37.tar.gz

    # tar -xzvf lsyncd-1.37.tar.gz

    # cd lsyncd-1.37

    # ./configure

    # make

    # make install

    Note: It’s a very small programm, it might look like nothing is happening

    This install does not install the configuration file, so we’ll do that manually now:

    #cp lsyncd.conf.xml /etc/

    Edit /etc/lsyncd.conf.xml

    See http://pastebin.com/4z0Xg2Pk

    Create the /usr/local/sbin/lsyncd-execute file

    # nano /usr/local/sbin/lsyncd-execute

    See http://pastebin.com/y6jQ3EVH

    Give the /usr/local/sbin/lsyncd-execute script permissions to be executed

    # chmod +x /usr/local/sbin/lsyncd-execute

    Launch lsyncd in debug for testing

    We’re ready to give it a go, may as well run it in debug for fun and to learn how lsyncd does what it does:

    # lsyncd –conf /etc/lsyncd.conf.xml –debug

    Watch the log for errors, if none are found, launch lsyncd. You can also add lsyncd to /etc/rc.local so it will be started every time your server / desktop starts

    Good luck!

    Best, Egbert

  • Michael Langford

    This would be fantastic if integrated in with encryptions/Amazon S3

  • Michael Langford

    This would be fantastic if integrated in with encryptions/Amazon S3

  • Marius Kjeldahl

    Does this solution actually handle deletes, i.e. will a delete be properly propagated across machines? If your solution is similar to a “two-way” rsync, then if the file exists at either end, will it not be synced back if it is only deleted on one machine?

  • Marius Kjeldahl

    Does this solution actually handle deletes, i.e. will a delete be properly propagated across machines? If your solution is similar to a “two-way” rsync, then if the file exists at either end, will it not be synced back if it is only deleted on one machine?

  • Lauri

    instead of key without passphrases you could use ssh-agent for passwordless login.

  • Lauri

    instead of key without passphrases you could use ssh-agent for passwordless login.

  • Pingback: links for 2010-09-01 | andy.edmonds.be

  • http://fak3r.com fak3r

    Great writeup, I’ve been meaning to try out ZFS on Linux via FUSE, formerly ran ZFS on FreeNAS (FreeBSD). I’ve built a new RAID1 array from 2 1TB drives with ext4 on them. They’re the ‘green’ drives that use less power, and they do run cooler than the others, so having redundant storage using less power (considering I was running them in a separate server before it’s really less) was my goal, but having the advantages of ZFS would be the next step. Thanks for the detail!

  • http://www.largepot.net Large Pot

    I like this article! Will come again next time for sure, thank again

  • http://www.largepot.net Large Pot

    I like this article! Will come again next time for sure, thank again

  • http://yserver.blogspot.com/ Rapideo

    Keep Posting Thumbs Up!

  • http://yserver.blogspot.com/ Rapideo

    Keep Posting Thumbs Up!

  • Pingback: 独自のオープンソースをDropboxのクローン – fak3r構築HOWTOを | Japan – News And Articles

  • http://www.bestexternalhdd.com/buy/top-10-1tb-external-hard-drives.html 1TB External Hard Drives

    Windows7 starter does not support or let youchange the background image on your desktop. Sorry I had the same problemand founditin my manual

  • Avichi

    FAK3R
    Well I check my SMD drive absolute path and found that it was of the form

    smb://MYGROUP;UserFN%20UserLN@servername/public
    so I set the source path on lsyncd.conf to this and kept the same for the target, and restarted the service with lsyncd –conf /etc/lsyncd.conf.xml –debug

    Seems the service tried to connect to my SMB share but says the SSH port 22 on my “host” was refused, so I checked the port 22 on my Windows Vista Desktop(host) to be open, which is set to accept both protocols. Any thoughts?
    Avichi
    ****START ERROR LOG****
    Fri Sep 24 11:28:43 2010: Starting up
    Fri Sep 24 11:28:43 2010: watching smb://MYGROUP;UserFN%20UserLN@servername/public
    Fri Sep 24 11:28:43 2010: add_dirwatch(smb://MYGROUP;UserFN%20UserLN@servername/public, p->dirname:NULL, …)
    Fri Sep 24 11:28:43 2010: BUILDPATH(-1, smb://MYGROUP;UserFN%20UserLN@servername/public, (null)) -> smb://MYGROUP;UserFN%20UserLN@servername/public
    Fri Sep 24 11:28:43 2010: ERROR: Cannot add watch smb://MYGROUP;UserFN%20UserLN@servername/public (2:No such file or directory)
    Fri Sep 24 11:28:43 2010: dumped tosync stack.
    ssh: connect to host smb port 22: Connection refused

    rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
    rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7]
    Fri Sep 24 11:28:44 2010: Forked binary process returned non-zero return code: 12
    Fri Sep 24 11:28:44 2010: ERROR: Initial rsync from smb://MYGROUP;UserFN%20UserLN@servername/public to /var/www/html/docs failed.
    ****END ERROR LOG****

  • Avichi

    FAK3R
    Well I check my SMD drive absolute path and found that it was of the form

    smb://MYGROUP;UserFN%20UserLN@servername/public
    so I set the source path on lsyncd.conf to this and kept the same for the target, and restarted the service with lsyncd –conf /etc/lsyncd.conf.xml –debug

    Seems the service tried to connect to my SMB share but says the SSH port 22 on my “host” was refused, so I checked the port 22 on my Windows Vista Desktop(host) to be open, which is set to accept both protocols. Any thoughts?
    Avichi
    ****START ERROR LOG****
    Fri Sep 24 11:28:43 2010: Starting up
    Fri Sep 24 11:28:43 2010: watching smb://MYGROUP;UserFN%20UserLN@servername/public
    Fri Sep 24 11:28:43 2010: add_dirwatch(smb://MYGROUP;UserFN%20UserLN@servername/public, p->dirname:NULL, …)
    Fri Sep 24 11:28:43 2010: BUILDPATH(-1, smb://MYGROUP;UserFN%20UserLN@servername/public, (null)) -> smb://MYGROUP;UserFN%20UserLN@servername/public
    Fri Sep 24 11:28:43 2010: ERROR: Cannot add watch smb://MYGROUP;UserFN%20UserLN@servername/public (2:No such file or directory)
    Fri Sep 24 11:28:43 2010: dumped tosync stack.
    ssh: connect to host smb port 22: Connection refused

    rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
    rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7]
    Fri Sep 24 11:28:44 2010: Forked binary process returned non-zero return code: 12
    Fri Sep 24 11:28:44 2010: ERROR: Initial rsync from smb://MYGROUP;UserFN%20UserLN@servername/public to /var/www/html/docs failed.
    ****END ERROR LOG****

  • Pingback: http://fak3r.com/2009/09/14/howto-build-your-own-open-source-dropbox-clone/ by @fak3r – … — biotweets.org

  • Pingback: RT @Jim_Croft: http://fak3r.com/2009/09/14/howto-build-your-own-open-source-dropbox-clone/ by @fak3r … — biotweets.org

  • ChrisW

    absolutely sweet setup, really like how its work. i think i’m missing something though, my files update when added or removed but when i ‘update’ files they dont seem to update? am i missing a flag somewhere?

    • http://fak3r.com fak3r

      Yes, this should work, you can see it as an option in block of options in the source code of lsyncd.c starting on line 130 here:
      http://code.google.com/p/lsyncd/source/browse/trunk/lsyncd.c?r=31

      and line 748 it lists IN_CLOSE_WRITE as an option to use to call rsync. What does your /etc/lsyncd.conf.xml look like? Do you have a line like this?

      • ChrisW

        cool, close but its actually (with the forward slash at the end)

        • ChrisW

          so its “-lt%r”/>

          • http://fak3r.com fak3r

            Cool, glad you fixed it, I’ll update my docs too

  • ChrisW

    absolutely sweet setup, really like how its work. i think i’m missing something though, my files update when added or removed but when i ‘update’ files they dont seem to update? am i missing a flag somewhere?

    • http://fak3r.com fak3r

      Yes, this should work, you can see it as an option in block of options in the source code of lsyncd.c starting on line 130 here:
      http://code.google.com/p/lsyncd/source/browse/trunk/lsyncd.c?r=31

      and line 748 it lists IN_CLOSE_WRITE as an option to use to call rsync. What does your /etc/lsyncd.conf.xml look like? Do you have a line like this?

      • ChrisW

        cool, close but its actually (with the forward slash at the end)

        • ChrisW

          so its “-lt%r”/>

          • http://fak3r.com fak3r

            Cool, glad you fixed it, I’ll update my docs too

  • Pingback: How to Make Your Own Open Source, Dropbox-like Sync and Backup Service | JustGeeks.de

  • Pingback: How To Make Your Own Dropbox-Like Sync And Backup Service | Lifehacker Australia

  • Pingback: HOWTO build your own open source Dropbox clone – fak3r | It's About Time

  • Pingback: Your own open source dropbox-like service

  • James

    I would LOVE it if there was a windows alternative to setting up your own dropbox-style syncing service. I get annoyed at having a mere 5gb to play with, and seeing as i have access to many high capacity servers, it would be a great boon to find a home-brew syncing service that would allow me to sync all my music, all my photos, everything.

    this was an interesting read.

    Thanks

    • http://fak3r.com fak3r

      Check out the guys at Sparkleshare http://www.sparkleshare.org/ – only test builds are out for Linux so far, but they’re saying they’ll have Mac and Windows support. They are building it in Mono so you’d think a port to Windows via .NET would be straightforward, but I’d prefer they work on a solid backend protocol (maybe I should create fak3r-sync? ;) ) and then have others build front ends. I’m thinking more along the lines of how git has taken off; Linus came up with the idea and app, but it was all commandline. Then users/developers created front ends, webapps and other things on top of it. I think this would be the best way to go, I’m not a programmer, but if I could cobble together enough of the server side and make it run in Linux, and then just leave making GUIs to access up to others to make UIs for different platforms we may be able to pull something together. Hmmm…thanks, you’ve got me thinking about this again!

  • James

    I would LOVE it if there was a windows alternative to setting up your own dropbox-style syncing service. I get annoyed at having a mere 5gb to play with, and seeing as i have access to many high capacity servers, it would be a great boon to find a home-brew syncing service that would allow me to sync all my music, all my photos, everything.

    this was an interesting read.

    Thanks

    • http://fak3r.com fak3r

      Check out the guys at Sparkleshare http://www.sparkleshare.org/ – only test builds are out for Linux so far, but they’re saying they’ll have Mac and Windows support. They are building it in Mono so you’d think a port to Windows via .NET would be straightforward, but I’d prefer they work on a solid backend protocol (maybe I should create fak3r-sync? ;) ) and then have others build front ends. I’m thinking more along the lines of how git has taken off; Linus came up with the idea and app, but it was all commandline. Then users/developers created front ends, webapps and other things on top of it. I think this would be the best way to go, I’m not a programmer, but if I could cobble together enough of the server side and make it run in Linux, and then just leave making GUIs to access up to others to make UIs for different platforms we may be able to pull something together. Hmmm…thanks, you’ve got me thinking about this again!

  • Pingback: Twitted by lryo17

  • Twrang

    First of all, thanks for a great HOWTO!

    I have tried to read through the comment in order to find out if my concerns have been discussed. But could not find anything concrete.

    I have on my LAN network one server and three clients, each of them with two users. I want to synchronize users’ home directory with the server in such a way that the home directory is the same on all clients, regardless of wich client made the change.

    My concern is, where should synchronization take place if you want a two-way synchronization?

    1) Should all clients push and pull changes from the server?:

    Client 1 is turned on

    Client 1 makes a modification

    Client 1: “I have a modification for you” —> Server

    Client 1: “Do you have modification for me?” —> Server

    *

    *

    Client 2 is switched on

    (No modification)

    Client 2: “Do you have modification for me?” —> Server

    or …

    2) Should the server handle all the synchronization?

    Client 1 is turned on

    Server: “I have a modification to you(if any)” —> Client 1

    Server: “Do you have modification for me?” —> Client 1

    Client 1 makes a change

    Server: “I see you have modification for me” —> Client 1

    *

    *

    Client 2 is switched on

    Server: “I have a modification to you” —> Client 2

    Server: “Do you have modification for me?” —> Client 2

    Client 2 makes a change

    Server: “I see you have modification for me” —> Client 2

    Server: “I have a modification to you” —> Client 1

    or …

    3) Should clients push modifications to the server and the server then distribute out to all clients?

    Client 1 is turned on

    Client 1 makes a modification

    Client 1: “I have a modification for you” —> Server

    Server: “I have a modification for you, client 1, client 2 and client 3″

    Could any of thees solutions end up in an endless loop?

    Is there a standard way to do this?

    Pardon the long explanation. Thanks

  • Twrang

    First of all, thanks for a great HOWTO!

    I have tried to read through the comment in order to find out if my concerns have been discussed. But could not find anything concrete.

    I have on my LAN network one server and three clients, each of them with two users. I want to synchronize users’ home directory with the server in such a way that the home directory is the same on all clients, regardless of wich client made the change.

    My concern is, where should synchronization take place if you want a two-way synchronization?

    1) Should all clients push and pull changes from the server?:

    Client 1 is turned on

    Client 1 makes a modification

    Client 1: “I have a modification for you” —> Server

    Client 1: “Do you have modification for me?” —> Server

    *

    *

    Client 2 is switched on

    (No modification)

    Client 2: “Do you have modification for me?” —> Server

    or …

    2) Should the server handle all the synchronization?

    Client 1 is turned on

    Server: “I have a modification to you(if any)” —> Client 1

    Server: “Do you have modification for me?” —> Client 1

    Client 1 makes a change

    Server: “I see you have modification for me” —> Client 1

    *

    *

    Client 2 is switched on

    Server: “I have a modification to you” —> Client 2

    Server: “Do you have modification for me?” —> Client 2

    Client 2 makes a change

    Server: “I see you have modification for me” —> Client 2

    Server: “I have a modification to you” —> Client 1

    or …

    3) Should clients push modifications to the server and the server then distribute out to all clients?

    Client 1 is turned on

    Client 1 makes a modification

    Client 1: “I have a modification for you” —> Server

    Server: “I have a modification for you, client 1, client 2 and client 3″

    Could any of thees solutions end up in an endless loop?

    Is there a standard way to do this?

    Pardon the long explanation. Thanks

  • Pingback: Shared links from the week [Digest] October 16, 2010 | @puck

  • Pingback: Xentience (Internal) Blog » How to Make Your Own Open Source, Dropbox-like Sync and Backup Service

  • Pingback: hardware links « Rants

  • http://fak3r.com fak3r

    There was a big influx of new hits/posts on this article last week thanks to Lifehacker Australia posting about it here; http://www.lifehacker.com.au/2010/10/how-to-make-your-own-dropbox-like-sync-and-backup-service/ – they even came up with a pretty sweet logo: http://cache.gawker.com/assets/images/lifehacker/2010/10/opensource-dropbox.jpg

    Very cool that so many are (still) interested in this project – and that’s what it has become; a project. I’ll be releasing code to setup a complete command-line Dropbox like implementation on Linux in about a week. Code will be hosted on github.com and I’m hoping it will spur others to work on cross platform front-ends to talk to it. So far the technology is there, I’m just using what others have built, it’s just a matter of hooking it all up! After all, why reinvent the wheel? (not that I could ;) )

    Thanks again for all the comments and support!

  • http://fak3r.com fak3r

    There was a big influx of new hits/posts on this article last week thanks to Lifehacker Australia posting about it here; http://www.lifehacker.com.au/2010/10/how-to-make-your-own-dropbox-like-sync-and-backup-service/ – they even came up with a pretty sweet logo: http://cache.gawker.com/assets/images/lifehacker/2010/10/opensource-dropbox.jpg

    Very cool that so many are (still) interested in this project – and that’s what it has become; a project. I’ll be releasing code to setup a complete command-line Dropbox like implementation on Linux in about a week. Code will be hosted on github.com and I’m hoping it will spur others to work on cross platform front-ends to talk to it. So far the technology is there, I’m just using what others have built, it’s just a matter of hooking it all up! After all, why reinvent the wheel? (not that I could ;) )

    Thanks again for all the comments and support!

  • Pingback: Segnalibri al 21 ottobre 2010 | Ubuntu block notes

  • http://www.localbusinesscoachonline.com/coachnotes Vernessa Taylor

    There is a DropBox connection app in my recently installed PCLinuxOS. Any chance you did any of the work on this? (I’m over in Windows right now so I can’t check the actual name of the rpm.) Mainly, I’m curious because every app that’s built and placed into a .rpm or .deb isn’t safe enough for use with something as sensitive as your dropbox.

    • http://fak3r.com fak3r

      Hey there, no I have not worked on anything for PCLinuxOS – or any other Linux for that matter. What I have done is writing a ‘setup’ script to get this working for other, and do a true 2-way sync that the current directions don’t cover. I will be releasing this later this month (nov 2010) and would love to expand it’s use to other platforms at that time.

  • http://www.localbusinesscoachonline.com/coachnotes Vernessa Taylor

    There is a DropBox connection app in my recently installed PCLinuxOS. Any chance you did any of the work on this? (I’m over in Windows right now so I can’t check the actual name of the rpm.) Mainly, I’m curious because every app that’s built and placed into a .rpm or .deb isn’t safe enough for use with something as sensitive as your dropbox.

    • http://fak3r.com fak3r

      Hey there, no I have not worked on anything for PCLinuxOS – or any other Linux for that matter. What I have done is writing a ‘setup’ script to get this working for other, and do a true 2-way sync that the current directions don’t cover. I will be releasing this later this month (nov 2010) and would love to expand it’s use to other platforms at that time.

  • Pingback: Austoon Daily » HOWTO build your own open source Dropbox clone

  • KWisher

    Followed this Howto and I cannot get it to work. Ran lsyncd in debug mode and saw no errors. No files are being copied to the target machine. Any suggestions? Please let me know if you need to see my config file.

    • http://fak3r.com fak3r

      So take a look at all of your logfiles, is there anything in /var/log/syslog? Did lsyncd launch without error, can you see it via a ps? Feel free to post your config if you’d like, happy to take a look.

  • KWisher

    Followed this Howto and I cannot get it to work. Ran lsyncd in debug mode and saw no errors. No files are being copied to the target machine. Any suggestions? Please let me know if you need to see my config file.

    • http://fak3r.com fak3r

      So take a look at all of your logfiles, is there anything in /var/log/syslog? Did lsyncd launch without error, can you see it via a ps? Feel free to post your config if you’d like, happy to take a look.

  • Rasmus Hedin

    I think is a excellent Idea. I’ve looked at the sparkleshare which is another ambitious approach, although based on git – giving it version control but not good for large folders.

    I’m really interested in setting this up to work on OSx and Windows, are you interested in expanding this to some kind of shared project?

  • Rasmus Hedin

    I think is a excellent Idea. I’ve looked at the sparkleshare which is another ambitious approach, although based on git – giving it version control but not good for large folders.

    I’m really interested in setting this up to work on OSx and Windows, are you interested in expanding this to some kind of shared project?

  • Danji

    Im not sure how this is actually a dropbox clone.. rsync only syncs one way, so all changes made on the client are uploaded to the server (via lsynd automatically).

    For example: I have 2x clients and both run lsynd with the –delete option for rsync.
    ClientA copies fileA to sync folder
    FileA is synced to rsyncserver
    ClientB copies fileB to sync folder
    ClientB lsync kicks in and rsyncs fileB to the rsyncserver, but the –delete option removes fileA from the server.

    This isnt sync! The killer feature of dropbox is the ability to sync the same files across multiple clients, this doesnt do that.. as far as I can tell its just a step up from running rsync on a cron.

    Removing the –delete option just dumps 2 sets of files into the sync folder on the server, nothing is ever pushed back to the clients or managed.

    I get that this would work in a client client model, but the article isnt really that clean about it.

    Please, if I’ve missed something obvious I will stand humbly corrected :)

    Cheers

    • http://fak3r.com fak3r

      Yes, currently we’re missing a true 2-way sync with this model, but I’ve been working on addressing this using Unison in place of rsync which deals with this scenario. I’m going to release a ‘setup’ script that automates this and propagates the proper ssh keys. Once this is done it’s easy to setup multiple clients that will work just like a dropbox setup. I’ll announce it here but will have it up on my github account by the end of the month (nov 2010)

      • http://fak3r.com fak3r

        and while I’ll post more about it soon, this project has become lipsync, which sets up a lightweight service that provides command-line, Dropbox like syncing. Through it’s use of Unison it can provide the true 2-way syncing this idea did not. You can git it here https://github.com/philcryer/lipsync

  • Danji

    Im not sure how this is actually a dropbox clone.. rsync only syncs one way, so all changes made on the client are uploaded to the server (via lsynd automatically).

    For example: I have 2x clients and both run lsynd with the –delete option for rsync.
    ClientA copies fileA to sync folder
    FileA is synced to rsyncserver
    ClientB copies fileB to sync folder
    ClientB lsync kicks in and rsyncs fileB to the rsyncserver, but the –delete option removes fileA from the server.

    This isnt sync! The killer feature of dropbox is the ability to sync the same files across multiple clients, this doesnt do that.. as far as I can tell its just a step up from running rsync on a cron.

    Removing the –delete option just dumps 2 sets of files into the sync folder on the server, nothing is ever pushed back to the clients or managed.

    I get that this would work in a client client model, but the article isnt really that clean about it.

    Please, if I’ve missed something obvious I will stand humbly corrected :)

    Cheers

    • http://fak3r.com fak3r

      Yes, currently we’re missing a true 2-way sync with this model, but I’ve been working on addressing this using Unison in place of rsync which deals with this scenario. I’m going to release a ‘setup’ script that automates this and propagates the proper ssh keys. Once this is done it’s easy to setup multiple clients that will work just like a dropbox setup. I’ll announce it here but will have it up on my github account by the end of the month (nov 2010)

  • Pingback: Liens d’octobre 2010 | SkyMinds.Net

  • andrey i. mavlyanov

    it’s only working for the PC. once you need your files on iphone or android you NEED dropbox.

    • http://fak3r.com fak3r

      This is true, you’d have to have ssh and rsync clients running on your phone to allow syncing to the mainserver.

  • andrey i. mavlyanov

    it’s only working for the PC. once you need your files on iphone or android you NEED dropbox.

    • http://fak3r.com fak3r

      This is true, you’d have to have ssh and rsync clients running on your phone to allow syncing to the mainserver.

  • Johnnyraycantu

    Is this essentially and ssh network? Could i connect with a windows client using something like winSCP?

    • http://fak3r.com fak3r

      Sure, share the pub keys, figure out how to make it into a command and I’m sure you could have winSCP working with it.

  • Johnnyraycantu

    Is this essentially and ssh network? Could i connect with a windows client using something like winSCP?

  • Pingback: Use Linux to Build Your Own Dropbox | LearnByDoingIT

  • Pingback: Delicious Bookmarks for November 16th from 17:06 to 17:16 « Lâmôlabs

  • Pingback: Dropbox Homemade » realweb

  • http://www.sportsbettingchampsystemscam.com champ

    Hello, Thanks for the reminder about the finer details. Much appreciated. Keep up the good work!

  • Jd

    Great idea !
    I suggest ajaxplorer http://www.ajaxplorer.info/wordpress/ on the server as a webfilemanager. Ajaxplorer is slower but as more features than dropbox web interface.

    • Jd

      (has) more….

    • http://fak3r.com fak3r

      Whoa, good call, I’ll take a look at that, looks very nice!

  • Jd

    Great idea !
    I suggest ajaxplorer http://www.ajaxplorer.info/wordpress/ on the server as a webfilemanager. Ajaxplorer is slower but as more features than dropbox web interface.

    • Jd

      (has) more….

    • http://fak3r.com fak3r

      Whoa, good call, I’ll take a look at that, looks very nice!

  • julien23

    seems to be applicable to Qnap Nas as well : http://forum.qnap.com/viewtopic.php?f=15&t=37705

  • julien23

    seems to be applicable to Qnap Nas as well : http://forum.qnap.com/viewtopic.php?f=15&t=37705

  • http://twitter.com/benjoux benoit kornmann

    Brilliant! It works!

    Two comments though.

    1/ I don’t like passwordless keys, so my keys have password. There is a workaround for passwordless login, at least in ubuntu:
    Generate your key via the “password en encryption keys” utility (in application>accessories), setup a password and allow this key to be unlocked at login.
    Then type this

    SSH_AUTH_SOCK=”$(find /tmp/keyring*/ -perm 0755 -type s -user $USER -name ‘*ssh’ |head -n 1)”; lsyncd –conf ~/etc/lsyncd.conf.xml

    The first line will pass the unlocked key (stored in the RAM and not written n the disk, therefore more secure) to lsyncd.

    2/You may want to add in your tutorial how to have the daemon started at login

    Thank you so much for your Tutorial!

  • Heretic529

    i got to setting up lsync and every time i run it in debug mode it says there is a problem with line 31

    • http://fak3r.com fak3r

      hmmm…what is line 31? is it in the lsyncd config? if so try and use the default .conf it comes with and just change the

  • SPeedY

    Hey fak3r,

    I just came across this post and find it extremely interesting. Specially because in my workspace we are not allowed to use dropbox, but we will loved to have an internal system like dropbox.

    How is your linux script going?
    Will there be a way to sync with windows machine?

  • http://www.facebook.com/people/Logan-Kellar/100000324480557 Logan Kellar

    Or you could try ifolder, that is equally annoying to set up as this though.

  • Pingback: More programming-related links of interest | Loshin.com

  • Pingback: Augenmusik » Blog Archive » SubversiveBox: Your own Dropbox for Subversion

  • Luckysmack

    Hvae there been any updates on this? Some of the comments seem to be almost out of order. Do you have a link to this project on your github account? I would like to follow it if possible

    • http://fak3r.com fak3r

      Yep, I have been meaning to make a post updating things, but haven’t. Will put a notice at the top of this page pointing to lipsync https://github.com/philcryer/lipsync the continuation (formalization) of this project. I want to make it usable enoough for anyone to install/use, so far I have an installer script that does much of that, but needs more testing on non Debian/Ubuntu platforms. Thanks for your interest and let me know if you have further questions.

    • http://fak3r.com fak3r

      Yes, I’ve updated this post at the top, but the project I’ve started on github is called lipsync: https://github.com/philcryer/lipsync

  • http://zyrax.net/ feiming

    It probably won’t work if one of the server is behind NAT without port forwarding.

    My suggestion is sync over XMPP.Files can be transfered using XMPP file transfer and easily sync without the need of public IP or port forwarding.

    • http://fak3r.com fak3r

      Well I agree with you, but hey, without NAT and port forwarding this website wouldn’t work :) I am interested in your idea of using XMPP, I don’t know nearly enough about it, but I’ve used it way back when Jabber was just starting out, and using it for something like this sounds like it would solve a lot of networking issues/setup. Can you issue ‘rync’ commands over it, or does it have a built in mech? If you have more detail I’d like to hear it, I’ll poke around and see what I can learn, but ya, I think this could be a huge enhancement over the setup I defined here!

  • DiabloczDiablocz

    Are you running out of 2GB Dropbox free space? Here is a tip for you http://alturl.com/jmise

  • Pingback: Interesting Links | blogwalking.web.id

  • Pingback: Dropbox & open source alternative :Creel

  • http://doliver.co.uk/ David Oliver

    Wow, this looks great. Is lipsync usable at this stage?

    • http://fak3r.com fak3r

      I’ve checked in some fixes that addressed some errors in the setup file yesterday, and I’m actively working on it again. I have a VPS setup so I’m going to run lipsync on my home server (for the Server) and on the VPS and laptop for the clients.

  • Nec

    I’m sorry I’m not getting it : In a setup with one server (S) and two hosts (A and B), how does lsync daemon running on B get informed A has changed something? The way I understand it is A -> S and that’s all… May you detail that, please?

    Moreover, how is this system reacting when B is disconnected, A changes a file (that gets commited on S), then B gets reconnected?

    • http://fak3r.com fak3r

      I’ve posted a short diagram to show how that would work – my ‘workaround’ is to have the clients check with the server every now and then via cron to see if there is anything new for them. I’d rather have a push/comet server style option, but for now, this is how I’m running it. http://www.gliffy.com/publish/1601804/

  • Pingback: Hak5 – Technolust since 2005 » Episode 904 – Writing software without a line of code, crafting packets with hping, case mod cable mangement & cathodes and more

  • Pingback: Episode 904 – Writing software without a line of code, crafting packets with hping, case mod cable mangement & cathodes and more | HAK5

  • Anonymouse

    Just a fly-by comment: Once your server is set-up for use, is it possible to integrate this with Android so that one can sync Contacts Calendars or even something like documents?

    • http://fak3r.com fak3r

      I’m not sure, can you use rsync and ssh from Android? If so, it might be easy to setup. I don’t have an Android phone, but would certainly help in getting the install script working with one if you wanted to try it out. Check out the project here https://github.com/philcryer/lipsync and feel free to post to the mailing list if you’re interested. Thanks!

      • SOSDD

        Evernote is a Gui install for Android that can perform function from Windows.  Don’t know what they use for the backend though.

  • Pingback: lipsync: Ein Open-Source Dropdox Klon | Paul Staab's Blog

  • Pingback: P2P4U Soccer

  • http://profiles.google.com/rushikh Rushir Parikh

    Is this Linux only or will it work on a Mac?

    • http://fak3r.com fak3r

      Hey, right now it’s Linux, but we’re actively working on OSX support, it’s one of the next things I want to have working. We hope to have something to test out soon, the holdup is getting Lsyncd running in OSX, and I think we’re close. Take a look at the code here: https://github.com/philcryer/lipsync

  • Pingback: Complete crossplatform dropbox replacement « hattb

  • Pingback: Make my own “dropbox, ubuntu one” server at home Drija

  • Pingback: » Piensalo antes de guardar tus archivos en Dropbox Comentarios a las tecnologías web

  • Pingback: Another dropbox alternative – somewhat | digital playground

  • Pingback: DropBox Like using Lsyncd « Hardcore Linux

  • Pingback: Alternativas para o Dropbox

  • Vinh Nguyen

    Seems like the process is easier now.  Linux Journal [mentioned](http://www.linuxjournal.com/content/drop-your-dropbox-and-sparkleshare-instead) sparkleshare:
    http://sparkleshare.org/

    • http://fak3r.com fak3r

      NOTE: fixed link http://www.linuxjournal.com/content/drop-your-dropbox-and-sparkleshare-instead

      we’ve spoken of Sparkleshare on this page quite a bit, and it’s a great looking project. A few of the cons though is that it has a ton of Mono dependancies, and it uses github or gitosis as a backend. You can read about git’s use of storing large binaries elsewhere, but suffice to say it might not be the best fit. The aim with project lipsync (http://lipsync.it) is to get an open backend running, then worry about using a frontend, like Sparkleshare, later. To me those would bring the best of both solutions together, but we’ll see. 

  • Pingback: links for 2011-06-29 « Bloggitation

  • http://twitter.com/nicolai_a Nicolai Amter

    Great, might look into see if it can be installed on a Drobo FS

  • Pingback: Alternative to Dropbox (on my server)? Drija

  • Pingback: remote access connection managerを初めて見ました。

  • Pingback: Open Source Dropbox alternative « Jonathan Arbib

  • Pingback: Filesystem to quickly get recent modifications Drija

  • Pingback: Automatically sync directories via FTP during off hours? Drija

  • mark

    If you wanted to access from windows,  you could simply use apache and WebDAV to mount a drive in windows.. windows 7 has good built in WebDAV support… ther others need clients installed… then you could use the free ios app WebDAV browser for your ipad/phone/pod/

  • Guest

    ExpanDrive is nice and cheap. Maps ssh/sftp etc. to a drive letter.

  • Nadimrafik

    Great work!! Needed!

    Re: P2P: I always thought ed2k was a very civilised filesharing setup that could be adapted really easily

  • Pingback: Roll Your Own Dropbox-style Remote Backup System | iPhone 2 die 4

  • Pingback: Atlantis | Roll Your Own Dropbox-style Remote Backup System

  • Antonio Almeida

  • Pingback: LipSync, toi aussi dispose de ton dropbox maison ! | Zat's Hall

  • Pingback: Dropbox si open-source | 24ore lajme

  • Pingback: Dropbox ngjane ne open-source

  • Pingback: Create your own ‘Dropbox’ « 0ddn1x: tricks with *nix

  • Pingback: DEFCON 19: Taking your ball and going home | fak3r

  • Pingback: European alternatives to Dropbox? Drija

  • Pingback: Свой Dropbox из подручных материалов - execbit.ru

  • Simon Schneider

    Awesome, this actually works! Just installed it on my ArchLinux Box (had to adopt some scripts to use right place/group for the files) but now it works great. Are you still working on it? I’m really interested in it and thought about making an ArchLinux package for it.

    Is it possible to have more than one client running at the same time?

    • http://fak3r.com fak3r

      Hey, glad it worked for you, did you use the latest from the project site for lipsync? http://lipsync.it/ I have some open issues there, and haven’t been working on it much in the last few weeks, but will be getting back to it. If you want to sub your changes for Arch I’ll integrate them. Thanks

  • Pingback: Super Nerdy Cool » Real time file synchronization like Dropbox via Unison

  • Pingback: » HOWTO build your own open source Dropbox clone 31team blog

  • Pingback: Build yo | Kibermed

  • Pingback: Eigener Dropbox Server? Zumindest fast!

  • Anonymous

    Niice – thought of using OwnCloud as the web-based frontend? This would make a sweeet addition to OwnCloud..

    • http://fak3r.com fak3r

      Yes – I have it on my server and I’ve used it, it’s very slick. I’m working with the folks at unhosted.org to understand their webdav concept more to see if that’s also a fit, but just for the web frontend, OwnCloud is awesome. Not sure if lipsync would need to auth to that via webdav then or what, but I think making it into a module of OwnCloud would be a good option.

  • http://twitter.com/niu_tech niu tech

    Have you tried FTPBox: http://ftpbox.org or Syncany: http://syncany.org ? Both are open source alternatives to Dropbox!

  • btrfs in the future

    What a nice feature to combine simple scripts with lsyncd and desktop notifications!

    Having possible conflicts in mind, if lipsync would trigger two-way csync (the complexity of the n-way csync2 may not be needed) for the transports, it could be superior to rsync + watching deletion (the bitpocket appoach). (Avoid loosing changes because of a later change on another replica!)

    I second your approach to get the syncing and copying of arbitrary file types (including large binaries)  done right first, and keep it independent from any specific version (history) control system. We can always use our favorite version control, snapshoting tool on top of a synced files folder. Different source code projects use different version control systems, and on a long time backup server we may only want to commit / take a snapshot of all files once a day etc.

  • Pingback: HOWTO build your own open source Dropbox clone – fak3r | Nerdfront

  • Pingback: Lazy "dropbox" - does anyone know such?

  • http://techwizworld.net/ Techplex Engineer

    There is now an lsyncd package