Hi, i am trying to solve my synchronization problem between my laptop and desktop. Let me explain . I have one directory, which contains all development tools and project workspaces i use. That means stuff like Eclipse IDE, various Java SDKs, Maven, Maven repository, Apache, PHP, MySQL, PostgreSQL, Tomcat, JBoss, database tools, whatever i need, in "portable" form (that means everything is contained in this one folder, and if it is on other computer inside the same directory on same drive, everything works fine). While most of the stuff is static, there are are files which do change - Eclipse workspaces, local Maven repository cache (.m2 directory), Eclipse preferences, database files etc, and they are at various places all around this directory structure. What i would like to have is to synchronize all this stuff from my main computer to one central place, then synchronize it to other places (laptop, computer i have at mom's house if i will get a important call when i am not in the city where i work). So it is pretty much A<==>B and C<==>B synchronization, with a possibility of D<==>B synchronization . It is about 230k files and about 13GB size. Files are added, modified, deleted, sometimes even few times in a second. Desktops and laptop all run Windows 8, the "server" is running Ubuntu, but the drives are shared via SMB, so Windows can see it as shared folders mounted with local drive names. Remote computers can access the local server via OpenVPN, so no issue with SMB or other protocols even from remote locations. I tried various solutions, with not much success so far : 1) owncloud - it is slow! Even if i synchronized only the workspaces (about 2,5GB of data), it took it 2,5 days to get the data to the DAV server. Sure, i understand it is not meant for such use cases, and i would accept that initial upload just takes a lot of time. What i cannot accept is how their synchronization client - constant 15% CPU usage 24/7 is not acceptable. So owncloud at least for time being is not usable for me. 2) Bittorent Sync - at first it looked fine, it was relatively quick (13GB of stuff synced in few hours), but... Even with time synced to the same NTP server and even after i doublechecked the times are exactly the same, it sometimes decides that the copy on the "server" is the newer one that the copy i just changed, and overwrites my local changes. That is an unacceptable behaviour, so that means BT Sync is out of question too, at least until it reaches stable version, maybe i will try it after that again. 3) DirSync Pro - installed, configured, initial backup gone through fine it seems, bit slower than BT Sync, but still in acceptable timeframe - and then it crashed and the synchronization settings were gone. Not really looking forward to do all this again. Going to try iFolder now, but if you got other alternatives, i would like to hear about them.
I normally use an app called "second copy" to do syncs between servers etc. if I am doing migrations. http://www.centered.com/ (they say it's a backup program but I have never used it as such) Have you considered doing multithreaded robocopy scripts? (assuming you are using windows machines when syncing)
I think i missed the requirement that i want the synchronization happening in background, without user interaction, ideally right after i made the file change. And uhm, it should be free, it is just for my personal use, that is i am too lazy to copy everything over and/or setup everything everywhere over and over again. Robocopy would end up similar to owncloud CPU usage unfortunately, as that was the problem with owncloud - it was not using directory polling for file changes, instead it was going through all the folders over and over again, every minute - except with tens or hundreds of thousands of files that is not a goo idea.
im currently using microsoft#s skydrive to do exactly what your doing (but with all my data), before that I used dropbox
Well yeah, Skydrive could be an option, thankfully i have one of those 25GB accounts for free . My dropbox has 28GB as well, but that is time limited (a 20GB promo for buying HTC One). What the problem is with these services is well... trust. Considering the content (company stuff), i really wouldn't like these to be somewhere in the "cloud" at a different provider. So if possible, i would like to explore options where i could i use my server (which is online 24/7) first. But yeah, Skydrive could be an last resort option if nothing else will work.
Another one bites the dust . Tried SparkleShare, it looked fine from technical standpoint, except one thing - GIT, which is used as the backed is a distributed versioning system. Which means everyone has a copy of repository. Which means that my 13GB of mostly binaries (true, mostly not changing ones) would be duplicated at least once, that means 26GB or more space used for 13GB of data... Even with 256GB SSD i use it is way too much. Next in the list : http://www.seafile.com/en/download/ http://www.cis.upenn.edu/~bcpierce/unison/ (unfortunately no longer developed)
So after a bit of fight, it seems i will use Seafile for now, even if it is not the ideal self-hosted solution, but it goes as close as it gets for now. The negatives is that it requires 4 extra ports outside of my standard HTTP server, i could get rid of two by proxying two HTTP ports to my main HTTP server, but i still need to open two extra ports on the firewall. It also needed ~10GB for initial synchronization of my 13GB directory, but after the initial sync and restart of the client it went down to few megabytes. On positive side the performance is very good (less than hour to add all files), and so far it looks fine. Will see later if it starts making similar mistakes as btsync or not. Skydrive is on the list for now, as i prefer self-hosted service if possible.
I used Owncloud on Linux. It was my first experience of this type of backup as I normally use synctoy to an external HDD. It seemed ok for what it did and has clients for Windows and Mac as well as Linux. I was planning to use it on my dual boot but I've drifted back to Windows again it seems. http://owncloud.org/
@monkiboi: if you read the opening post, you will see owncloud as my first item. It is super slow for anything more than few files (2.5GB in 50k files took 2 and half days to upload) and the client hogs CPU because it doesn't use filesystem notifications, but instead it is crawling the whole sync folder every minute - except crawling 15GB of data and hundreds of thousands of files takes much longer than that. @alecamused: Skydrive, Google drive and others have the issue of trust - i cannot just use that kind of cloud services for company data.
What you want is bit level sync, which isn't done by many of those already listed. Essentially after the first sync, it then only looks for bit changes, rather than copying the whole file across. Have a look at Synkron http://synkron.sourceforge.net/ It's fast, really fast
Microsoft's Sync Toy is perfect and very functional for this, you can make it do every direction and type of sync and have several set up at once. I believe it works with network share folders too, but I haven't actually tried it for that.
@notmeagain: AFAIK "rsync is actually not recommended for two-way sync". @Burnout21, @boiled_elephant: In case Seafile will fail me, then i will look at those options too. For now, seafile works as it should, and doesn't even use extreme amounts of disk space on server side, as i turned history off, so it keeps only the latest version. It also runs nicely in background, it seems that it monitors file changes in synchronized directories, waits for minor idle moments, then it "commits" all accumulated changes to the server at once. Pretty much exactly what i needed.
More than Sync Toy there, then, as Sync Toy is immediate - it doesn't perform backup/history functions or keep multiple copies of synched files. So sounds like you're onto the winner Also, I don't know that Sync Toy is live - I think you have to run it manually or on schedules. It was really designed for backup & archival purposes.
Did I mention Synkron makes an exact copy of the directory that allows you to access your files without the need for an application to restore them. I use to love using Déjà Dup, but I it requires the use of the app to restore if anything went a miss. When running multiple platforms having a hardware failure meant I could be without access at the worst of times. Déjà Dup backed up to a mirco usb drive that sat flush with the side of my laptop. If the laptop donked out I could pull the flash drive and get access, however access would require a linux based machine with Déjà Dup. Not ideal at all.
Whether it'll help for your needs but i use a program called ViseVersa Free to sync my folders onto my external drives. http://www.tgrmn.com
