Migrating Servers with Syncthing

I’m in the process of making time to migrate my blog (and some other things) to a better server and giving it some much-needed love to repair some of the broken links on here. So I thought I would take a moment and share some thoughts about Syncthing and perhaps a use for it that are not clear at first.

First, Syncthing is amazing. It’s come A LONG way over the past year and it’s become my primary data sync solution. I know that most people use it as a replacement for Dropbox and combine it with Owncloud to replace their god-awful file sync. (Seriously, Owncloud devs; I’ve lost A LOT of data and I’ve gotten to the point where I’m just about to completely retire it.) There are some other applications of Syncthing that I’m surprised no one has thought of yet. Maybe I’m the first or I’m just completely nuts. Either way I’ve used to migrate windows file servers and it’s worked amazing handling millions of files without a single hitch. It also provides a very simple way having a backout if things go wrong (if you have the resources) without having to restore backups. So without further adieu here’s a quick guide for migrating Windows file servers. This could be done with Linux file servers as well replacing Robocopy with Rsync. I haven’t done a Linux server with this method so YMMV.

Oh and this should go without saying; ALWAY MAKE SURE YOU BACKUPS ARE TRUSTED BEFORE DOING THIS! You don’t want to risk your job, reputation, or your company’s / client’s data on this. (Even though it works REALLY well)

Step One: Know your data!

This isn’t going to work for all cases, I wouldn’t recommend it over a WAN, VPN, etc. I don’t see anyway to easily and reliably QOS this. I also wouldn’t recommend this for data that needs a lot of custom permissions as Syncthing doesn’t transfer file permissions; there is a work around though (Robocopy). The more files you have the more resources Syncthing is going to take. While in my experience it’s hard to completely overwhelm a decent system; make sure your network/systems/storage/etc can handle the loads. That being said I’ve found this process is less resource intensive than using Robocopy.

Step Two: Setting up Syncthing.

This requires a bit of forethought. There’s many way to set up Syncthing. You could create a service, task, run it from the cmd line, etc. I found the easiest way is to use SyncTrayzor to handle Syncthing. It does a great job at handling any file conflicts, keeps Syncthing updated, and gives you a little more control over the process. You’ll just need to decide what will work best for your enviroment.

Step Two (a): Just get the data there dammit!

You need both the source and destination systems added to each other BEFORE you add the folder(s) to sync. On your source system add the folders and make sure they’re folder masters. Do not share with your source system at this point. Now setup the destination folder(s) but DO NOT select folder master. Keep in mind the Folder ID is CASE SENSITIVE so don’t screw this up or you’re going to have problems. Now on your source system share the folder(s) to your destination system. Wait a few moments and you’ll see a notice on your destination system that a folder wants to be shared. Ignore it. Instead share the folder(s) you’ve created with the source system and restart Syncthing. Wait a few moments and you’ll see things reconnect and start to Sync. That’s it. Your data will sync and you don’t have to baby sit it (like you do with Robocopy). This is amazing for production data as the changes will get replicated to the destination. So you can leave this up for as long as you need until you get the window to cut over the servers.

At this point you’re saying, BUT PERMISSIONS!? OWNERS!? MODIFY DATES!? … you can fix all of this with Robocopy. Stop Syncthing and run your normal server migration scripts. Since the data is already there all you’re doing is fixing metadata at this point. Then cut over the server normally.

Step Two (b): We need the data, but we need all that pesky metadata too!

So you need permissions, owners, etc. along with your data. This makes things a little more tricky as Syncthing doesn’t transfer this kind of stuff. The best way around this problem I’ve found is to robocopy the folder structure with permissions, restore a backup, or run your normal robocopy scripts FIRST. Then follow the same steps above. The sync will be much faster if the data is already there and since the permissions are set ahead of time then most of your data will have the correct permissions as it comes over. This is useful for migrating from one production system to another production system where you need to keep up security. Obviously Syncthing isn’t designed for this kind of usage so use common sense and test in your enviroment first! When you’re ready to cut over the system disable Syncthing and follow up with a robocopy to make sure all the metadata is correct.

Step Three: Cleanup

Once the systems are cut over, shares disabled, etc. You can just remove Syncthing from the systems… or… if you need to make sure that you have a backout plan; make the destination system is a folder master and remove the folder master from the source system effectively reversing the source / destination roles. This way if something happens you can get the old system up and running without much of a problem.

Happy Computing!