Sync clients' files with server - Electron/node.js

Question

My goal is to make an Electron application, which synchronizes clients' folder with server. To explain it more clearly:

1. If client doesn't have the files present on the host server, the application downloads all of the files from server to client.
2. If client has the files, but some files have been updated on the server, the application deletes ONLY the outdated files (leaving the unmodified ones) and downloads the updated files.
3. If a file has been removed from the host server, but is present at client's folder, the application deletes the file.

Simply, the application has to make sure, that client has EXACT copy of host server's folder.

So far, I did this via wget -m, however frequently wget did not recognize, that some files changed and left clients with outdated files.

Recently I've heard of zsync-windows and webtorrent npm package, but I am not sure which approach is right and how to actually accomplish my goal. Thanks for any help.

Answer 1

You can use rsync which is widely used for backups and mirroring and as an improved copy command for everyday use. It offers a large number of options that control every aspect of its behaviour and permit very flexible specification of the set of files to be copied.

It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.

For your use case:

1. If the client doesn't have the files present on the host server, the application downloads all of the files from a server to the client. This can be achieved by simple rsync.

2. If the client has the files, but some files have been updated on the server, the application deletes ONLY the outdated files (leaving the unmodified ones) and downloads the updated files. Use: –remove-source-files or -delete based on whether you want to delete the outdated files from the source or the destination.

3. If a file has been removed from the host server but is present at the client's folder, the application deletes the file. Use: -delete option of rsync.

   rsync -a --delete source destination

Answer 2

Given it's a folder list (and therefore having simple filenames without spaces, etc.), you can pick the filenames with below code

# Get last item from each line of FILELIST
awk '{print $NF}' FILELIST | sort >weblist

# Generate a list of your files
find -type f -print | sort >mylist

# Compare results
comm -23 mylist weblist >diffs

# Remove old files
xargs -r echo rm -fv <diffs

you'll need to remove the final echo to allow rm work

Next time you want to update your mirror, you can modify the comm line (by swapping the two file arguments) to find the set of files you don't have, and feed those to wget.

or

rsync -av --delete https://mirror.abcd.org/xyz/xyz-folder/ my-client-xyz-directory/

Answer 3

rsync is a good approach but you will need to access it via node.js

An npm package like this may help you: https://github.com/mattijs/node-rsync

But things will get slightly more difficult on windows systems: How to get rsync command on windows?

Answer 4

If you have ssh access to the server an approach could be using rsync through a Node.js package.

There's a good article here on how to implement this.