Here I want to honor a very old program which is called "rsync" which can be used for fast copying of files to/from remote computers. Rsync has been extended with plenty of new functionalities, and generally I prefer rsync to scp when I have to copy more than one file. Let me give a few examples:
Example 1): speedIn my last post I presented a little script which generates an arbitrary amount of files with random contents. For this test I generated 1000 files which were 1k large each. Copying them from my home PC kepler to the remote computer lululu (via a 10/2Mbit ADSL line) will be signifcantly slower with scp than rsync:
bassler@kepler:~/test$ time scp * lululu:test/. bassler@lululu's password: real 1m19.619s user 0m0.188s sys 0m0.204s
bassler@kepler:~/test$ time rsync * lululu:test/. bassler@lululu's password: real 0m14.780s user 0m0.048s sys 0m0.032s
That is 80 seconds for scp and 15 seconds for rsync. (Yes, the target directory was wiped empty before I issued the rsync command.) The overhead of scp becomes significant if many files are to be copied. If I copy a single 1M file instead of 1000 1k files, then there is no noticeable speed difference.
Example 2: transferring very large filesRsync has a nice resume option for transferring large files, if transfer for some reason is interrupted. Basically what your options should look like:
rsync -rvv --inplace --append --progress /disk1/Movies/* lululu:Movies/.afaik, this is not possible with scp in a straightforward way.
Example 3: tunnel your way past firewalls and gatewaysThis is probably the coolest feature of rsync! If I am at home and want to update the files on my office PC baslup I face the problem that the PC is hidden behind the university firewall/gateway lululu. So with scp I would have to copy the files to lululu first, and from there again copy it to my office PC baslup.
Now, if rsync is installed on all PCs in the chain, I can issue a very simple command which will establish a link directly to my office PC.
rsync -va -e "ssh lululu ssh" test/* baslup:test/
You can add several PCs in the chain. Here fufufu is added:
rsync -va -e "ssh lululu ssh fufufu ssh" test/* baslup:test/the only annoying thing is, you have to enter your password multiple times, (unless you add the ssh keys for password less logins).
Ok, finally a list of the most important options for rsync:
-r recurse into dirs (you might just as well use -a instead)
-a archive: recurse into dirs, preserves symlinks, permissions, timestamps, group and owner.
-u update: don't overwrite newer files on receiver
-n dry run, copies no files, but shows what would be done
--del delete files/dirs from remote server which do not exist locally
-z enable compression. Yes, if you have files with lots of air in them, you can achieve dramatic transfer speedups. Works well for uncompressed scientific data, bitmap files, large textfiles etc, but is much slower when transferring mp3, jpg, gz, zip, pdf and similar compressed files...
Thank you, developers of rsync! This has saved me alot of hassle.