To elegantly fill a hard disk, use rsync

11 September 2020

A friend is going away for a while and will have time to kill and uncertain access to the internet. You have a spare external drive and you decide to copy over your collection of manga films, or whatever they like. And of course let’s just state at the outset that this is a theoretical scenario, because you would never consider infringing copyright.

The perfect tool for this is rsync:

  • Create a new folder and symlink to everything you want to put on the new disk with ln -s
  • Let’s assume you are in this folder as the current working directory and that the external disk is called Exernal
nice -n 20 rsync -L -r -n -v --human-readable --progress ./ /Volumes/External
  • For an explanation of what all the flags are doing, please see below. The key is that rsync does a try run, following the symlinks, and gives you the total size at the end of the list of files. If this is less than your external hard disk capacity, great. If it’s more, you need to delete a few of the symlinks and re-run until the total will fit on the disk
  • When you’re ready, run the above without the -n flag to run it for real

Note that with 1TB of data and a 50MB/s transfer speed, which is typical over USB to a portable hard disk, it will take a while: 1000000×150×160×160=5.61000000 \times \frac {1}{50} \times \frac {1}{60} \times \frac {1}{60} = 5.6 hours. This is why we use nice, to make the copying operation low priority.

The command above explained in more detail:

# Run rsync with lowest priority
nice -n 20 rsync
# Copy links i.e. transform symlinks into referent
# Recursive i.e. recurse into directories
# Dry run -- remove when you want to run for real
# Verbose
# Human readable file sizes
# Show progress in the output
# Source
# Destination