rsync Command Examples

Rsync (Remote Sync) is a synchronization tool for copying files and directories in a system or between systems. The single biggest advantage of rsync is that it copies only the changed files and thus reduces the CPU consumption and saves the bandwidth and time while copying files.

Advantages of rsync over scp

I already showed you some practical examples of scp command. scp, short for secure copy, is also used for copying files between two remote system over ssh.

But rsync has certain advantages over scp that make it a better choice.

• rsync copies only the files that are changed while scp copies every file and overwrites them if needed. So a better speed is guaranteed in rsync.
• rsync can also work without encryption. This can reduce the overhead. But this should be considered when there is no security risk of transferring without encryption.
• One can resume incomplete transfer in rsync while scp doesn’t allow it.

If you don’t have rsync installed, you can install it using your package manager.

In Ubuntu and Debian, you can install rsync with the following command.

sudo apt install rsync

Practical examples of rsync command in Linux

The rsync command has the following structure

rsync [OPTIONS] Source Destination

Where source and destination can be local paths or remote paths in the format username@hostname:path/to/file

Let’s see how to use rsync command with some helpful examples.

1. Sync Local Files [One-way Sync]

To copy local files from directory A into Backup-A-dir,

rsync A/ Backup-A-dir/

This command copies the files (if changes found) of directory A into Backup-A-dir. This won’t copy any extra files in Backup-A-dir into A that are not in A initially. This is why it is called as one-way sync.

2. Sync Remote Files [One-way Sync]

To sync files between a remote system and a local system, the command is similar. Note that the source location and destination location can either be local file system paths or remote system(ssh) paths.

rsync dev/build deploy@us-linuxhandbook.com:~/Backup

3. Two-way Sync

The above command copies the files from the source to the destination. But if the system had some extra files that were not on the source, those will not be deleted by One-way sync. If you want that, you’ll have to use two-way sync.

To maintain both endpoints (directories or files) at the same state with the same files and no extras on either side, just append the –delete option to the original command.

rsync A/ Backup-A-dir/ --delete

4. Delete Source Files After Copying

If you need to delete the files in source once the transfer is complete, –remove-source-files option should come in handy.

rsync A/ Backup-A-dir/ --remove-source-files

You should be careful with the above command. You should delete source only if enough copies are made and the data is not needed anymore in the source.

5. Include and Exclude Files

If you need (or do not need) to transfer files whose name match a pattern,
–include and –exclude option can be used. Each option should be given a pattern after ‘=’ sign.

rsync A/ Backup-A-dir/ --include=*.py --exclude=*.tmp.py

The above command copies all the files with extension .py except for those with an extension .tmp.py.

You can use regular expression as a pattern. Learn more about it here.

Tip: If the list of patterns for either of the options is huge, you can store it in a file and you can pass their names to the –include-from and –exclude-from option.

6. rsync Over SSH

If you want to transfer the files over SSH, you need to specify ssh with -e option.

rsync -e ssh A/ seeni@LinuxHandBook:~/Backup-A-dir/

This is the most preferred way of transferring files to a remote system since it’s encrypted. One should note that there will be an overhead involved due to the encryption. So it might take more time than normal transfer.

In order for this to work, you should enable ssh logins configured on the server side. The public key and private key also has to be at their respective paths.

To learn more about ssh and its setup, read these LinuxHandBook articles about client-side SSH configuration and server-side SSH configuration.

7. Verbose Mode

Most of the commands in Linux have a verbose option to log the command’s action in the terminal. rsync is no exception.

Pass either -v option or —verbose option to verbose the command. It will list the actions that are being done and its progress. This will be very helpful while debugging.

rsync A/ Backup-A-dir/ -v -r

Output will be similar to the one below

$ rsync A/ Backup-A-dir/ -v -r
sending incremental file list
created directory Backup-A-dir
./
file1.txt
file2.txt
file3.txt
file4.txt
file5.txt
file6.txt
sent 388 bytes received 168 bytes 1,112.00 bytes/sec
total size is 0 speedup is 0.00

8. Dry Run – Run But Don’t Copy

In case you wanted to know the files that are going to be copied without actually doing the transfer, you can use –dry-run( or -n) option.

It does all the operation as the normal rsync command does except for the copying part. It will list the files that will be copied or deleted (if needed) and then it will stop just before copying.

rsync -v A/ Backup-A-dir/ --dry-run

Which will result in

$ rsync -v A/ Backup-A-dir/ –dry-run
sending incremental file list
created directory Backup-A-dir
./
file1.txt
file2.txt
file3.txt
file4.txt
file5.txt
file6.txt
sent 172 bytes received 72 bytes 488.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)

Note that you need to use -v option to see the above output of the dry run command. If not, the dry run will happen, but it won’t display the results.

9. Display Progress of Transfer

If you want to display the progress of the transfer with rsync, use the –progress option.

rsync A/ Backup-A-dir/ --progress

Above command will show a progress bar similar to the one below:

$ rsync -r A/ Backup-A-dir/ –progress
sending incremental file list
created directory Backup-A-dir
./
file1.txt 0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=5/7)
file2.txt 0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=4/7)
file3.txt 0 100% 0.00kB/s 0:00:00 (xfr#3, to-chk=3/7)
file4.txt 0 100% 0.00kB/s 0:00:00 (xfr#4, to-chk=2/7)
file5.txt 0 100% 0.00kB/s 0:00:00 (xfr#5, to-chk=1/7)
file6.txt 0 100% 0.00kB/s 0:00:00 (xfr#6, to-chk=0/7)

10. Compress and Transfer Data

If you want to save network bandwidth and time, you can compress the data to be transferred using -z option. It will be decompressed automatically in the destination.

This trick can save a lot of network time and cost when data to be transferred is huge. It should be avoided for small files since the overhead processing outweighs the total time.

rsync -z A/ Backup-A-dir/

Note that one should avoid using -z option while doing local transfers and small files. It will slow things down unnecessarily.

11. Recursively copy the Files and Directories

All the above commands copy only files, not sub-directories (the same case with every Linux command). As a result, files inside those sub-directories are not copied. This can be eliminated by recursive copying.

To recursively copy the files and directories, -r option can be used.

rsync -r A/ Backup-A-dir/

12. Archive and Preserve metadata

If you want to preserve the symbolic links, timestamps, file permissions, user & group ownership of the files, -a option can be used.

rsync -a A/ Backup-A-dir/

This option also includes the functionality of -r option. So it recursively copies the files and preserves the metadata of copied files.

13. Set File Size Limit

To avoid transferring the large files, you can set a file size limit with the help of –max-size option. This helps you to keep a check on file size that will be copied.

rsync --max-size='100K' A/ Backup-A-dir/ 

14. Set Bandwidth Limit

If you have any constraints on network speeds, you can specify those with the help of –bwlimit option. bwlimit should be expressed in kbps.

rsync --bwlimit=100 A/ Backup-A-dir/

15. Resume download with rsync

If the download/transfer is incomplete, you can use the rsync command to keep incomplete downloads so that transfer can resume on the next time when the same command is issued.

To resume transfer, –append option can be used.

rsync --append A /Backup-A-dir/

This StackExchange thread is a very good resource to learn more on this topic.

All the above commands are basic and copies the file in a single flow or a process. If there is 5 TB of data and 1 TB transfer takes 2 hours, the whole transfer will last for 10 hours. But there is an another way to speed up the transfer

Bonus Tip: Speed Up Rsync / Parallel Transfer

parallel is a GNU utility used to execute jobs in parallel. Rsync can be easily coupled with rsync.

Install GNU parallel in Ubuntu/Debian system using

sudo apt install parallel

Before we move further, let us understand how parallel works. Let’s consider an analogy.

Assume that there are 1000 eggs and 100 baskets each with a drone. The task is to deliver it to the customers. Manager assigns each basket 10 eggs and orders the drone to deliver. Each drone does an operation (in our case rsync process of 10 files). Manager oversees all the drones. Note that drones will never know that there are other jobs in parallel. Only the Manager does.

Similarly, rsync executes the transfer of files like a drone. parallel acts as the manager.

parallel splits the files to be sent and spawns a specified number of rsync process each with a different list of files. parallel just manages all the process. rsync has no knowledge of other processes in parallel nor it has a functionality of parallel transfer. Parallel provides parallel transfer by bundling tasks.

So parallel command consists of two parts which are piped. One is arguments (eggs/files) and other is the parallel command(manager).

ls A/* | parallel -j 20 rsync A/{} /Backup-A-dir/

In the above command, each file/directory in output as a result of the first command before ‘|’ symbol acts an argument. {} represents the argument from the left in the parallel command.

-j n – It is used to set n number of jobs or workers. In our case, n is 20.
the next part is the usual rsync command for every argument.
After the commands are generated, they will be bundled into 20 processes and they will be executed in parallel.

Note that you can add any options to the above rsync command as you would do without parallel. Just add the rsync options (like -z,-a,-e ssh) after the word rsync in the above command.

If you want to understand more about parallel, refer this page.

Common Errors with rsync

You might encounter errors while using rsync. Below is some common ones along with their troubleshooting tips.

1. rsync permission denied

This is an error that might happen when you use paths for which you don’t have appropriate rights. For Example:

rsync B/ /home/

Above command will cause permission denied error, if you don’t have the write permissions on /home/(normal users usually don’t do)

2. rsync failed to set times on <path>

This happens when the file system can’t handle modification times for files and directories. You can learn more about this on this page.

I hope you learned some good examples of sync command in this article. If you have any suggestions or comments, please feel free to drop them below.