Page tree

This page gives some examples for simple file transfer tasks using the command rsync.

The advantage of rsync over cp or scp is that if you continue it copies just the changes and new files since the last call.

The examples on this page always add or replace files to the target folder. No files are deleted!

Note: Replace username below with your DESY login name (this is NOT the email address!).

Scenario 1: Copy on the same computer/node/host from one directory to another

This scenario is commonly used for a backup to another filesystem e.g. dCache backup or to an attached external harddrive.

To copy a complete folder to the current directory call:

rsync -av /beegfs/cssb/user/username/MyProject .

This will create a 1:1 copy of the MyProject folder at the current location. Please note there is no / at the end of the path!

Caution: Take care when you use command line completion (with the Tab key) like in the following example:

rsync -av /beegfs/cssb/user/username/MyProject/ .
rsync -av /beegfs/cssb/user/username/MyProject/* .

Both commands above do the same and copy the contents of the folder to the current location! This means there will be no top-level MyProject directory and just the files and subdirectories are created at the current location.

Using rsync on Windows

The rsync command is also installed at most MS Windows-based microscope transfer PCs. The only difference to the examples above is that Windows uses drive letters like C:\. For Unix compatibility they are changed to e.g. /cygdrive/c using Cygwin bash and rsync on the Windows PCs. So the following example assumes that you mapped your BeeGFS project folder as drive Z:\ and your data arrive at drive X:\ (e.g. K3 data). Now you can transfer the data like:

cd /cygdrive/z
rsync -av /cygdrive/x/supervisor_20191009_111635 .

Another option for Windows 10 users would be the Linux Subsystem from the Microsoft store. If you start the bash from the icon on your desktop you are in a special directoy. Your normal Windows 10 user directory is located at (replace username with your Windows login name):

cd /mnt/c/Users/username

Unlike the Cygwin implementation you do not see shared or mapped network drives as /mnt/driveletter! So you must use one of the scenarios described below.

Scenario 2: Copy data from a remote server to a local folder

This scenario is for external CryoEM facility users or if you want to transfer some data to your local analysis PC.

The syntax used is nearly the same except you have to add the username and servername for the copy process. The character : is important for rsync to determine the URL and switch to remote transfer mode. The servername can be any accessible server in the Internet supporting access via ssh.

To copy a complete remote folder to the current local directory call like:

rsync -av username@max-cssb-display.desy.de:/beegfs/cssb/user/username/MyProject .

Scenario 3: Copy local data to a remote server

To copy a complete local folder to the remote directory call like:

rsync -av ./MyNewFolder username@max-cssb-display.desy.de:/beegfs/cssb/user/username

Then this folder can be accessed as /beegfs/cssb/user/username/MyNewFolder.

If you do not provide a full remote path the data will go to your home directory like:

rsync -av ./MyNewFolder username@max-cssb-display.desy.de:

Then this folder can be accessed as /home/username/MyNewFolder.

Public analysis PC

The public analysis PC has a 9TB local drive for processing data which is available to everybody as /data.

The data must be transferred to this space:

rsync -av MyFolder dataprocessor@cssbhpi047.desy.de:/data
  • No labels