ASAP3 : Accessing commercial beamtime data using rsync or SFTP

How it works

For industrial users, an alternative download mechanism has been established.
Instead of using the recommended SFTP download mechanism, rsync over SSH or SFTP can be used.

As both protocols use SSH as the transport mechanism, only port 22 has to be allowed for outgoing traffic.

Please note, you can only download data via this mechanism. Creating or uploading data is not possible.

Requisite

• Valid DESY account in the COMMERCIAL namespace
• Registry resource ps-export has to be granted
  → please contact ITT in order to create the DESY account and granting the resource
  → resource should only be granted to COMMERCIAL users
• Allow outgoing traffic on port 22 to desy-ps-export.desy.de in your home institute
• Read access on desired beamtimes to download
  → the access is managed by the Gamma Portal

SSH Key Fingerprint

The host desy-ps-export.desy.de has the following SSH fingerprint:

You should compare the received fingerprint with the documented fingerprint here, in order to be sure you are connecting to the correct machine.

Testing Access

If the conditions are met, please try first to login interactively on desy-ps-export.desy.de from your home institute via PuTTY (Windows) or SSH (any Linux machine)

If you receive an interactive shell, you have successfully established a connection to the download server.

To test the download before any real beamtime has been started, you can download a test file in /home/public/README.txt

The ASAP³ core filesystem is mounted under /asap3 (Directory Structure) and lists all available beamtimes.

To check if you can already access the beamtime, you can interactively browse the directory structure.
If you receive a permission denied for the directory of your beamtime, you do not have access and access has to be granted via the Gamma Portal

Notice: The home directory on the download server is limited by a quota of 500 MB and there is no backup available!
Do NOT store any important data on your home directory!
The login with SSH public keys from external is disabled, as this is not allowed by policy.

Examples

Linux: rsync over SSH

In order to use rsync over SSH from a standard Linux box, you need the rsync command line tool installed.

To download a beamtime via rsync, you can use the following command on your machine.

For a detailed explanation of the --archive option of rsync, please refer to the man page: man rsync

Windows: WinSCP

To download the data from Windows machine, WinSCP can be used to download the data.

Install WinSCP on your local machine, afterwards you can use the following guide:

1. Start WinSCP and create a new site. As a protocol, use SFTP
   
   
2. In the next Window, compare and accept the fingerprint
   
3. You can now browse the available data on the right side