ASAP3 : Accessing beamtime data using FTP

FTP Download is deprecated since 2023-08-17

A new and easier to use download method is available: Accessing Beamtime Data using SFTP (Recommended)
The FTP Registration procedure with the Gamma Portal is NOT required for the new SFTP method.

If you have been using FileZilla to download data, follow these steps to migrate your client configuration: Client Migration from FTP to SFTP

The FTP download method will remain available until 2024.

How it works

A FTP server provides interactive access for DOOR users to beamtime directories using the FTP protocol. It allows you to browse and download beam time data using any suitable FTP client. Access is granted according to the working group definitions in the Gamma Portal. As we are using a secure implementation of the FTP protocol, you might have to configure your client. You can find configuration examples below, see Configuring FTP clients.The FTP server uses a port range of 30000 - 32000 to transfer data. These ports need to be open for outgoing traffic at the receiving site, wich is usually the case.

To access your data:

Use the 'FTP Configuration' menu entry of the Gamma Portal to select a specific beam time you want to access and click 'FTP register'. If you do not select a beam time, the FTP server will point you to a README file. See 'testing the service' below.
Use a suitable FTP client to connect to psftp.desy.de. We tested our setup using filezilla,lftp anf fireftp. See Configuring FTP clients below for some information on these clients.
Use your DOOR account to authenticate to the FTP server. DOOR user names and passwords are case sensitive.
If you want to access a different beam time, simply log out of the FTP server, use the 'FTP Configuration' menu again to de-register the old beam time and register for the new one, then log in to the FTP server and thus to the new beam time.

Testing the service without having done a beam time yet:

If you have a DOOR or DESY account, you can verify that the service works for you.
Just connect to the service using a FTP client e.g. filezilla.
Right-klick on the file named README.txt
A text editor should open and display a short hint.

Configuring FTP clients

Access to psftp.desy.de is encrypted to protect your privacy. To connect to the server for the first time, you need to accept the servers certificate and - depending on the client you use - the certificates of the signing organizations. We tested the following clients and describe the steps to take:

FileZilla (Linux, macOS, Windows)

Features

filezilla is a GTK based GUI.
It works out of the box (you must accept the certificate) on at least Fedora >= 36, Ubuntu >= 18.04, Debian >= 9, Windows >= 10, CentOS Linux 7 and AlmaLinux >= 8.
You can browse directory structures on both the FTP server and your local machine
You can view files like text files or JPEGs. In fact you can configure it to display the contents of any file for wihch you have a display program
It has a queue concept allowing you to start and stop transfers
It can recursively download directory trees
It can to several transfers simultaneously

Configuration

Use the generic address psftp.desy.de for the file server
Configure it to use explicit TLS when connecting to psftp.desy.de.
Accept the certificate by clicking ok.

Usage example

When you are logged in to your beamtime, you can use filezilla to browse directory structures, recursively download directory trees and even look at single files. Here is an example of the GUI.

lftp (Linux)

Features

lftp is a command line client available with most Linux distributions.
You can browse directory structures
It can be used in batch mode or within scripts and is able to copy or sync whole directory trees.
You can use the cat command to view text files
It is very robust when doing transfers

Configuration

lftp now should work out of the box for new users.

Due to a change of our CA, users that configured lftp by providing part of the (old and now obsolete) certificate chain now have to remove this by:

cd ~/.lftp/certs
# Remove the trust chain for DFN issued certificates
rm desy.pem
cd ~/.lftp
vi rc 
# ---> now remove (or comment) the line
set ssl:ca-file "~/.lftp/certs/desy.pem"' 
# exit vi

The servers welcome message might hint to maintenances or other events of interest. If you do not see the message, you can add a line containing debug 3 to your lftp configuration file, ~/.lftp/rc. An example file might look as follows:

debug 3

Usage example

lftp can be used to synchronize whole directory trees.

Notes

Debian 8 needs another line in the configuration file: "set ftp:ssl-protect-list no" (omit the "'s..)
The server is now IPv6 enabled. On some client OSes (e.g. Red Hat Enterprise Linux 6), lftp gets stuck after the login prompt. Here is how you can check if your client is affected. The solution is to put the following line to your lftp configuration file (~/.lftp/rc) :
```
set dns:order 'inet inet6'
```
which would then look like
```
set ssl:ca-file "~/.lftp/certs/desy.pem"
set dns:order 'inet inet6'
debug 3
```

ASAP3 : Accessing beamtime data using FTP

How it works

Configuring FTP clients

FileZilla (Linux, macOS, Windows)

Features

Configuration

Usage example

lftp (Linux)

Features

Configuration

Usage example

Notes

Attachments: