Maxwell : Transferring Data from and to Maxwell

For many use cases rclone is a very good choice to transfer data from or to the Maxwell cluster. It allows to mount webdav instances like Sync & Share in user space. See the very good documentation on rclone.org. For some example configuration for DESY Sync & Share have a look at the corresponding documentation.

The nextcloud-client and nextcloudcmd are not supported on the cluster anymore. Use rclone or the various ways to mount Sync & Share in userspace (see below). For handling shares through the Nextcloud share API have a look at curl and Sync&Share for some basic examples.

Mounting Windows and DesyCloud Directories on Maxwell login nodes

It's beneficial to have an access token and a .netrc-file, see below. If present, it goes in essence like this:

[@max-display002 ~]$ mount-desycloud
... successfully mounted  davs://desycloud.desy.de/remote.php/dav under /run/user/..../gvfs/dav:host=desycloud.desy.de,ssl=true,prefix=%2Fremote.php%2Fdav/files/user
... accessible under /tmp/user/desycloud/

[@max-display002 ~]$ rsync -v ~/BLABBER /tmp/user/desycloud/

[@max-display002 ~]$ umount-desycloud 
Unmounted davs://desycloud.desy.de/remote.php/dav  

Accessing remote data with Thunar

XFCE comes with Thunar as the default filemanager. You can start thunar from the command line or via the filemanager icon in the panel. Once done just type the URLs mentioned above into the location folder. Opening several windows one can transfer files for example from windows homedir to desycloud by dragging and dropping folder.

Accessing remote data with SSHFS

On Unix based hosts you can use SSHFS to mount a remote folder on your local desktop without any admin rights. For example on your local Ubuntu desktop (make sure fuse and sshfs are installed!):

[@localhost ~]$ mkdir -p $HOME/Maxwell                          # IF $HOME resides in AFS use some local FS (e.g. /scratch; /tmp) 
[@localhost ~]$ sshfs max-wgs:/ $HOME/Maxwell                   # Makes everything from Maxwell available on your local desktop. 
[@localhost ~]$ fusermount -u $HOME/Maxwell                     # Remove the mount
[@localhost ~]$ sshfs max-wgs:/home/$USER $HOME/Maxwell         # Just mount the maxwell home-directory 
[@localhost ~]$ fusermount -u $HOME/Maxwell                     # Remove the mount
# sshfs has a number of option. In particular enabling compression might be helpful for larger transfers
[@localhost ~]$ sshfs -h 
usage: sshfs [user@]host:[dir] mountpoint [options]

general options:
    -o opt,[opt...]        mount options
    -h   --help            print help
    -V   --version         print version

SSHFS options:
    -p PORT                equivalent to '-o port=PORT'
    -C                     equivalent to '-o compression=yes'
[...]

On Windows hosts options are currently somewhat limited. Best choices are WinSCP and Cyberduck. Both do not allow to map network drives though. Microsoft WSL has no plans to support fuse, which will make WSL largely unusable for this kind of approach.  You can however access some GPFS-folder from windows through an smb-export.