ASAP3 : Online Analysis Resources

Introduction

In order to support near realtime analysis, computing resources from Maxwell can be reserved for a beamtime. These reserved resources are only available for the beamtime and also allow to access the beamline filesystem from the Maxwell node.
User access is only possible with a specific user account via the use of SSH public keys.

The reserved resources can be used interactively and/or via sbatch SLURM jobs. For additional details about SLURM, have a look at the Maxwell space.

These resources are reserved and allocated via the startBeamtime script. See below on how to reserve and use the resources.

Resource Allocation via startBeamtime

Online resources can be allocated from the startBeamtime script, either with Pre-Start or without Pre-Start. Additionally, it is possible to add options for resource allocation, e.g. nodes with GPUs or dedicated nodes for the beamline.

See startBeamtime --help for all available options:

% startBeamtime --help
usage: startBeamtime  [-h] [--beamtimeId BEAMTIMEID] [--beamline BEAMLINE]
                      [--verbose] [--pre-start] [--online]
                      [--feature FEATURE] [--resources PARTITION] [--asapo]
                      [--archive]

optional arguments:
  -h, --help            show this help message and exit
  --beamtimeId BEAMTIMEID
                        beamtime-id of your experiment
  --beamline BEAMLINE   beamline that belongs to the beamtimeId
  --verbose             more verbose output
  --pre-start           BETA: Reserve Maxwell Online Resources
  --online              BETA: Reserve Maxwell Online Resources without Pre-
                        start
  --feature FEATURE     BETA: Specify additional node features, only accepts
                        gpu
  --resources PARTITION
                        BETA: Specify resource location, either common or
                        dedicated
  --asapo               Initialize ASAP::O
  --archive             Enable Archiving for Industry beamtime

2 modes of resource reservation are available, with or without pre-start.

Resource Selection

The flags --resources PARTITION and --feature FEATURE allow you some basic resource selection.

If none of the flags are specified, a single CPU node is taken from the common resources.

With --feature gpu, you will get a single node with a GPU (nVidia P100 as of 2021-08-16).

Some beamlines have acquired Maxwell resources for beamtimes, which allows them to use more resources. With --resources dedicated, these resources will be used.
The reservation in Slurm will use all available nodes from these dedicated resources.

One notable exception are beamtimes with proposal type P aka Industrial Beamtimes. These beamtimes have to specify always --resources dedicated due to licensing constraints.

With Pre-Start

Pre-start is used before the beamtime starts. Storage resources are then only allocated on the core fileystem and a shared node from Maxwell will be allocated. This allows you to setup your scripts, pipelines etc. in advance of a beamtime.

See below how to start the beamtime

# startBeamtime --beamtimeId 12345678 --beamline p00 --pre-start
<snap>
2021-08-12 09:46:01,464 >  [Maxwell Online Analysis Resources]-----------------
2021-08-12 09:46:09,678 >  [Maxwell Online Analysis Resources]-----------------finished.
2021-08-12 09:46:09,685 >  Beamtime has been pre-started.
Maxwell Online Analysis Resources have been configured for preparation phase.

Storage resources were only created for the Core-Filesystem:
/asap3/petra3/gpfs/p00/2021/data/12345678

For information about the user account and associated SSH key, please check the
shared folder in the Core-Filesystem

Now you can access the beamtime via the core filesystem and start with preparation. Once the experiment and real beamtime starts, startBeamtime has to be called again:

# startBeamtime --beamtimeId 12345678 --beamline p00

This allocates the storage resources on the beamline filesystem and also mounts it on the Maxwell node. The reservation will be updated as well, so check beamtime-metadata-<beamtimeId>.json for changed reservation- and node names!

The SSH key is copied to the beamline filesystem in the shared folder. Any server with access to the beamline filesystem can now access the Maxwell resources as well.

Without Pre-Start

If you do not want to use the pre-start option, you can use the --online flag:

# startBeamtime --beamtimeId 12345678 --beamline p00 --online

This will skip the pre-start phase and start directly with the online phase.

Using Allocated Resources

After you have started your beamtime with Online Resources, first check the Beamtime Metadata File. The onlineAnalysis key contains all relevant information:

• Name of the reserved node(s)
• Name of the reservation, required to use for jobs with sbatch
• Beamtime specific account name
• Relative path to the SSH private key

To start using the resources, you must use the account specified in userAccount and the private key from sshPrivateKeyPath:

# ssh -l<userAccount> -i <sshPrivateKeyPath> <reservedNode>

The key is available on both, the beamline- and core filesystem. On the reserved node, the beamline filesystem is mounted on /beamline/<beamline>/current

You should always read and write data from the core filesystem. Only write small results back to the beamline filesystem, e.g. if they are needed for changing the setup of the running experiment.