ASAP3 : Overview of ASAP3 Storage at Beamlines

The ASAP3 storage is divided into two GPFS file-systems; the beamline file-system and the core file-system. The beamline file-system is optimised for the ingestion of data in high speed bursts, while the core file-system has been optimised for capacity and parallel concurrent access. Data taken during a beamtime is first written to the beamline file-system, and is then concurrently synced to the core file-system from where it can be accessed and analysed with a delay of a few minutes. Once the beamtime is stopped, the data corresponding to this beamtime can only be accessed from the core file-system.

Configuring the beamline file-system

For mounting the file system see Mounting the storage system at the beamline

Start/Stop a Beamtime

The Start/Stop-script can be executed from the beamline-staff groups, e.g. p11staff.

Start a Beamtime

ssh username@psana001.desy.de
 
[psana001.desy.de]$ startBeamtime --beamtimeId <beamtimeID> --beamline <beamline>

example for beamtime '1100362' on beamline P11

startBeamtime --beamtimeId 1100362 --beamline P11

This will create the following directories under your mount/share point:

beamline file-system

/gpfs/
├── local
├── current
    ├── raw
    ├── processed
    ├── shared
    └── scratch_bl
└── commissioning
    ├── raw
    ├── processed
    ├── shared
    └── scratch_bl

Meaning of the different directories:

raw:
- will be migrated into the core file-system
- will be written to tape
- will be shown in the Gamma-portal
processed
- will be migrated into the core file-system
- will be written to tape
- will be shown in the Gamma-portal
shared
- will be migrated into the core file-system
- will be written to tape
- will be shown in the Gamma-portal
scratch
- will not be migrated into the core file-system

Stop a Beamtime

ssh username@psana001.desy.de
 
[psana001.desy.de]$ stopBeamtime --beamtimeId <beamtimeID> --beamline <beamline>

example for beamtime '1100362' on beamline P11

stopBeamtime --beamtimeId 1100362 --beamline P11

The directory 'current' will be un-linked and thus the data will cease to be visible on the beamline file-system.

The data created during this beamtime remains visible in the core file-system under

example structure

/asap3/
└── <facility>
    └── gpfs
        ├── <beamline>
            ├── <year>
				├── data
                |    ├── <beamtime-ID>
                |       ├── raw
                |       ├── processed
                |       ├── shared
                |       └── scratch_cc
                └── commissioning
                    └── <commissioning-tag>
                        ├── raw
                        ├── processed
                        ├── shared
                        └── scratch_cc

example for beamtime '1100362' on beamline P11

/asap3/
└── petra3
    └── gpfs
		└── P11
	        └── 2015
    	        └── 1100362
					├── raw
					├── processed
					├── shared
				    └── scratch_cc