There a number of storage systems attached to the cluster. As a very rule of thumb:
- Store everything of crucial importance in your home-dir or experiment-specific GPFS folder when available.
- Archive data in dCache if available for your group.
- Store everything easily reproducible and in particular when of temporary nature in /beegfs/desy
More details can be found below
IMPORTANT
Any files without OWNER (account does not exist anymore) will be removed from DESY BEEGFS, HOME, GPFS-FS-SCRATCH without prior notifications! Similar policies apply for group-storages, please verify with your admins! if in doubt!
Overview of storage systems
The administration and troubleshooting slightly various across installations. The table below summarizes responsibilities. If in doubt it's always a good idea to contact uco@desy.de.
| Filesystem | Remarks | Mounts | Contact |
|---|---|---|---|
| AFS | Good for static data/documents and software. Not good for multithreaded applications and especially batch jobs. | /afs/desy.de/ | DESY IT uco@desy.de for questions about increasing quote, general problems |
| dCache | Good for mass storage of scientific data. Not suitable for volatile data. | /pnfs/desy.de/ | DESY IT maxwell.service@desy.de to arrange (missing) mounts. uco@desy.de for general questions |
| DESY beegfs | Reasonably fast. Without backup or snapshots. Deleted files are not recoverable. Not intended for archiving data, only for actively used data! It's scratch space really! Do not use more than 10 TB over a longer period (more than 2 weeks). If you have an ongoing project that requires more space in the long term, please email us | /beegfs/desy/ | DESY IT Note: execute mk-beegfs on maxwell will create /beegfs/desy/user/<userid> . Ignore LD_PRELOAD error messages if you get any ... Contact maxwell.service@desy.de for all other issues. |
| sync&share | Good for data sharing. | none | DESY IT uco@desy.de for all issues |
| CFEL GPFS | Fast. Good for scientific data of CFEL. | see below | CFEL IT / DESY IT DESY CFEL admins for access/usage issues. maxwell.service@desy.de for technical issues |
| CSSB GPFS | Fast. Good for scientific data from CSSB. Lifetime of data is up to CSSB policy. Get in touch with CSSB admins for details! | /gpfs/cssb | CSSB IT / DESY IT it@cssb-hamburg.de for access/usage rights & quota management. |
| EXFEL GPFS | Fast. Good for scientific data from EuXFEL experiments and analysis. | see below | EXFEL IT / DESY IT EXFEL admins for access/usage issues.maxwell.service@desy.de for technical issues |
| FS GPFS | Fast. Good for scientific data from PETRA III and FLASH experiments and analysis. | see below | ASAP3 Team it-asap3@desy.de ASAP3 team for all issues. |
| Scratch / TMP | Scratch. Limited space and subject to erasure without prior notice. | /scratch /tmp | DESY IT Unmanaged temporary space. |
| HOME | Fast. With snapshots. Best space for everything important and reasonably small. | /home | DESY IT maxwell.service@desy.de for all issues. Quota will not be extended! |
Characteristics of storage systems substantially differ for different storage systems. Tables below give a rough overview. Contact maxwell.service@desy.de for open questions.
| Filesystem | Mount Point | Quota | Size | Lifetime | tokens | Backup | Snapshot | Network | Protocol | Throughput | Availability | Remote |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AFS | /afs/ | per Volume | per Volume | unlimited | yes | yes1 | yes2 | eth | afs | slow | yes | yes |
| BeeGFS | /beegfs/desy/ | no (see Remarks) | 1.2PB | unlimted | no | no | no | ib | beegfs | >10GB/s | yes | no |
| dCache | /pnfs | no | unlimited | unlimited | no | no | no | eth | nfs | varies | yes | yes |
| sync&share | none | 90TB | unlimited | unlimited | no | no5 | no | eth | https | slow | webdav sync-client | yes |
| GPFS CFEL | /gpfs/cfel | no | 1.6PB | no | no | yes | ib | gpfs | >10GB/s | CFEL only | no | |
| GPFS CSSB | /gpfs/cssb | yes initial: 50 TB | 10.1PB | unlimited | no | no | yes3 | ib | gpfs | >10GB/s | CSSB only | no |
| GPFS EXFL | /gpfs/exfel/d | 18PB | ||||||||||
| /gpfs/exfel/data | 1.1PB | |||||||||||
| /gpfs/exfel/exp | 42GB | |||||||||||
| /gpfs/exfel/u | 2.8PB | |||||||||||
| GPFS FS | /asap3/petra3/gpfs | no | 14.7PB | long | no | yes | yes4 | ib | gpfs | >10GB/s | FS only | no |
| /gpfs/petra3/scratch | no | 40TB | 3 month | no | no | no | ||||||
| /gpfs/petra4/scratch | no | 120TB | ||||||||||
| scratch | /scratch | no | few GB | none | no | no | no | - | local | fast | yes | no |
| HOME | /home on GPFS | 30GB hard | 30TB | account | no | no | yes5 | ib | gpfs | >10GB/s | yes | no |
IMPORTANT
Any files without OWNER (account does not exist anymore) will be removed from DESY BEEGFS, HOME, GPFS-FS-SCRATCH without prior notifications! Similar policies apply for group-storages, please verify with your admins! if in doubt!
IMPORTANT: Snapshots
- For information about backup & recovery: check IT-Services
- AFS-snapshots are located in <afs-home>/.OldFiles
- GPFS CSSB Snapshots are located in /gpfs/cssb/.snapshots/@<time-stamp>
- GPFS-ASAP3 Snapshots are located in /asap3/.snapshots/@<time-stamp>
- GPFS-Home snapshots are located in /home/.snapshots/@<time-stamp>/$USER
- Please have a look at https://it.desy.de/services/storage_services/desy_sync__share/index_eng.html for details on backup and security measures on Sync&Share.
Where to store Scientific data
Available for everyone are AFS, BeeGFS, Desycloud and dCache.
- AFS is secure, but suitable only for rather small data volumes.
- Desycloud offers significantly more space, but uploads or downloads are not very fast.
- DESY-BeeGFS and GPFS-Scratch are fast, but exclusively for temporary data without any level of security. Please note the policy!
- dCache: is the only option for long-term storage of larger amounts of data. If your group doesn't have dCache storage space but would like to "buy in", get in touch with osm.service@desy.de .
- GPFS home-directories: 30GB hard limit (can't be increased), don't store data in the home-directory.
In addition to generic storage resources, some group specific resources are available:
- CSSB BeeGFS: space dedicated to CSSB
- GPFS-FS resources: space dedicated to FS-experiments.
- GPFS-CFEL resources: space dedicated to CFEL.
- GPFS-EXFL resources: space dedicated to European XFEL.
- IF your group needs large amounts of fast, secure storage space, a group-owned GPFS appliance might be a solution. Get in touch with maxwell.service@desy.de if you need to know more.
Where to store Software
Most applications are small enough to be deployed in (almost) arbitrary locations. Suitable storage systems are
- AFS: globally accessible space. Good for software needed on Maxwell, Desktop, BIRD likewise. Keep in mind, that restrictive ACLs will cause problems on Maxwell! AFS is not suitable for multi-host application, it will have horrible side-effects on performance!
- GPFS-home: apart from the space limitations, well suited for software installations. Don't use it for group-shared installations!
- BeeGFS: For applications shared within a group use group directories. Contact maxwell.service@desy.de if you need a new group directory.
- dCache DO NOT use for software installations.
Where to store Documents
Assuming that documents are usually small
- AFS: secure with flexible ACLs. Globally accessible. Prime choice for documents.
- Desycloud: well suited for sharing documents. Globally accessible. Not convenient to use in the HPC environment. No backups whatsoever, make sure to secure any valuable or sensitive data.
- GPFS-home: secure & fast, but limited in space and not easily accessible outside maxwell
- Don't store documents on BeeGFS, dCache or GPFS (except for documenting experiments).
Where to store Temporary data
- BeeGFS: designed as scratch, it's perfect for large temporary data.
- scratch: local space, but fairly limited volumes.
- Don't store temporary data on dCache or AFS.
Best Place(s) for storing information - overview
| AFS | BeeGFS | dCache | DESY-Cloud | HOME | Local Scratch | GPFS-Scratch | FS/EuXFEL GPFS | |
|---|---|---|---|---|---|---|---|---|
| scientific data | - | - | ++ | (o) | - | - | -- | ++ |
| temporary data | - | + | -- | - | - | + | + | (o) |
| software | (o) | (+) | - | - | + | -- | - | + |
| documents | + | -- | - | + | + | -- | -- | - |