Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Overview of storage systems

The administration and troubleshooting slightly various across installations. The table below summarizes responsibilities. If in doubt it's always a good idea to contact uco@desy.de.

...

  1. For information about backup & recovery: check IT-Services
  2. AFS-snapshots are located in <afs-home>/.OldFiles
  3. GPFS Snapshots are located in /asap3/.snapshots/@<time-stamp>
  4. GPFS-Home snapshots are located in /home/.snapshots/@<time-stamp>/$USER

Where to store Scientific data

Available for everyone are AFS, BeeGFS, Netapp, Desycloud and dCache.

...

  • GPFS home-directories:  20GB hard limit, don't store data in the home-directory.
  • GPFS-FS resources: space dedicated to FS-experiments.
  • GPFS-CFEL resources: space dedicated to CFEL.
  • GPFS-EXFL resources: space dedicated to European XFEL.
  • IF your group needs large amounts of fast, secure storage space, a group-owned GPFS appliance might be a solution. Get in touch with maxwell.service@desy.de if you need to know more.

Where to store Software

Most applications are small enough to be deployed in (almost) arbitrary locations. Suitable storage systems are

  • AFS: globally accessible space. Good for software needed on Maxwell, Desktop, BIRD likewise. Keep in mind, that restrictive ACLs will cause problems on Maxwell! AFS is not suitable for multi-host application, it will have horrible side-effects on performance!
  • Netapp: will be retired. Don't start storing anything there.
  • GPFS-home: apart from the space limitations well suited for software installations. Don't use it for group-shared installations! 
  • For Applications shared within a group: use BeeGFS group directories, AFS or group specific resources where available.
  • DO NOT dCache for software installations.  Feel free to use BeeGFS, but be aware of the limitations.

Where to store Documents

Assuming that documents are usually small 

  • AFS: secure with flexible ACLs. Globally accessible. Prime choice for documents.
  • Desycloud: well suited for sharing documents. Globally accessible. Not convenient to use in the HPC environment. No backups whatsoever, make sure to secure any valuable or sensitive data.
  • Netapp: well suited but not accessible outside the HPC site.
  • Don't store documents on BeeGFS, dCache or GPFS (except for documenting experiments).

Where to store Temporary data

  • BeeGFS: designed as scratch it's perfect for large temporary data.
  • scratch: local space, but fairly limited volumes.
  • Don't store temporary data on dCache, AFS, netapp or GPFS.


Best Place(s) for storing information - overview


AFSBeeGFSdCacheDESY-CloudHOMENetapp

Local Scratch

GPFS-Scratch
scientific data--++----
temporary data-+----++
software+---+---
documents+--+----

...