Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

After creating files on the DUST scratch disk (i.e., somewhere under /nfs/dust/atlas/), you might want to upload it to ATLAS' Rucio namespace, so that colleagues at other institues can copy/replicate your data.

Prerequisites

On a NAF workgroup server, you will need the ATLAS environment to be setup including the rucio client

> export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase

> source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh

> lsetup rucio

and to upload your data, you also need a valid grid proxy

> voms-proxy-init -voms atlas

Creating a dataset

If you have not have already a Rucio dataset, i.e., a container for your files, you can create a new one with something like

> rucio add-dataset user.thartman:mytestdataset

with the syntax user.<YOURATLASNAME>:<NEWDATASETNAME>

Uploading Files to a grid storage

As the DUST space is only available internally to DESY/NAF-users, we have to upload files to a Rucio Storage Element in the Grid.
To upload a file called "ubuntu_sandbox.tar.xz" to my Rucio scope "user.thartman"

> rucio upload ubuntu_sandbox.tar.xz --scope user.thartman --rse DESY-HH_SCRATCHDISK

here we used the DESY-HH_SCRATCHDISK storage element to send the file to, because it is close by in the network and is a scratchdisk to transient data. For a list of all available storage elements check out

> rucio list-rses

Please ask your group and the ATLAS data management team, what storage to use - especially if you plan to use a lot of data or want to keep your data over longer time periods.
Else you might get angry emails fast.

Attaching a file to a dataset

The uploaded file is now on the choosen Rucio Storage Element, but not yet attached to the previously created dataset. But we might want to collect a number of files in one dataset, so that we can easily ship the whole collection within the ATLAS collaboration with replication rules etc. .

So, we attach the file to the dataset

> rucio attach user.thartman:mytestdataset user.thartman:ubuntu_sandbox.tar.xz

and can further operate with the dataset as adding further files or setting up replication rules to other Rucio Storage Elements.

Uploading a directory and registering it directly to a dataset

The separate steps to create a dataset, upload a file and attach the file to a dataset can be tedious.
Fortunately, rucio uploads supports also a bulk upload with attaching the files to a dataset; unfortunately the syntax is a bit cumbersome:

> rucio upload --rse DESY-HH_SCRATCHDISK --scope user.thartman user.thartman:anotherdataset MYDIRECTORY.d/

would upload files in a directory MYDIRECTORY.d/ to my scope user.thartman and register it in a dataset user.thartman:anotherdataset.
Please note, that this is not recursive - only files in MYDIRECTORY.d/ will be uploaded but no files in sub-directories of MYDIRECTORY.d/