Grid : CVMFS Repository/Stratum Metadata

Created by Thomas Hartmann, last modified on Aug 27, 2020 17:06

CVMFS supports meta-data information on a stratum repository levels. See an overview and documentation under

For example, for the hdf-cvmfs stratum 0 hosting the hdf.desy.de and containers.desy.de repositories the (added) release server infos and the list of hosted repos

It is in principle possible to query also a repository metadata JSON via HTTP, if one knows the the JSON's commit/hash (marked with a trailing capital M)

curl http://grid-cvmfs-null.desy.de:8000/cvmfs/containers.desy.de/data/01/e60338a2db9a71f5068a776ed2b96b512d4d60M | cvmfs_swissknife zpipe -d  #github

e.g., to search for the JSONs locally on the storage

  [root@hdf-cvmfs] find /storage/cvmfs/hdf.desy.de/data/ -name "*M"

optional: check also the other repo special files (the M${hash} variable should probably the the signed hash of the metadata JSON - no idea, if one can deduce the zipped JSON file from there...)

  cat /storage/cvmfs/containers.desy.de/.cvmfspublished

Release Node/Stratum-0 Metadata

To add the basic metadata on a fresh spawned stratum0/release node, run as root

  cvmfs_server update-info

and update the default JSON (the command seems not to read from a file - should only be necessary once on a new stratum-0, so we skip puppetifying it)

{
  "administrator" : "Grid@DESY",
  "email" : "grid@desy.de",
  "organisation" : "DESY",
  "custom" : {
    "default" : "none"
  }
}

Repository Metadata

The cvmfs_stratum0 puppet module writes on a release node for each locally configured repository a JSON file to

  [stratum-0] /etc/cvmfs/metadata/REPONAME.json

e.g.,

{
  "administrator" : "Grid@DESY DEBUG",
  "email" : "grid@desy.de", 
  "organisation" : "HDF/DESY",
  "description" : "Demonstrator Repository for the Helmholtz Data Federation",
  "url" : "https://confluence.desy.de/display/grid/DESY+CVMFS+Repositories",
  "recommended-stratum0" : "http://grid-cvmfs-null.desy.de:8000/cvmfs/",
  "recommended-stratum1s" : [ "http://grid-cvmfs-one.desy.de:8000/cvmfs/hdf.desy.de" ],
  "custom" : {
    "Puppet" : {
      "hiera" : "cvmfsstratum::meta::repo::hash",
      "module" : "module-cvmfs_stratum0",
      "template" : "module-cvmfs_stratum0/templates/etc/cvmfs/metadata/repo.json.erb"
    }
  }
}

if necessary, overwrite in Hiera the defaults with like

cvmfsstratum::meta:
    'REPONAME':
        'administrator' : 'Grid@DESY DEBUG'

set the repository infos on the Stratum-0 with

  cvmfs_server update-repoinfo -f /etc/cvmfs/metadata/REPONAME.json REPONAME.desy.de

(the update-repoinfo can read from file, so that we can probably puppetify it later on with an exec, if chanegs become frequent - probably not)

Adding a Repository to the global CVMFS Monitoring

Fork on Github the CVMFS monitoring (git) repository into your own account → https://github.com/cvmfs/monitor-repos

   git clone git@github.com:YOURGITHUBUSERNAME/monitor-repos.git

or (via HTTP)
     git clone https://github.com/thdesy/monitor-repos.git

in the repo ensure, that you add the original repo as upstream and get the recent changes from there, so that you don't diverge
     git remote add upstream git@github.com:cvmfs/monitor-repos.git
     git fetch upstream
     git merge upstream/master

add/commit the same CVMFS repo JSONs to your fork, push the changes as a new branch to github and then place a merge request on the original monitor-repos repository to merge your new branch into their master

  scp stratum0:/etc/cvmfs/metadata/REPONAME.json  git/MONITOR-REPOS.FORK/repos/
  git add repos/REPONAME.json
  git commit -a -m "added shiny metadata JSON for repo REPONAME"
  git push origin master:NEWPULLREQUESTBRANCH
  # open pull request on https://github.com/cvmfs/monitor-repos from your fork:BRANCH