CERN is blocking external access to their AFS cell between Oct.25 and Nov.01 as smoke test to identify remaining dependencies before dropping AFS for good.
https://cern.service-now.com/service-portal?id=outage&n=OTG0066507
We have received various reports from users, who observed problems during this time frame. Apparently primarily during compilation/linking, processes seem to got stuck.
While no obvious dependencies on /afs/cern.ch appeared - like open file handles - users identified tools, that bought pseudo-dependencies on the paths. For exmaple, a git shell extension seems to try to walk through the namespace and got stuck, when looking up /afs/cern.ch, which became prominent during linking.
Blanking out the CERN AFS path seems to 'fix' most issues, i.e., bind mounting an empty directory over /afs/cern.ch, so that it appears to be empty and lookups do not block
Collecting Observations
For further debugging, we would like to collect observations and information about the tools and environments in use, that might stumble over CERN's AFS cell not responding.
While most obvious problems like environment paths to /afs/cern.ch/... are more easy to spot, please also have a look for your tools you use (git, Eclipse,...), that might try funny things like walking through the namespace.
→ Please use the page >> User Observations << to let us know your observations.
gcc compiler from sft.cern.ch
gcc compiler releases/versions in /cvmfs/sft.cern.ch seem to carry over dependencies on /afs/cern.ch - which becomes a problem, if an experiment's setup sources a gcc version from the sft repo.
E.g., grep'ing brutally through the gcc releases reveals a number of /afs/cern.ch references in libraries, setup scripts (and install logs)
> grep -r "/afs/cern.ch" /cvmfs/sft.cern.ch/lcg/releases/gcc/
Binary file /cvmfs/sft.cern.ch/lcg/releases/gcc/4.8.1/x86_64-slc6/lib64/libgmpxx.so.4.3.1 matches
/cvmfs/sft.cern.ch/lcg/releases/gcc/4.8.1/x86_64-slc6/lib64/libgmp.la:libdir='/afs/cern.ch/sw/lcg/external/gmp/5.1.1/x86_64-slc6-gcc48-opt/lib'
...
Binary file /cvmfs/sft.cern.ch/lcg/releases/gcc/4.8.1/x86_64-cc7/lib64/libobjc.a matches
...
Binary file /cvmfs/sft.cern.ch/lcg/releases/gcc/4.8.1/x86_64-cc7/bin/g++ matches
...
/cvmfs/sft.cern.ch/lcg/releases/gcc/4.8.1/x86_64-cc7/setup.sh:then LCG_contdir=/afs/cern.ch/sw/lcg/contrib
...
probably also newer releases are similarly tainted (letting grep walk and download through the gcc subtree)
- for `8.3.0-cebb0` or `9.2.0-afc57` I find only afs references in the install.logs - the dependencies have to seep into from somewhere else...