Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...



Contents

Table of Contents
maxLevel1

Expand

Table of Contents
l




Production Plans

  • MC12 - mostly finished
  • proc9 - ongoing
  • SKIM12 - first part finished. Second part to start soon
  • The Phase III data will also be skimmed, as soon as proc9 is ready on the grid.
  • More realistic beam background overlay files (simulated) are in preparation (
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDP-1609
    ). Once these are prepared, we will plan to have a new MC campaign. More details will be announced soon.

Production Status

Date: Mon, 22 Jul 2019 11:19:24 -0500 Reprocessing of all events for proc9 is ongoing now, both at KEKCC and on the grid. Details are available in https://agira.desy.de/browse/BIIDP-1587. Grid processing is perhaps a bit less. Most MC12 samples are now finished. Additional signal requests will be accepted, but jobs are not likely to saturate the system for a while, at least. The first skim campaign for MC12 is now complete. Details of available analysis skims are given at https://confluence.desy.de/display/BI/MC12+Skim+Production. The second MC12 skim campaign, which will include charm and semileptonic FEI skims, is currently under preparation and will be processed soon. Please contact your skim liaison if you have any further skim requests.

Data Production Status

  • Raw data processing
    • proc10/bucket8: complete
    • Proc11 (2019a/b/c) will be launched around 17th Apr. 2020
    • Prompt processing of 2020a/b data will start in late spring, in preparation for summer conferences.
  • MC13 production

    • MC13a (Run-independent MC) production: Keep producing both generic samples and signal samples

    • MC13b (Run-dependent MC) production: ongoing
  • Skim
    • SkimP10x1 (Proc10 skims): ongoing
    • SkimM13ax1 (MC13 skims): ongoing

Production Status

Full resource usage

Data production summary page : Data Production Status

Data (re)processing:

  • No jobs to run on grid now

MC production:

  • MC13a/MC13b productions are ongoing

Analysis skimming:

  • SkimP10x1/SkimM13ax1 are on-going.


Central Services

Dirac (dirac.cc.kek.jp, b2dchsv01-b2dchsv06.cc.kek.jp, b2dchsv08.cc.kek.jp)

  • DateData, Issue, Tickets...

DB Production (b2dchdb1.cc.kek.jp, b2dchdb2.cc.kek.jp, b2dcsdb1.cc.kek.jp, b2dcsdb2.cc.kek.jp)

  • Date, Issue, Tickets...

"Web" servers

  • 2019-12-07 Ganglia Monitor for the "Web" servers still shows "remnant plots" after 2hours` check.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2166


Anchor
DDM
DDM

DDM (bldirac01.sdcc.bnl.gov)

  • 2018-03-01 DDM deletion task seems stuck
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-808

Conditions DB ()

Monitor

Issue in access to DIRAC Web Portal

LFC

  • Date, Issue, Tickets...

File Transfers and Replication Status

  • See also 120357662 for related issues.
  • 2020-04-03 00:15 UTC File Transfer failures: Pisa-DATA-SE 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2349
     
  • 2020-04-02 21:20 UTC  Activity restarted since 2020-04-02 18:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2273
  • 2020-04-02 16:555 UTC  No activities in both Throughput and Successful transfers have been observed since 2020-04-01 23:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1247

LFC

  • Date, Issue, Tickets...

File Transfers and Replication Status

  • 2273
  • 2020-03-20 21:15 UTC    No activities in both Throughput and Successful transfers have been observed since 2020-03-20 15:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2273
  • No activities in both Throughput and Successful transfers have been observed since 2020-02-06 01:00 UTC. 

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2273

  • There is no activity during the last three hours (01/Jan/2020 since 9:00 to 12: 15 UTC) in the "Replication Status plot"
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2097
  • There is no activity in the last two hours in both plots "Throughput" and "Successful transfers"  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2144
  • No activity in file transfer monitoring since 01/Mar/2020 at 19:00 UTC 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2097

FTS

  • Any problem in the FTS service or FTS monitoring are to be recorded here. Site/SE specific issues are to be recorded under each SIte/SE
  • Note that the FTS dashboard we use is an "old" instance and not well-maintained. We, Belle II members in general, do not have access to the "new" monitoring. When the dashboard is down, the shifters just need to notify the expert and skip the corresponding part of their work. The expert should check the new monitoring, for the access to the monitoring page is limited.
  • 2020-02-06 15:20 UTC File transfer failure from Roma3-TMP-SE to KIT-TMP-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2275
  • 2020-01-02 13:20 UTC File transfer failure from KMI-TMP-SE and from KEK-Disk-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219
  • 2020-01-02 9:15 UTC File transfer failure from KMI-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219
  • 2019-08-31  File transfer failures for past 48 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1987
  • 2019-09-03 File transfer failures for past 24 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1988

Replication Status

  • 2020-04-02 08:00 UTC, Problems on RepTrend:CESNET-TMP-SE and on RepTrend:KEK2-TMP-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2347
  • 2020-03-30 01:00 UTC and 2020-04-02 01:00 UTC, RepTrendAll: all lines at zero
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2342
  • 2020-03-20 19:30 UTC  - No activity
  • 2020-02-13 Zero number of "Done" at all SE; the number of Scheduled is increasing
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2286
  • 2019-12-21 Decreasing Done Jobs with many Scheduled Jobs 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2169
  • 2019-12-15 Zero Replication Efficiency 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2183
  • 2019-1-19 almost zero done, with a increasing numbers of scheduled jobs for more than 5 SEs and more than 5 hours.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1618
  • 2018-07-02   No Donetransfer,  several scheduled and rapid increase of Waiting replication

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1125

Job Status Plot

  • Date, Issue, Tickets...
  • No job status plots for 15 sites while MC13a production is ongoing 2020-02-07

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2277

Job Summary

  • Date, Issue, Tickets...
  • Following JIRA ticket updated :
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1553



SEs

SE Common Issues

  • Issues with individual SEs should be recorded below (Primary SEs or Other SEs).

Raw data

SEs

SEsLink to JIRA ticket: 

Raw data SE: KEK-RAW-SE (srm://kek2-se02.cc.kek.jp:8444/srm/managerv2?SFN=/belle/RAW)

  • 2019-07-24 15:30 UTC: all transfers failed (0/72) between KEK-RAW-SE and BNL-TMP-SE

Raw data SE: BNL-TAPE-SE (srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape)

  • date, issue, tickets

Primary SEs

Primary SE: BNL-TMP-SE (dcblsrm.sdcc.bnl.gov)

  • SE Health check by DDM : download, upload do not work since 2019-07-29 02:51:59 UTC High failure rate as source BNL-TMP-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2340

    Solved and Verified https://ggus.eu/index.php?mode=ticket_info&ticket_id=146329
  • 2020-02-09 File transfer failure from SIGNET-TMP-SE to KEK-DISK-TMP-SE, BNL-TMP-SE and DESY-TMP-SE have been low, ~50 %, for > 4 hours.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-
    1929
    2280
  • SE Health check Check by DDM: download, upload do not work since 2019-07-28 23:07:23 UTC. Failure on download have been observed since 2020-02-07 09:19:40 (5 hours)
  • No Replication Trend Plot for BNL-TMP-SE 2020-01-02 09:30 UTC 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-
    1929
    2220
  • SE Health check by DDM : download

    , upload do

    does not work since 2019-05-16 07

    -21 14

    :

    09

    11:

    11

    21 UTC.

     

  •  UNAVAILABLE files
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-
    1929GGUS ticket https://ggus.eu/?mode=ticket_info&ticket_id=142410 already exists
    1302
  • SE Health check by DDM : download , upload do does not work since 2019-0705-15 21 14:0903:11 UTC. 25 UTC

Primary SE: CESNET-TMP-SE (dpm1.egee.cesnet.cz) 

  • SE Health Check by DDM: Failure on upload have been observed since 2020-03-27 09:35:52 (3 hours) 
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-
    1929GGUS ticket
    2335

     https://ggus.eu/index.php?mode=ticket_info&ticket_id=142410 has submitted
  • SE Health check by DDM : download does not work since 2019-05-16 07:11:21 UTC.

  •  UNAVAILABLE files 146324
  • Replication status: Scheduled Jobs Only 2020-03-19 15:50 UTC
  • Replication status. Plot is not shown 2019-12-31 9:25 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1302
  • SE Health check by DDM : download does not work since 2019-05-15 21:03:25 UTC
Primary SE: CESNET-TMP-SE (dpm1.egee.cesnet.cz)
  • 1352
  • Replication status. Plot is not shown 2019-12-31 7:45 UTC
  • Plot is not shown 2019-12-25 05:00 UTC.
  • SE Health check by DDM : remove file, remove directory, ls do not work since 2019-07-10 06:32:47 UTC.

Primary SE: CNAF-TMP-SE (storm-fe-archive.cr.cnaf.infn.it)

  • SE Health check Check by DDM: remove file does not work Failure on download have been observed at since 2019-05-13 08:27:21 UTC .and since 2020-02-19 15:44:39 (7 hours)
  • File transfer failure for source have been observed since 2019-12-23 02:00 UTC
  • SE Health check by DDM : remove file, remove directory, download, upload, ls do not work since 2019-04-25 23:13:00 UTC.
  • 2019/04/11: File transfer failures from NTUCC_DATA_SE to CNAF-TMP-SE, Updated
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1637
    2019/01/27 File transfer failures from CNAF-TMP-SE to NTUCC-DATA-SE.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1637
  •  Cotinuous timeout failure between NTU-CC-TMP-SE and CNAF-TMP-SE

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1310

Primary SE: DESY-TMP-SE (dcache-se-desy.desy.de)

  • 2019/07/22: Many file transfer failures from this SE since at least 24h 2020-02-09 File transfer failure from SIGNET-TMP-SE to KEK-DISK-TMP-SE, BNL-TMP-SE and DESY-TMP-SE have been low, ~50 %, for > 4 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-19332280
  • 2020-04-01, downtime,
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2344

Primary SE: KEK-DISK-TMP-SE (srm://kek2-se03.cc.kek.jp:8444/srm/managerv2?SFN=/disk/belle/TMP)

  • Date, Issue, Tickets...
  • File transfer failure from SIGNET-TMP-SE to KEK-DISK-TMP-SE, BNL-TMP-SE and DESY-TMP-SE have been low, ~50 %, for > 4 hours. 2020-02-09
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2280
  • 2020-01-02 13:20 UTC File transfer failure from KMI-TMP-SE and from KEK-Disk-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219

Primary SE: KEK2-TMP-SE (srm://kek2-se03.cc.kek.jp:8444/srm/managerv2?SFN=/belle/TMP)

  • No done activities. Number of jobs with status "done" is zero. 2020-02-07 15:40 (UTC)
  • SE Health Check by DDM: Failure on ls, upload have been observed since 2019-11-10 07:23:23 (5 hours)
  • Following JIRA tickets submitted: BIIDCO-1866
  • Number of jobs with status "done" is zero. 2019-07-05 7:07
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1866

Primary SE: KISTI-TMP-SE (belle-se-head.sdfarm.kr)

  • NOTE: This site is banned, there is no need to create a ticket related to SE Health check
  • SE Health check by DDM

    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1364
    GGUS ticket https://uggus.eu/index.php?mode=ticket_info&ticket_id=137825 has submitted 2018-10-18 14:21No new assignment of MC production data blocks to this destination
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-848

  • SE Health check by DDM : download, upload do not work since 2019-09-22 23:38:00 UTC.

Primary SE: KIT-TMP-SE (dcachesrm-kit.gridka.de)

  • Date, Issue, Tickets...
  • 2020-04-01, downtime,
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2345

Primary SE: KMI-TMP-SE (nsrmfe01.hepl.phys.nagoya-u.ac.jp )

  • Date, Issue, Tickets...
  • 2020-01-02 09:15 UTC File transfer failure from KMI-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219
  • KMI-TMP-SE with Scheduled jobs overwhelming Done ones since 9:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2169

Primary SE: Napoli-TMP-SE (belle-dpm-01.na.infn.it )

  • DateData, Issue, Tickets...

Primary SE: SIGNET-TMP-SE (dcache.ijs.si )

  • Date, Issue, Tickets...
  • SE Health Check by DDM: Failure on ls, upload have been observed since 2020-03-26 22:07:52 (8 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2333
  • SIGNET-TMP-SE SE Health Check by DDM: Failure on ls, upload have been observed since 2020-02-13 19:59:11 (13 hours)
  • 2020-02-09 File transfer failure from SIGNET-TMP-SE to KEK-DISK-TMP-SE, BNL-TMP-SE and DESY-TMP-SE have been low, ~50 %, for > 4 hours.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2280
  • File transfer failure for destination have been observed since 2019-12-23 02:00 UTC

Other SEs

Adelaide-TMP-SE (coepp-dpm-01.ersa.edu.au)

  • Date, Issue, Tickets...
  •  Adelaide SE is banned
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2184
  •  Adelaide SE: coepp-dpm-01.ersa.edu.au possible issue in host certificate  https://ggus.eu/?mode=ticket_info&ticket_id=144411

CYFRONET-TMP-SE (dpm.cyf-kr.edu.pl)

  • Date, Issue, Tickets...

CINVESTAV-TMP-SE (jaguar-se.fis.cinvestav.mx)

  • Date, Issue, Tickets...

Frascati-TMP-SE (atlasse.lnf.infn.it)

  • Date, Issue, Tickets...

HEPHY-TMP-SE (hephyse.oeaw.ac.at)

  • Date, Issue, Tickets...

IPHC-TMP-SE (sbgse1.in2p3.fr)

  • Date, Issue, Tickets...

LAL-TMP-SE (grid05.lal.in2p3.fr)

  • Date, Issue, Tickets...

Melbourne-TMP-SE (b2se.mel.coepp.org.au)

  • transfer rate to be zero

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-896

  • Melbourne-DATA-SE banned for write
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-927

McGill-TMP-SE  (storm02.clumeq.mcgill.ca)

  • Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-516
    McGill-TMP-SE will be decomissioned in early 2018.

MPPMU-TMP-SE (grid-srm.rzg.mpg.de)


NTU-TMP-SE (bgrid3.phys.ntu.edu.tw)

  •  NTU-TMP-SE banned for write 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1993
  • 2019-08-31  File transfer failures for past 48 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1987

NTU-CC-TMP-SE (belle2grid3.cc.ntu.edu.tw)

  • 202/01/13, 2019/12/17, 2019/12/11 File transfer failures to NTUCC-DATA-SE. There are no activities in Throughput and Successful transfers. -
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2174
  • 2019-10-06 File transfer failures for past 24 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2053
  • 2019/8/23 file transfer failure to NTU-CC-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1915
    2019/
    1977
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1987
  • FTS transfer failure as SOURCE NTU-CC-DATA-SE to BNL-TMP-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1953

    Solved and verified GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=142550 has submitted
  • 2019/01/27 File transfer failures from CNAF-TMP-SE to NTUCC-DATA-SE.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1637
     
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1892
  • File transfer failure and cancellation to NTUCC-DATA-SE happened 2018-12-22
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1551
  • Frequent timtout has observed between NTU-CC-TMP-SE and CNAF-TMP-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1310

    GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=137334 has submitted 2018-09-22 05:10 UTC
  • NTUCC-TMP-SE banned for write 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1333

Pisa-TMP-SE (stormfe1.pi.infn.it)

  • Date, Issue, Tickets...2020-03-28 17:40 UTC - Failed Transfer in some connections involving PISA-TMP-SE as source

PNNL-TMP-SE (se.hep.pnnl.gov) 

  • Being decommissioned. No need to report any issues. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-838

Roma3-TMP-SE (storm-01.roma3.infn.it)

  •  Date, Issue, Tickets...

TAU-TMP-SE (tau-se.hep.tau.ac.il)

  • Date, Issue, Tickets...File transfer from TAU-DATA-SE failed 2020-03-28 16:15 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2255
  • File transfer to TAU-SE failed: 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2255
    GGUS ticket 145074 https://ggus.eu/?mode=ticket_info&ticket_id=145074

Torino-TMP-SE (se-srm-00.to.infn.it)

  • Date, Issue, Tickets...

ULAKBIM-TMP-SE (torik1.ulakbim.gov.tr)

UMiss-TMP-SE (umiss005.hep.olemiss.edu)

  • Date, Issue, Tickets...

UVic-TMP-SE(charon01.westgrid.ca)

  • File Transfer failures : File Transfer Efficiency is too low from UVic-DATA-SE. since about 2018-12-18 1:00 (UTC) 

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1491



Sites

Sites Common Issue

  • Date, issue for sites wide

ARC.DESY.de

  • Health checker info. : "Short pilot jobs" has been found since 15:20:00 UTC on 2019/06/03.(details)2020-03-30 - 18:00 UTC, downtime,
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2343
  • 2020-04-01 - 07:00 UTC, downtime,
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-18592344

ARC.DESY-test.de

  • A test queue for the new CE.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1469

ARC.KIT.de

  • Downtime: 2019-07-25 07:00 to 2019-07-25 11:00 (UTC"Pilot Submission Failure" has been observed since 2020-03-19 12:24 UTC (for 2 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-
    1939Downtime: 2019-07-22 08:00 to 2019-07-22 08:30 (UTC) 
    2314
  • "Pilot Submission Failure" has been observed since 2020-03-15 23:24 UTC (for 1 hours) (details).
  • Pilot Submission Failure" has been observed since 2020-03-13 04:24 UTC (for 2 hours)
  • "Pilot Submission Failure" has been observed since 2020-03-08 11:24 UTC

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2314

  • "Aborted Pilot" has been observed since 2020-02-17 21:59 UTC (for 1 hours) 
  • 2020-04-01, Downtime,
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-19312345

ARC.LMU.de

  • This is a test site. Do not need to report any issue.

ARC.LMU2.de

  • Banned as currently no resource behind the CE

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-239

ARC.Melbourne.au

  • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 2 hours)

ARC.MPPMU.de

  • "Failed Pilot" has been observed since 2020-04-02 12:30 UTC (for 2 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2346

  • "Failed Payload Job" has been observed since 2020-04-02 12:30 UTC (for 2 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2346
  • "Failed Payload Job" has been observed since 2020-03-26 12:30 UTC (for 2 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2346
  • "Failed Payload Job" has been observed since 2020-01-05 02:34 UTC (for 5 hours)
  • "Failed Payload Job" has been observed since 2019-09-30 14:27 UTC (for 8 hours)
  • "Failed Payload Job" has been observed since 2019-09-29 15:27 UTC (for 7 hours)
  • Job submission check : Pilot submission failure has been found since 00:26:00 UTC on 2019/04/21.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1386
  • Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-128

ARC.SIGNET.si

  • "Short Pilot" has been observed since 2020-01-04 02:34 UTC (for 4 hours)
  • "Failed Payload Job" has been observed since 2020-01-04 02:34 UTC (for 4 hours)
  • "Short Pilot" has been observed since 2019-12-24 18:28 UTC (for 4 hours)
  • "Short Pilot" has been observed since 2019-11-10 05:30 UTC (for 1 hours)
  • Health checker info. : "Aborted Failed pilot jobs" has been found since 0120:20:00 UTC on 2019/06/0308/28.(details
  • Health checker info. : "Short pilot jobs" has been found since 1013:20:00 UTC on 2019/0508/2701.(details)
  • "Failed pilot jobs" has been found at 15:20:00 UTC on 2019/05/22.(details)
  • "Short pilot jobs" has been found at 15:20:00 UTC on 2019/05/22.(details)
  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/05/21.(details)
  • Job status check: many Stalled jobs on 2019/05/14 at 7:00 UTC.
  • Health checker info. : "Short pilot jobs" has been found at 07:20:00 UTC on 2019/05/14.(details)
  • Health checker info. : "Short pilot jobs" has been found at 15:20:00 UTC 2019/04/05 and at 14:20:00 UTC on 2019/04/12.
  • Job status check: Application finished with errors (5% of the jobs) at 11:15 UTC on 2018/12/21.

  • "Failed to install DIRAC on " has been found since 20:20:00 UTC on 2018/11/03.

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1420

  • Health checker info. : "Aborted pilot jobs" has been found since 20:20:00 UTC on 2018/10/20.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1383
    Health checker info. : "Failed pilot jobs" has been found since 06:20:00 UTC on 2018/10/03.(details)
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1350

CLOUD.CC1_Krakow.pl

  • Not used in production yet. Seeing no jobs (no plot) is not a problem
DIRAC

CLOUD.

Beihang

DESY.

cnJob submission check : Pilot submission failure has been found since 04:21:00 UTC on 2019/07/04.

de

  • Newly commissioned site. Problems should be reported. (With a separate ticket from BIIDCO-2270)
  •   Being configured (BIIDCO-2270). No report necessary.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1312
    Health checker info. : "Short pilot jobs" has been found since 16:20:00 UTC on 2019/06/30.
    2270
  • 2020/04/01, Downtime,
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-18072344

DIRAC.Beihang.cn

  • Site is banned.
  • "Failed Payload Job" has been observed since 2019-04-19 11:15 UTC  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1812
  • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/04/18.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1807
  • Health checker info. : "Short pilot jobs" has been found since 06:20:00 UTC on 2019/04/17.
  • "Application finished with errors" (100% currently) on 2019/04/10 00:15 UTC. Problem reported since (at least) 2019/04/07 07:00 UTC.
  • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2018/12/08.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1534
  • Job status check: "application finished with errors" (100% currently) on 2018/10/26.
  • Job submission check : Pilot submission failure has been found since 09:24:00 UTC on 2018/09/21. (details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1312
  • The number of jobs limited.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-289
  • All the upload trials are failing against all the SEs configured: OutputSE (KMI-TMP-SE, PNNL-TMP-SE), Fail-over SEs(DESY-TMP-SE, Napoli-TMP-SE, PNNL-TMP-SE, KIT-TMP-SE) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-43
  • Large % of failed jobs in DIRAC status plot (Added 2016-11-03 22:45:00 UTC) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-38
    •  "Application finished with errors" (100% currently) on 2019/04/10 00:15 UTC. Problem reported since (at least) 2019/04/07 07:00 UTC

DIRAC.BINP.ruDIRAC.BINP.ru

  •  New jobs do not run since 2020-03-23 around 10:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2326
  • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 10 hours)
  • "Short Pilot" has been observed since 2019-11-10 05:30 UTC (for 1 hours)
  • Job status check: "Application Finished With Errors" (39% of the jobs over the last 24h) at 7:00 UTC on 2019/05/15.
  • Job status check: Application finished with errors (27% of the jobs over the last 24h) at 8:00 UTC on 2018/12/22.
  • Health checker info. : "Failed to install DIRAC on " has been found at 22:20:00 UTC on 2018/09/15

DIRAC.BINP-VM.ru

  • "Pilot Submission Failure" has been observed since 2020-01-17 05:53 UTC (for 17 hours)
  • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
  • Health checker info. : "Aborted pilot jobs" has been found at 06:20:00 UTC on 2019/02/21
  • Job submission check : Pilot submission failure has been found since 10:23:00 UTC on 2019/01/14.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1607
  • Job status plots, "Application Finished With Errors" (2018-02-11 but lasting for at least a month)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-749

DIRAC.CINVESTAV.mx

  • Health checker info. : "Short pilot jobs" has been found at Job status: 80% of jobs had Input Data Resolution errors in past 24 hours, observed on 2020-03-04 at 8:00 UTC.
  • "Pilot Submission Failure" has been observed since 2020-02-18 00:59 UTC (for 22 hours) "Pilot Submission Failure" has been observed since 2020-02-12 11:59 UTC (for 11 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1277
  • Health checker info. : "Short pilot jobs" has been found at 14:20:00

    UTC

    on

    2019/04/14.

  • Job submission check : Pilot submission failure has been found at 13:27:00 UTC on 2019/03/19.
  • Health checker info. : "Short pilot jobs" has been found at 15:20:00 UTC on 2018/12/06. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1524

DIRAC.DESY.de

  • Test site. Not in use in MC production

DIRAC.IITG.in  

  • Health checker info. : "Short pilot jobs"Aborted Pilot" has been found since 15:20:00 UTC on 2019/07/29. observed since 2020-03-28 16:30 UTC (for 14 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1686
    Health checker info. : "Short pilot jobs
    2070
  • AID: "Aborted Pilot" has been

    found

    observed since 2019-10-16 22:

    20:00 UTC on 2019/07/28.

    38 UTC: JIRA ticket created
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-

    1686

    2070

  • "Aborted

    pilot jobs

    Pilot" has been

    found since 12:20:00 UTC on 2019/07/10. (screenshot)

    observed since 2020-03-19 13:24 UTC (for 1 hours)

    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2070

  • Job status check: "Application finished with errors" on 2019/07/10 at 00:00 UTC (screenshot)
  • Pilot submission failure has been found since 08:23:00 UTC on 2019/06/07. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1474
  • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/05/16.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1686
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1768
  • Job status check: many "Application finished with errors" (overall 66% during past 24 hours) on 2019/05/15 at 7:00 UTC.
  • Job status check: many "Application finished with errors" on 2019/05/14 at 7:00 UTC.
  • Job status plots, 100% "Application Finished With Errors", 10:00:00 UTC on 2019/04/08. Still unchanged as of 2019/04/26. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1823
  • Health checker info. : "Aborted pilot jobs" has been found at 00:20:00 UTC on 2019/04/08.
  • Job status check: Application finished with errors (95% of the jobs over the last 24h) at 8:00 UTC on 2018/12/22.
  • Job status check: Input Data Resolution issues (100% of the jobs) on 2018/12/21 at 8:48 UTC.

DIRAC.IITH.in

  • Health checker info. :
  • "Short pilot jobs" has been found since 15:20:00 UTC on 2019/06/03.(details)
  • "Aborted pilot jobs" has been found at 22:20:00 UTC on 2019/06/03.(details)
  • Health checker info. :

    DIRAC.IITH.in

    • "Pilot Submission Failure" has been observed since 2020-03-02 01:05 UTC (for 5 hours).

    • "Pilot Submission Failure" has been observed since 2020-03-01 13:05 UTC (for 9 hours)

    • "Pilot Submission Failure" has been observed since 2020-02-18 03:59 UTC (for 19 hours) 

    • "Pilot Submission Failure" has been observed since 2020-02-17 15:59 UTC (for 7 hours) 

    • "Aborted pilot jobs" has been found at 22:20:00 UTC on 2019/06/0203.(details)
    • Health checker info. : "Short Aborted pilot jobs" has been found at 1422:20:00 UTC on 2019/0506/1102.(details)
    • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/04/04Health checker info. : "Short pilot jobs" has been found at 07:20:00 UTC on 2019/03/29.(details
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1768

    DIRAC.LMU.de

    • Not in use in MC production
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-26
    • Banned for now.

    DIRAC.MIPT.ru

      Health checker info. : "Short pilot jobs
    • "Failed Payload Job" has been observed since 2020-01-16 21:53 UTC (for 1 hours)
    • "Failed Payload Job" has been

    • found since 12:20:00 UTC on 2019/05/25.(details)
    • observed since 2020-01-15 02:53 UTC (for 4 hours) (details)

    • "Failed Payload Job" has been observed since 2020-01-04 07:34 UTC (for 62 hours)
    • "Aborted Pilot" has been observed since 2019-12-25 03:28 UTC (for 3 hours)
    • "Aborted Pilot" has been observed since 2019-12-21 03:28 UTC (for 3 hours) 
    • "Aborted Pilot" has been observed since 2019-12-20 10:28 UTC (for 5 hours) 
    • Health checker info. : "Aborted pilot jobs" has been found since 13:20:00 UTC on 2019/04/20.
      Jira
      serverDESY JIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1816
    • Health checker info. : "Aborted pilot jobs" has been found since 11:20:00 UTC on 2019/04/06 and since 05:20:00 UTC on 2019/04/12. and since 20:20:00 UTC on 2019/04/17.Health checker info. : Short pilot jobs" has been found at 23:20:00 UTC on 2019/04/10 and 15:20:00 UTC on 2019/04/14.

    DIRAC.Nagoya.jp

    • "Failed Payload Job" has been observed since 2020-01-04 23:34 UTC (for 7 hours)
    • "Short Pilot" has been observed since 2019-11-19 05:35 UTC (for 1 hours)
    • Health checker info. : "Short pilot jobs" has been found at 07since 09:20:00 UTC on 2019/0310/2909.(details

    DIRAC.Nara-WU.jp

    • Under commissioning from 2018-11-13
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1768
      1432

    DIRAC.

    Nagoya

    NDU.jp

    • Date, IssuesIssue, ticketsTickets...

    DIRAC.

    Nara-WU

    Niigata.jp

    • Under commisioning from 2018-11-13
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1432
      "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
    • "Short Pilot" has been observed since 2019-11-19 05:35 UTC (for 1 hours)

    • Health checker info. : "Short pilot jobs" has been found at 15:20:00 UTC on 2019/

      03/28
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1758

    DIRAC.NDU.jp

    • Date, Issues, tickets...

    DIRAC.Niigata.jp

    • 10/09.

    • Job submission check : Pilot submission failure has been found since 19:26:00 UTC on 2019/05/26. (details)

    • Health checker info. : "Aborted pilot jobs" has been found since 12:20:00 UTC on 2019/05/18.
    • Job submission check : Pilot submission failure has been found since 13:30:00 UTC on 2019/05/14. (details)
    • Health checker info. : "Aborted pilot jobs" has been found at 06:20:00 UTC on 2019/04/21.

    DIRAC.Niigata2.jp

    • Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1926

    DIRAC.Osaka-CU.jp

    Job submission check : Pilot submission failure has been found since 14:26:00 UTC on 2019/06/03. (details)

    jp       

    • "Failed Payload Job" has been observed since 2020-04-02 20:30 UTC (for 3 hours) (details). 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-168
    • Job submission check : Pilot submission failure has been found since 06:21:00 UTC on 2019/04/02
    • Job submission check : Pilot submission failure has been found since 06:21:00 UTC on 2019/04/02.
      Jira
      serverDESY JIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1434
    • 2348


    DIRAC.Osaka-CU.jp

    • Site is banned
    • Job submission check : Pilot submission failure has been found since 07:23:00 UTC on 2018/12/04. (details)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1434
    • Health checker info. : "Short pilot jobs" has been found since 22:20:00 UTC on 2018/03/17.
      → Ask site admin to check the status 2018-03-17 10:00 JST. (DB access failure again from DIRAC.Osaka-CU.jp to PNNL from 2018-03-16 11:00 UTC)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-290

    DIRAC.PAU.in

    • "Pilot Submission Failure" has been observed since 2020-01-22 20:53 UTC (for 2 hours)

    DIRAC.PNNL.us

    • Site to be decommissioned
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-919

    DIRAC.PNNL2.us

    • Site to be decommissioned
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-920

    DIRAC.PNNL-CASCADE.us

    • Seeing no jobs (no plot) is not a problem

    DIRAC.PNNL-PIC.us

    • Seeing no jobs (no plot) is not a problem

    DIRAC.RCNP.jp

    • Health checker info. : "Aborted pilot jobs"Pilot Submission Failure" has been found since 05:20:00 UTC on 2019/07/04. observed since 2020-03-28 00:30 UTC (for 6 hours) 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1914

    DIRAC.SSU.kr

    • Down time: Soongsil Site Power Off June 26 - July 1
      2336
    • Jira
      serverDESY
      JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1890
      2327
      Job submission check : Pilot submission failure has been found since 07:21:00 UTC on 2019/06/26
    • "Failed Payload Job" has been observed since 2020-03-17 18:24 UTC (for 6 hours) 
    • "Failed Payload Job" has been observed since 2020-01-05 02:34 UTC (for 4 hours)
    • "Short Pilot" has been observed since 2020-01-04 02:34 UTC (for 4 hours)

    DIRAC.LocalTest.jp

    • Health checker info. : "Short pilot jobs" has been found since 09:20:00 UTC on 2019/10/09

    DIRAC.SSU.kr

    • Date, Issue, Tickets...

    DIRAC.TIFR.in

    • "Failed Payload Job" has been observed since 2020-01-12 05:53 UTC (for 11 hours)
      Jira
      serverDESY
      JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1893
    • Date, Issue, Tickets..

    DIRAC.TIFR.in

    • 2235
    • "Short Pilot" has been observed since 2020-01-12 04:53 UTC (for 11 hours)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1911
    • Job submission check : Pilot submission failure has been found at 14:25:00 UTC on 2019/05/11. (details)
    • 2234

    • Health checker info. : "Short pilot jobs" has been found
    • since
    • at 14:20:00 UTC on 2019/
    • 05/
    • 10
    • .(details)
    • Health checker info. : "Short pilot jobs" has been found since 13:20:00 UTC on 2018/10/22.
    • Job status plots, "Application Finished With Errors" has been found at about 00:00:00 JST on
    • /06.
    • 2018/07/06. (details)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1132
    • Health checker info. : "Short pilot jobs" -- Already reported: 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-971
    •  RunningLimit is set for MCProduction=1
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1006
    • Job stalled at input data resolution
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-714

    DIRAC.TMU.jp

    • Health checker info. : "Failed to install DIRAC on "Short Pilot" has been found since 14:20:00 UTC on 2019/04/24.
    • Health checker info. : "Belle II software could not be installed on " has been found since 04:20:00 UTC on 2019/04/17
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1804
    • observed since 2020-01-24 02:53 UTC (for 28 hours)
    • "Short Pilot" has been observed since 2020-01-24 02:53 UTC (for 4 hours)
    • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
    • "Pilot Submission Failure" has been observed since 2019-10-27 12:30 UTC (for 2 hours)

    • Health checker info. : "Short pilot jobsFailed to install DIRAC on " has been found since 0614:20:00 UTC on 2019/0304/2924.(details)
    • Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1763
      Job submission check : Pilot submission failure has been found at 13:27:00 UTC on 2019/03/19.
    • Health checker info. : "Short pilot jobs" has been found since 10:20:00 UTC on 2018/11/02
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1522

    DIRAC.Tokyo.jp

    • Decommissioned
    • Date, Issue, Tickets..

    DIRAC.UAS.mx

    • Downtime
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1940
    • 2020-04-03 19:00 UTC  "Job Status Plot" shows 100% Job finished with errors
    • 2020-03-27 19:53 UTC  "Job Status Plot" shows 100% Job finished with errors
    • Health checker info. : "Belle II software could not be installed on " has been found since 15:20:00 UTC on 2019/04/25
    • Job submission check : Pilot submission failure has been found since 00:21:00 UTC on 2019/04/04. (details)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1772
    • Health checker info. : "Belle II software could not be installed on " has been found since 01:20:00 UTC on 2019/02/20.
    • Job submission check: 100% failed with errors from 22:00 2019/01/08 till 04:00 2019/01/09 (UTC)
    • Health checker info. : "Belle II software could not be installed on " has been found since 04:20:00 UTC on 2018/12/17. 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1508
    • Health checker info. : "Belle II software could not be installed on " has been found since 16:20:00 UTC on 2018/11/14.
    • Job submission check : Pilot submission failure has been found since 01:26:00 UTC on 2018/09/21. (details)

    DIRAC.UVic.ca

    • "Failed Payload Job" has been observed since 2020-01-05 00:34 UTC (for 6 hours)

    DIRAC.UVic-local.ca

    • "Failed Payload Job" has been observed since 2020-01-05 00:34 UTC (for 6 hours)
    • Health checker info. : "Short pilot jobs" has been found since 17at 14:20:00 UTC on 20182019/09/17 Added to 10/06.
    • User jobs failed on the site:
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1286
    • Job submission check : Pilot submission failure has been found since 15:24:00 UTC on 2018/09/16 (emailed comp-dc-operations, create JIRA ticket when able)

    DIRAC.UVic.ca

    • Date, Issue, Tickets..

    DIRAC.UVic-local.ca

    • Job
      1975
    • Job status check: "Input Data Resolution" issues (13% overall, 100% in past hours) on 2019/05/16 at 7:00 UTC.
    • Health checker info. : "Short pilot jobs" has been found since 04:20:00 UTC on 2019/05/16.(details)
    • Health checker info. : "Belle II software could not be installed on " has been found since 04:20:00 UTC on 2019/05/13.

    DIRAC.Yamagata.jp

    • Health checker info. : "Short pilot jobs" has been found since 21:20:00 UTC on 2019/06/05.(details)

      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1861

    • Health checker info. : "Short pilot jobs" has been found at 22:20:00 UTC on 2019/03/13.(details)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1761

    DIRAC.Yonsei.kr

    • Job submission check : Pilot submission failure "Short Pilot" has been found since 18:21:00 UTC on 2019/07/26. observed since 2020-03-28 16:30 UTC (for 14 hours) 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-19482334
      Health checker info. :
    • "Short pilot jobsPilot" has been found at 06:20:00 UTC on 2019/07/04.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1416
    • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2018/12/08.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1416
    • observed since 2020-03-26 21:30 UTC (for 9 hours)
    • "Short Pilot" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
    • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
    • "Failed Payload Job" has been observed since 2019-12-30 21:34 UTC (for 1 hours)

    DIRAC.LocalTest.jp

    • Date, Issue, Tickets..

    LCG.CESNET.cz

    • Input data resolution 2019/06/29 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1895
       GGUS ticket  https://ggus.eu/?mode=ticket_info&ticket_id=142070
    • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/07/03
    • "Failed Pilot" has been observed since 2020-03-29 22:30 UTC (for 10 hours)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
    • 1902 Job submission check : Pilot submission failure has been found at 07:26:00 UTC on 2019/05/20. (details)
    • 2341

    • "Failed Payload Job" has been observed since 2020-01-04 21:34 UTC (for 10 hours)
    • "Failed Pilot" has been observed since 2019-12-24 16:28 UTC (for 6 hours)
    • Health checker info. : "Failed pilot jobs" has been found at 06:20:00 UTC on 2019/05/15.(details)
    • Job submission check : Pilot submission failure has been found at 06:26:00 UTC on 2019/05/15. (details)
    • Health checker info. : "Failed pilot jobs" has been found since 20:20:00 UTC on 2019/05/13.(details)
    • All jobs failing with InputDataResolution issues on 2019/05/13.
    •   Need some intervention to run Merge jobs
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-771

    LCG.CNAF.it

    • Health checker info. : "Short pilot jobs" has been found at 05:20:00 UTC on 2019/07/26 "Pilot Submission Failure" has been observed since 2020-04-01 21:30 UTC (for 10 hours)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-18622341
    • Health checker info. : "Short pilot jobs" has been found at 17:20:00 UTC on 2019/06/03.(details)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1862
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1750

    LCG.CYFRONET.pl

    Health checker info. : "Not enough disk space on n1086-amd" has been found since 04

    LCG.COSENZA.IT

    • "Failed Payload Job" has been observed since 2020-01-05 04:34 UTC (for 3 hours)
    • "Short Pilot" has been observed since 2019-11-22 04:35 UTC (for 3 hours)
    • "Failed Payload Job" has been observed since 2019-11-21 21:35 UTC (for 10 hours)
    • "Short Pilot" has been observed since 2019-11-11 10:30 UTC (for 4 hours)

    LCG.CNAF.it

    • "Failed Payload Job" has been observed since 2020-01-04 21:34 UTC (for 10 hours)
    • Health checker info. : "Short pilot jobs" has been found since 11:20:00 UTC on 2019/0710/2803.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1945
    •  Downtime 2019-07-11 10:00 to 2019-12-10 22:00 (UTC)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1920
    • Health checker info. : "Failed to install DIRAC on n1084-amd" has been found since 20(details)

    LCG.CYFRONET.pl

    • "Failed Payload Job" has been observed since 2020-03-20 13:24 UTC (for 1 hours)
    • "BLAH Error" has been observed since 2020-03-16 23:23:23 UTC (for 7 hours)
    • "Failed Payload Job" has been observed since 2019-12-23 10:28 UTC (for 12 hours)
    • "Short Pilot" has been observed since 2019-11-21 21:35 UTC (for 11 hours)
    • "Failed Payload Job" has been observed since 2019-11-21 21:35 UTC (for 11 hours)
    • "Failed Payload Job" has been observed since 2019-11-20 13:35 UTC (for 1 hours)

    • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/10/06.
    • Job submission check : Pilot submission failure has been found since 14:23:00 UTC on 2019/07/0131.
    • Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1909
      Health checker info. : "Short pilot jobs" has been found since 13:20:00 UTC on 2018/12/13.
      Jira
      serverDESY JIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1246

    LCG.DESY.de

    • The site to be retired 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1240
       – No more jobs to be submitted.

    LCG.Frascati.it

    •  Site is currently Banned due to hardware problem since 2019-07-05

    • Job submission check : Pilot submission failure has been found since 14:24:00 UTC on 2019/05/24. 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1882
      • GGUS 141688 ticket submitted.
    • Health checker info. : "BLAH ERROR" has been found since 15:20:00 UTC on 2019/05/21.(details)

    LCG.

    COSENZA.IT
    • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/07/22.(details)
    • Health checker info. : "Short pilot jobs" has been found since 01:20:00 UTC on 2019/07/01. 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1904

    LCG.HEPHY.at

  • Job submission check : Pilot submission failure has been found at 14:21:00 UTC on 2019/07/28. (details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1944
  • HEPHY.at

    • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 2 hours)
    • Health checker info. : "BLAH ERRORFailed pilot jobs" has been found since 15at 13:20:00 UTC on 2019/0710/2603.(details)
    • Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1941
       GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=142459 has submitted
      Health checker info. : "Failed pilot jobs" has been found at 15:20:00 UTC on 2019/05/22.(details)
    • Health checker info. : "Short pilot jobs" has been found at 15:20:00 UTC on 2019/04/12.
    • Health checker info. : "Failed pilot jobs" has been found at 02:20:00 UTC on 2019/01/30.(details) and at 02:20:00 UTC on 2019/01/31.(details)
    • submission check : Pilot submission failure has been found at 14:22:00 UTC on 2018/12/27.

    LCG.IPHC.fr

    • Downtime: 2019-04-01 00:00 to 2019-04-30 00:00 (UTC) 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1775
    • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 11 hours)

    LCG.KEK.jp

    • Job status: Large number of jobs finished with errors (61.0%) in last 24 hour period, from approx. 2020-01-01 00:00 - 02:00 UTC
    • Health checker info. : "Short pilot jobs" has been found since 05:20:00 UTC on 2019/10/09

    • Health checker info. : "Failed Short pilot jobs" has been found at 0014:20:00 UTC on 20182019/10/06/18.(details)

    LCG.KEK.jp

    • SiteDirector "Failed to check the availability" 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1934

    LCG.KEK2.jp

    • Health checker info. : "Short pilot jobs" has been found at 16:20:00 UTC on 2019/10/09.

    • Still all jobs failing with InputDataResolution on 2019/07/25.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1542
    • GGUS ticket : "KEK SE: PrepareToGet ETIMEDOUT for a specific file path"(140328) has been submited at 21:26:29 UTC on 2019/03/21.
    • Health checker info. : "Short pilot jobs" has been found at 11:20:00 UTC on 2019/03/22.(details)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1741

    • Health checker info. : "Failed pilot jobs" has been found since 13:20:00 UTC on 2018/12/21.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1559
    • all jobs are in "Input data resolution" status since 12.00 2018/12/18 UTC
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1542

    LCG.KEK-merge.jp

    • Still all jobs failing with InputDataResolution Health checker info. : "Short pilot jobs" has been found at 00:20:00 UTC on 2019/0708/2526.
      Jira
      serverDESY JIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1776
    • Still all jobs failing with InputDataResolution, found on 2019/07/24.
    • All jobs failing with InputDataResolution in past hours, found on 2019/07/23.
    • Health checker info. : "Short pilot jobs" has been found since 05:20:00 UTC on 2019/07/22.(details)
    • Health checker info. : "Short pilot jobs" has been found since 04:20:00 UTC on 2019/05/31.
    • Health checker info. : "Short pilot jobs" has been found since 21:20:00 UTC on 2019/05/25.(details)
    • Job status check: "Input Data Resolution" issues (53% overall) on 2019/05/16 at 7:00 UTC.
    • Job status check: still many "Input Data Resolution" issues seen on 2019/05/14 at 7:00 UTC.
    • Many jobs failing with InputDataResolution in past hours, found on 2019/05/13.
    • 1978
    •   Most jobs failing with InputDataResolution
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1776
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1777
    • "Belle II software could not be installed on cb268.cc.kek.jp" has been found since 14:20:00 UTC on 2019/04/05
    • Health checker info. : "Short pilot jobs" has been found since 20:20:00 UTC on 2019/04/02
    •   being commissioned...

    LCG.KISTI.kr

    • Jobs slots are disabled for SE maintenance from 2018-10-19 to 2018-10-23 "Short Pilot" has been observed since 2020-03-04 22:05 UTC (for 8 hours) 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1380
      2311
       
    • Health checker info. : "BLAH ERROR" has been found since 06:20:00 UTC on 2018/10/19.(details)

    • "Short pilot jobs" has been found at 06:20:00 UTC on 2018/10/09.(details)
    • BLAH error seems to be happen if jobs exceed the allocated # of queues, not a problem (Site specific feature)  
      Jira
      serverDESY JIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1259
    • A large number of Merge jobs in waiting status
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-773

    LCG.KMI.jp

    • Health checker info. : "Short pilot jobs" has been found at at 22:20:00 UTC on 2019/04/08 and at 15:20:00 UTC on 2019/04/12.
    • Job status plots, 100% "Application Finished With Errors", 10:00:00 on 2019/04/08
    • Job submission check : Pilot submission failure has been found since 21:25:00 UTC on 2019/02/01. 

    • Job status check: Application finished with errors (7% of the jobs in last 24 hours) on 2018/12/21 at 8:48 UTC.
    • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2018/12/13.(details)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1533
      "Failed Payload Job" has been observed since 2020-01-04 22:34 UTC (for 9 hours)
    • "Failed Payload Job" has been observed since 2019-11-19 12:35 UTC (for 2 hours)
    • Health checker info. : "Belle II software could not be installed on pwn22.local" has been found since 21:20:00 UTC on 2018/11/22.
    • Job submission check : Pilot submission failure has been found since 21:24:00 UTC on 2018/10/02. (details)

    LCG.LAL.fr

    • Downtime: 2019-07-23 from 07:00 to 12:00 (UTC)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1935
    • Job submission check : Pilot submission failure has been found since 17:22:00 UTC on 2019/06/16. 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1886
    • "Pilot Submission Failure" has been observed since 2020-02-17 21:59 UTC (for 1 hours) 2020-01-02 13:20 UTC
    • "Failed Payload Job" has been observed since 2019-11-18 05:35 UTC (for 2 hours)
    • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/05/01.(details)

    LCG.Legnaro.it

    • Date, Issue, Tickets...

    LCG.Napoli.it

    •  t2-recas-ce01.na.infn.it shows pilot submission error and this CE should  be banned till 2019 September.
    • Job submission check : Pilot submission failure has been found since 09:22:00 UTC on 2019/07/10.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1919
    • Health checker info. : "Short pilot jobs" has been found since 16:20:00 UTC on 2019/07/02. 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1912
    • "Pilot Submission Failure" has been observed since 2020-03-16 22:24 UTC (for 1 hours) (details).
    • "Failed Payload Job" has been observed since 2020-03-05 01:05 UTC (for 5 hours)
    • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 11 hours)
    • "Failed Payload Job" has been observed since 2020-01-04 02:34 UTC (for 4 hours)
    • "Pilot Submission Failure" has been observed since 2019-11-16 06:35 UTC (for 32 hours)
    • Health checker info. : "Failed pilot jobs" has been found at 14:20:00 UTC on 2019/10/06.
    • Job submission check : Pilot submission failure has been found since 12:27:00 UTC on 2019/10/02.

    •  t2-recas-ce01.na.infn.it shows pilot submission error and this CE should  be banned till 2019 September.

    • Stalled jobs
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1255

    LCG.NTU.tw


    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-

    1943

    2339

    • "Short
    • pilot jobs
    • Pilot" has been
    • found at 00:20:00 UTC on 2019/07/08. (screenshot)
    • Job submission check : Pilot submission failure has been found at 14:23:00 UTC on 2019/06/28.
    • Health checker info. : "CRL has expired" has been found since 11:20:00 UTC on 2019/06/19. 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1889
       GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=142372 has submitted
    • Downtime: 2019-07-12 09:00 to 2019-07-15 04:00.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1815
    • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/04/18.

    LCG.Pisa.it

    • Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1942
    • Job submission check : Pilot submission failure has been found at 06:22:00 UTC on 2019/07/24. (details)
    • Health checker info. : "Aborted pilot jobs" has been found since 22:20:00 UTC on 2019/07/23.(details) JiraserverDESY JIRA
    • observed since 2020-03-28 14:30 UTC (for 1 hours)
    • "Belle II software could not be installed" for "belle2grid3.cc.ntu.edu.tw" has been observed since 2020-03-15 06:23:22 UTC (for 2 hours)
    • "Belle II software could not be installed" for "belle2grid3.cc.ntu.edu.tw" has been observed since 2020-03-13 06:23:20 UTC (for 2 hours)
    • "CRL has expired" for "node39-0" has been observed since 2020-03-05 06:23:15 UTC (for 1 hours)
    • "Pilot Submission Failure" has been observed since 2020-03-03 13:05 UTC (for 17 hours)   
    • "Failed Payload Job" has been observed since 2020-01-05 01:34 UTC (for 6 hours)
    • "Failed Payload Job" has been observed since 2019-12-23 21:28 UTC (for 1 hours)
    • "Failed Payload Job" has been observed since 2019-12-10 05:28 UTC (for 1 hours)

    LCG.Pisa.it

    • "Failed Pilot" has been observed since 2019-10-28 03:30 UTC
      Jira
      serverDESY JIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
    • 1937
    • 1619

      GGUS ticket:  https://ggus.eu/?mode=ticket_info&ticket_id=144073
    • Job submission check : Pilot submission failure has been found since 10:25:00 UTC on 2019/04/1212 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1791

      GGUS ticket : "INFN-PISA: All CEs - LSF directory doesn't exists"(139815) has been submited at 00:03:38 UTC on 2019/02/21. Link to GGUS ticket. https://ggus.eu/index.php?mode=ticket_info&ticket_id=142999
    • Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1591

      "Failed to install DIRAC on so1wn8.pi.infn.it,n2wn13.pi.infn.it,n2wn18.pi.infn.it,so1wn6" has been found since 01:20:00 UTC on 2019/01/02.
    • Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1157

      "Short pilot jobs" has been found since 02:20:00 UTC on 2018/09/21.(details)

    LCG.Roma3.it

    • Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1949
    • Health checker info. : "BLAH ERROR"Pilot Submission Failure" has been found since 10:20:00 UTC on 2019/07/29. observed since 2020-03-30 21:30 UTC (for 58 hours)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1947
      1791

    LCG.Roma3.it

    • Health checker info. : "Failed pilot jobs" has been found at 14:20:00 UTC on 2019/04/20.
      Jira
      serverDESY JIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1538
    • Job status check: Application finished with errors (25% of the jobs in last 24 hours) and Stalled (36%) on 2018/12/22 at 8:00 UTC.
    • Job status check: Application finished with errors (10% of the jobs in last 24 hours) and Stalled (76%) on 2018/12/21 at 8:48 UTC.

    LCG.TAU.il

    • Health checker info. : "Not enough disk space on N/A" has been found at 06:20:00 UTC on 2019/07/01.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1907
    • Health checker info. : "Failed pilot jobs" has been found at 23:20:00 UTC on 2019/05/29.(details)"Failed Payload Job" has been observed since 2020-01-05 00:34 UTC (for 7 hours)
    • Health checker info. : "Failed pilot jobs" has been found since 19:20:00 UTC on 2019/05/24.(details)Downtime 2019-05-22 11:05 to 2019-05-24 11:00 (UTC

    LCG.Torino.it

    • "Pilot Submission Failure" has been observed since 2020-04-02 16:30 UTC (for 22 hours)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1841

    LCG.Torino.it

    • Job submission check : Pilot submission failure has been found since 16:22:00 UTC on 2019/07/23. (details)
      2215
    • "Pilot Submission Failure" has been observed since 2020-04-02 13:30 UTC (for 1 hours) 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1417Downtime: 2019-07-18 11:00 - 2019-07-23 15:00 
      2215
    • "BLAH Error" has been observed since 2020-02-09 05:53:05 UTC (for 9 hours)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1897
      2279
    • "Failed pilot jobsBLAH Error" has been found since 21:20:00 UTC on 2019/07/10.Health checker info. : "Failed pilot jobs" has been found since 15:20:00 UTC on 2019/07/08. (screenshot)observed since 2020-02-08 14:53:04 UTC (for 1 hours)
    • "Pilot Submission Failure" has been observed since 2019-12-28 18:34 UTC 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1417
    • Downtime:2019-06-28
    • 2215
    • LCG.Torino.it: Downtime from 2020-03-25 00:00 to 2020-
    • 2019-07-05 17:00  
    • 03-30 23:00 (UTC)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
    • 1897
    • Job submission check : Pilot submission failure has been found at 13:26:00 UTC on 2019/05/20. (details)
    • Job submission check : Pilot submission failure has been found at 14:25:00 UTC on 2019/05/11. (details)
    • Job submission check : Pilot submission failure has been found at 14:25:00 UTC on 2019/05/09
    • Job submission check : Pilot submission failure has been found at 06:25:00 UTC on 2019/03/23.
    • Health checker info. : "Failed pilot jobs" has been found since 20:20:00 UTC on 2018/11/01. 
      2329

    LCG.ULAKBIM.tr

    • The queue 'belle7' to be disabled. use only 'belle'
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1417

    LCG.ULAKBIM.tr

    • 1896
    • Health checker info. : "BLAH ERRORAborted pilot jobs" has been found since 2101:20:00 UTC on 2019/0708/23.(details) 01.

    OSG.BNL.us

    • Pilot submission failure observed since 2020-01-28 01:53 UTC
    • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 11 hours)
    • "Failed Payload Job" has been observed since 2019-12-20 14:28 UTC 
      Jira
      serverDESY JIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
    • 1938GGUS ticket
    • 2195

      Solved and verified GGUS ticket : https://ggus.eu/index.php?mode=ticket_info&ticket_id=
    • 142354 has submitted
    • Health checker info. : "BLAH ERROR" has been found at 06:20:00 UTC on 2019/07/23.(details)
    • Job submission check : Pilot submission failure has been found at 06:21:00 UTC on 2019/07/04.
    • 144665 has  been submitted
    • "Belle II software could not be installed" for "bgk01.sdcc.bnl.gov" has been observed since 2019-12-18 15:27:52 UTC
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-
      1913
    OSG.BNL.us
    • 2194
    • "Pilot Submission Failure" has been observed since 2019-12-05 05:28 UTC 
    • Health checker info. : "Belle II software could not be installed on " has been found since 19:20:00 UTC on 2019/02/14.
    • Job submission check: Jobs fail with errors or input data resolution the last 24h (6:00 UTC, 2019/01/09) 
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1596
    • Production jobs: UNAVAILABLE files
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1302
    • Number of concurrent MCProduction jobs restricted
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1256
    •  MCProduction jobs are mostly stalled
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1253

    OSG.CORI.us

    • OSG.CORI.us resource has been removed because CY18 allocation was not approved

    OSG.UMiss.us

    • Health checker info. : "Short pilot jobs" has been found since 21:20:00 UTC on 2019/07/10. (screenshot)
    • Health checker info. : "Short pilot jobs" has been found since 21:20:00 UTC on 2019/07/08. (screenshot)
    • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/07/03.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1863
    • Health checker info. : "Short pilot jobs" has been found since 09:20:00 UTC on 2019/06/27
    • Health checker info. : "Short pilot jobs" has been found since 17:20:00 UTC on 2019/06/04.(details)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1863
    • Health checker info. : "Short pilot jobs" has been found at 23:20:00 UTC on 2019/06/03.(details)
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1863
    • Health checker info. : "Aborted pilot jobs" has been found at 22:20:00 UTC on 2019/06/02.
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1856
    • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/05/20.(details)
    • Job status check: 100% of issues of Input Data Resolution on 2019/05/14 at 7:00 UTC.
    • Health checker info. : "Short pilot jobs" has been found at 07:20:00 UTC on 2019/05/14.(details)
    • Health checker info. : "Short pilot jobs" has been found since 22:20:00 UTC on 2019/05/12.(details)
      Updated
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1768
    • Job submission check : Pilot submission failure has been found since 12:27:00 UTC on 2019/05/04. (details)
    • Health checker info. : "Short pilot jobs" has been found since 07:20:00 UTC on 2019/05/11.(details)
    • Health checker info. : "Short pilot jobs" has been found since 20found since 14:20:00 UTC on 2019/05/01.(details)
      Health checker info. : "Short pilot jobs" has been found since 21:04/11 and  at 17:20:00 UTC on 2019/04/3014.(details)
      Health checker info. : "Short pilot jobs" has been found since 04:20:00 UTC on
    • Job status check: 34.7% appl. finshed with errors on 2019/04/2908.(details)
      Health checker info. : "Short pilot jobs" has been found since 20:20:00 UTC on 2019/04/22.
      Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/04/16.
    • Health checker info. : "Short pilot jobs" has been found since 14:20:00 UTC on 2019/04/11 and  at 17:20:00 UTC on 2019/04/14.
    • Job status check: 34.7% appl. finshed with errors on 2019/04/08.

    SSH.KMI.jp

    • Job status check:

    SSH.KMI.jp

    • "Pilot Submission Failure" has been observed since 2020-03-21 13:24 UTC (for 1 hours)
    • "Pilot Submission Failure" has been observed since 2020-03-15 05:24 UTC (for 1 hours)
    • Job status plot: input data resolution problems (for 7 hours) since 2019-12-24 00:00 UTC, approximately.
    • "Short Pilot" has been observed since 2019-12-24 05:28 UTC (for 1 hours)
    • Job status check: Application finished with errors (12% of the jobs in last 24 hours) on 2018/12/22 at 11:30 UTC.
    • Health checker info. : "Short pilot jobs" has been found at 20:20:00 UTC on 2018/08/13.

    Test.KIT.de

    • "Failed Pilot" has been observed since 2020-01-19 21:53 UTC (for 1 hours)
    • "Aborted Pilot" has been observed since 2020-01-16 21:53 UTC (for 1 hours)
    • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 11 hours)
    • "Pilot Submission Failure" has been observed since 2019-12-05 05:28 UTC 
    • "Aborted Pilot" has been observed since 2019-11-23 05:35 UTC (for 1 hours)
    • Test site for the opportunistic resources at KIT. No need to report problems.LCG.Pisa.it
    • 2020-04-01, downtime,
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-2345

    Test.ULAKBIM.tr

    • Test site for the SL7 resources at ULAKBIM. No need to report problems.
    • No activities expected currently.

    VCYCLE.Napoli.it

    • "Failed Payload Job" has been observed since 2020-01-05 04:34 UTC (for 3 hours)
    • "Failed Payload Job" has been observed since 2019-11-21 20:35 UTC (for 12 hours)
    • "Failed Payload Job" has been observed since 2019-11-29 00:35 UTC (for 6 hours)

      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-2143

    • "Failed Payload Job" has been observed since 2019-11-20 13:35 UTC (for 1 hours)

    • Opportunistic site (Empty plot is not a problem)
    •  Ban lifted
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1613
    • "Sudo CE Error: sudo execution fails with return code 1"
      Jira
      serverDESY JIRA
      serverIdd254f614-e16c-3f52-8b79-663658704a33
      keyBIIDCO-1612

    VCYCLE.HNSC01.it, VCYCLE.HNSC02.it

    • Opportunistic site (Empty plot is not a problem)


    Links