Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...



Contents

Table of Contents
maxLevel1

Expand

Table of Contents
l




Production Plans

  • MC13a: run-independent MC production is ongoing 

  • Proc10 - Proc10 skims will be proceed as soon as proc10 is finished, and similarly for MC13.  


Production Status

Number of total waiting  jobs dropped by ~50% 20/02/01  

Jira
serverDESY JIRA
serverIdd254f614-e16c-3f52-8b79-663658704a33
keyBIIDCO-2262

Number of running jobs dropped by ~50% at 20/01/29 13:00 UTC

Jira
serverDESY JIRA
serverIdd254f614-e16c-3f52-8b79-663658704a33
keyBIIDCO-2258

Full resource usage is expected for several weeks, at least.

Data production summary page : https://confluence.desy.de/display/BI/Data+Production+Status.

From DataProduction coordinator

Data (re)processing:

  • Proc10 (2019a/b) is nearly complete.

  • If no problems are discovered in calibration of the 2019c data set, the expectation is that processing of the 2019c data will be finished sometime in January.

MC production:

  • MC13a (run independenct) production of another 1 ab-1 is started. 

  • Run-dependent MC (MC13b) should start in the coming days.

Analysis skimming:

  • All requests for proc9 skims have been completed.

  • Proc10 skims will be proceed as soon as proc10 is finished, and similarly for MC13.  


Central Services

Dirac (dirac.cc.kek.jp, b2dchsv01-b2dchsv06.cc.kek.jp, b2dchsv08.cc.kek.jp)

  • 2020-02-04 01:00 UTC, The memory used of 'b2dchsv06.cc.kek.jp Memory last hour' has increased more than 5 GB for 20 minutes. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2272
  • 2020-02-02 23:00 UTC, B2plotdisplay does not work during about 23:00 - 23:50 02/02/2020. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2269
  • 2019-12-25 14:30 UCT, the memory usage of b2dchsv01.cc.kek.jp has been increasing linearly for last 24 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2209
  • 2019-12-17 DIRAC Production: b2dchsv[01-06].cc.kek.jp down
  • Date, Issue, Tickets...

DB Production (b2dchdb1.cc.kek.jp, b2dchdb2.cc.kek.jp, b2dcsdb1.cc.kek.jp, b2dcsdb2.cc.kek.jp)

  • 2019-12-06 Problem with Ganglia monitor for DB( still shows "remnant plot" after 2h check )
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2164
  • Data, Issue, Tickets...

"Web" servers

  • 2019-12-07 Ganglia Monitor for the "Web" servers still shows "remnant plots" after 2hours` check.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2166


Anchor
DDM
DDM

DDM (bldirac01.sdcc.bnl.gov)

  • 2018-03-01 DDM deletion task seems stuck
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-808

Conditions DB ()

Monitor

  • CDB access check URL should be correct
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2154
  • Issue in access to DIRAC Web Portal 

LFC

  • Date, Issue, Tickets...

File Transfers and Replication Status

  • See also 120357662 for related issues
  • No activities in both Throughput and Successful transfers have been observed since 2020-02-06 01:00 UTC. 

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2273

  • No activities in transfer on two SE is observed since 2020-02-03 22:00 UTC and 2020-02-04 03:00 respectively. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2271
  • There is no activity during the last three hours (01/Jan/2020 since 9:00 to 12: 15 UTC) in the "Replication Status plot"
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2097
  • There is no activity during the last three hours (01/Jan/2020 since 9:00 to 12: 15 UTC) in both plots "Throughput" and "Successful transfers"
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2097
  • There is no activity in the last two hours in both plots "Throughput" and "Successful transfers"  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2144
  • No activity shown
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2097

FTS

  • Any problem in the FTS service or FTS monitoring are to be recorded here. Site/SE specific issues are to be recorded under each SIte/SE
  • Note that the FTS dashboard we use is an "old" instance and not well-maintained. We, Belle II members in general, do not have access to the "new" monitoring. When the dashboard is down, the shifters just need to notify the expert and skip the corresponding part of their work. The expert should check the new monitoring, for the access to the monitoring page is limited.
  • 2020-02-06 15:20 UTC File transfer failure from Roma3-TMP-SE to KIT-TMP-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2275
  • 2020-01-02 13:20 UTC File transfer failure from KMI-TMP-SE and from KEK-Disk-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219
  • 2020-01-02 9:15 UTC File transfer failure from KMI-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219
  • 2019-08-31  File transfer failures for past 48 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1987
  • 2019-09-03 File transfer failures for past 24 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1988

Replication Status

  • 2020-02-13 Zero number of "Done" at all SE; the number of Scheduled is increasing
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2286
  • 2019-12-21 Decreasing Done Jobs with many Scheduled Jobs 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2169
  • 2019-12-15 Zero Replication Efficiency 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2183
  • 2019-1-19 almost zero done, with a increasing numbers of scheduled jobs for more than 5 SEs and more than 5 hours.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1618
  • 2018-07-02   No Donetransfer,  several scheduled and rapid increase of Waiting replication

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1125

Job Status Plot

  • Date, Issue, Tickets...
  • No job status plots for 15 sites while MC13a production is ongoing 2020-02-07

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2277

  • All of the sites show error (red) starting at about the same time 2020-01-03 24:00 UTC. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2223
  • Low activity, majority of plots empty since 2020-01-02 03:07 UTC.
  • There are no jobs since 2019-10-04 00:00
  • 2019-11-27 Jobs finished with error around 7:00UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2135
  • 2019-12-28 Majority of plots are empty around 11.00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2214

Job Summary

  • Date, Issue, Tickets...
  • Number of running and waiting jobs have decreased by more than 20% since approx. 2019-12-31 19:00 UTC. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2218
  • Following JIRA ticket updated :
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1553



SEs

SE Common Issues

  • Issues with individual SEs should be recorded below (Primary SEs or Other SEs)

Raw data SEsLink to JIRA ticket: 

Raw data SE: KEK-RAW-SE (srm://kek2-se02.cc.kek.jp:8444/srm/managerv2?SFN=/belle/RAW)

  • 2019-07-24 15:30 UTC: all transfers failed (0/72) between KEK-RAW-SE and BNL-TMP-SE

Raw data SE: BNL-TAPE-SE (srm://dcblsrm.sdcc.bnl.gov:8443/srm/managerv2?SFN=/pnfs/sdcc.bnl.gov/tape)

  • date, issue, tickets

Primary SEs

Primary SE: BNL-TMP-SE (dcblsrm.sdcc.bnl.gov)

  • 2020-02-09 File transfer failure from SIGNET-TMP-SE to KEK-DISK-TMP-SE, BNL-TMP-SE and DESY-TMP-SE have been low, ~50 %, for > 4 hours.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2280
  • SE Health Check by DDM: Failure on download have been observed since 2020-02-07 09:19:40 (5 hours)
  • No Replication Trend Plot for BNL-TMP-SE 2020-01-02 09:30 UTC 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2220
  • File Transfer Failures have been observed since 2019-12-23 21:30:00 UTC, approximately. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2204
  • SE Health Check by DDM: Failure on upload have been observed since 2019-12-23 03:38:01 (3 hours)
  • SE Health Check by DDM: Failure on upload have been observed since 2019-12-22 02:22:57 (4 hours)
  • SE Health Check by DDM: Failure on upload have been observed since 2019-12-21 14:29:51 (4 hours)
  • SE Health Check by DDM: Failure on upload have been observed since 2019-12-21 01:02:50 (5 hours)
  • SE Health Check by DDM: Failure on upload have been observed since 2019-12-20 00:03:32 (5 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2193

     Solved and verified :  GGUS ticket https://ggus.eu/index.php?mode=ticket_verify&ticket_id=144665
  • SE Health check by DDM : download does not work since 2019-05-16 07:11:21 UTC.

  •  UNAVAILABLE files
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1302
  • SE Health check by DDM : download does not work since 2019-05-15 21:03:25 UTC

Primary SE: CESNET-TMP-SE (dpm1.egee.cesnet.cz) 

  • Replication status. Plot is not shown 2020-01-15 13:01 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1352
  • Replication status. Plot is not shown 2020-01-02 8:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1352
  • Replication status. Plot is not shown 2019-12-31 9:25 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1352
  • Replication status. Plot is not shown 2019-12-31 7:45 UTC
  • Plot is not shown 2019-12-25 05:00 UTC.
  • SE Health check by DDM : remove file, remove directory, ls do not work since 2019-07-10 06:32:47 UTC.

Primary SE: CNAF-TMP-SE (storm-fe-archive.cr.cnaf.infn.it)

  • File transfer failure for source have been observed since 2019-12-23 02:00 UTC
  • SE Health check by DDM : remove file does not work since 2019-05-13 08:27:21 UTC.
  • SE Health check by DDM : remove file, remove directory, download, upload, ls do not work since 2019-04-25 23:13:00 UTC.
  • 2019/01/27 File transfer failures from CNAF-TMP-SE to NTUCC-DATA-SE.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1637
  •  Cotinuous timeout failure between NTU-CC-TMP-SE and CNAF-TMP-SE

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1310

Primary SE: DESY-TMP-SE (dcache-se-desy.desy.de)

  • 2020-02-09 File transfer failure from SIGNET-TMP-SE to KEK-DISK-TMP-SE, BNL-TMP-SE and DESY-TMP-SE have been low, ~50 %, for > 4 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2280

Primary SE: KEK-DISK-TMP-SE (srm://kek2-se03.cc.kek.jp:8444/srm/managerv2?SFN=/disk/belle/TMP)

  • File transfer failure from SIGNET-TMP-SE to KEK-DISK-TMP-SE, BNL-TMP-SE and DESY-TMP-SE have been low, ~50 %, for > 4 hours. 2020-02-09
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2280
  • 2020-01-02 13:20 UTC File transfer failure from KMI-TMP-SE and from KEK-Disk-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219
  • Replication status: No. of waiting jobs increasing, and File Transfer Failures have been observed since 2019-12-23 21:30:00 UTC, approximately. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2204

Primary SE: KEK2-TMP-SE (srm://kek2-se03.cc.kek.jp:8444/srm/managerv2?SFN=/belle/TMP)

  • Number of jobs with status "done" is zero. 2020-02-07 15:40 (UTC)
  • SE Health Check by DDM: Failure on ls, upload have been observed since 2019-11-10 07:23:23 (5 hours)
  • Following JIRA tickets submitted: BIIDCO-1866
  • Number of jobs with status "done" is zero. 2019-07-05 7:07
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1866

Primary SE: KISTI-TMP-SE (belle-se-head.sdfarm.kr)

  • No new assignment of MC production data blocks to this destination

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-848

  • SE Health check by DDM : download, upload do not work since 2019-09-22 23:38:00 UTC.

Primary SE: KIT-TMP-SE (dcachesrm-kit.gridka.de)

  • Date, Issue, Tickets...

Primary SE: KMI-TMP-SE (nsrmfe01.hepl.phys.nagoya-u.ac.jp )

  • 2020-01-02 09:15 UTC File transfer failure from KMI-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219
  • Replication status: No. of waiting jobs increasing, and File Transfer Failures have been observed since 2019-12-23 21:30:00, approximately. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2204
  • KMI-TMP-SE with Scheduled jobs overwhelming Done ones since 9:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2169

Primary SE: Napoli-TMP-SE (belle-dpm-01.na.infn.it )

  • Date, Issue, Tickets...
  • Failed transfers (source and destination) and failed jobs (27.01.20)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2257
  • No Replication trend Plot shown 2020-01-02  08:00 UTC 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2122
  • No replication trend plot shown since 2020-01-01 23:05 UTC.
  • Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2122

Primary SE: SIGNET-TMP-SE (dcache.ijs.si )

  • SIGNET-TMP-SE    SE Health Check by DDM: Failure on ls, upload have been observed since 2020-02-13 19:59:11 (13 hours)
    
  • FTS access failure from/to SIGNET-TMP-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2287

    GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=145542 has been submitted
  • 2020-02-09 File transfer failure from SIGNET-TMP-SE to KEK-DISK-TMP-SE, BNL-TMP-SE and DESY-TMP-SE have been low, ~50 %, for > 4 hours.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2280
  • File transfer failure for destination have been observed since 2019-12-23 02:00 UTC
  • File transfer failure as destination have been observed since 2019-11-30 1:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2145

    Solved and verified GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=144310 is submitted
  • SE Health Check by DDM: Failure on upload have been observed since 2019-11-30 05:49:18 (6 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2146

Other SEs

Adelaide-TMP-SE (coepp-dpm-01.ersa.edu.au)

  • Date, Issue, Tickets...
  •  Adelaide SE is banned
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2184
  •  Adelaide SE: coepp-dpm-01.ersa.edu.au possible issue in host certificate  https://ggus.eu/?mode=ticket_info&ticket_id=144411

CYFRONET-TMP-SE (dpm.cyf-kr.edu.pl)

  • Date, Issue, Tickets...

CINVESTAV-TMP-SE (jaguar-se.fis.cinvestav.mx)

  • Date, Issue, Tickets...

Frascati-TMP-SE (atlasse.lnf.infn.it)

  • Date, Issue, Tickets...

HEPHY-TMP-SE (hephyse.oeaw.ac.at)

  • Date, Issue, Tickets...

IPHC-TMP-SE (sbgse1.in2p3.fr)

  • Date, Issue, Tickets...

LAL-TMP-SE (grid05.lal.in2p3.fr)

  • Date, Issue, Tickets...

Melbourne-TMP-SE (b2se.mel.coepp.org.au)

  • transfer rate to be zero

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-896

  • Melbourne-DATA-SE banned for write
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-927

McGill-TMP-SE  (storm02.clumeq.mcgill.ca)

  • Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-516
    McGill-TMP-SE will be decomissioned in early 2018.

MPPMU-TMP-SE (grid-srm.rzg.mpg.de)


NTU-TMP-SE (bgrid3.phys.ntu.edu.tw)

  •  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2134
  •  NTU-TMP-SE banned for write 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1993
  • 2019-08-31  File transfer failures for past 48 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1987

NTU-CC-TMP-SE (belle2grid3.cc.ntu.edu.tw)

  • 202/01/13, 2019/12/17, 2019/12/11 File transfer failures -
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2174
  • 2019-10-06 File transfer failures for past 24 hours. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2053
  • 2019/8/23 file transfer failure to NTU-CC-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1977
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1987
  • FTS transfer failure as SOURCE NTU-CC-DATA-SE to BNL-TMP-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1953

    GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=142550 has submitted
  • 2019/01/27 File transfer failures from CNAF-TMP-SE to NTUCC-DATA-SE.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1637
     
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1892
  • File transfer failure and cancellation to NTUCC-DATA-SE happened 2018-12-22
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1551
  • Frequent timtout has observed between NTU-CC-TMP-SE and CNAF-TMP-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1310

    GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=137334 has submitted 2018-09-22 05:10 UTC
  • NTUCC-TMP-SE banned for write 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1333

Pisa-TMP-SE (stormfe1.pi.infn.it)

  • Date, Issue, Tickets...

PNNL-TMP-SE (se.hep.pnnl.gov) 

  • Being decommissioned. No need to report any issues. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-838

Roma3-TMP-SE (storm-01.roma3.infn.it)

  •  Date, Issue, Tickets...

TAU-TMP-SE (tau-se.hep.tau.ac.il)

Torino-TMP-SE (se-srm-00.to.infn.it)

  • Date, Issue, Tickets...

ULAKBIM-TMP-SE (torik1.ulakbim.gov.tr)

  • File transfer failures destination 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2253

UMiss-TMP-SE (umiss005.hep.olemiss.edu)

  • Date, Issue, Tickets...

UVic-TMP-SE(charon01.westgrid.ca)

  • File Transfer Failures have been observed since 2019-12-23 21:30:00 UTC, approximately. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2204
  • File Transfer failures : File Transfer Efficiency is too low from UVic-DATA-SE. since about 2018-12-18 1:00 (UTC) 

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1491



Sites

Sites Common Issue

  • Date, issue for sites wide
  • Job status plots: jobs are failing at 8 cites at the same time (circa 2019-11-28) 16.00-18.00 UTC : ARC.KIT.deDIRAC.UVic-local.caLCG.KEK.jpLCG.NTU.twLCG.Napoli.itTest.KIT.deOSG.BNL.us, LCG.CNAF.it  

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2140

ARC.DESY.de

  • Date, issue, JIRA

ARC.DESY-test.de

  • A test queue for the new CE.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1469

ARC.KIT.de

  • Downtime on 2020-01-16 09:00 to 2020-01-16 09:30
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2244
  • Downtime on 2020-01-15 13:00 to 2020-01-15 15:00 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2240
  • Downtime on 2019-01-13  8:00 UTC  to 2020-01-13 16:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2236

  • "Short Pilot" has been observed since 2020-01-05 00:34 UTC (for 7 hours)
  • "Failed Payload Job" has been observed since 2020-01-05 00:34 UTC (for 7 hours)
  • Following JIRA tickets submitted: BIIDCO-2140
  • "Short Pilot" has been observed since 2020-01-04 02:34 UTC (for 4 hours)
  • Following JIRA tickets submitted: BIIDCO-2140
  • Downtime 2019-12-11 13:00 UTC - 2020-01-24 22:00 (UTC) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2173
  • Pilot Submission Failure" has been observed since 2019-12-03 21:35 UTC

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2159

    Solved and verified GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=144364 has been submitted

  • "Short Pilot" has been observed since 2019-11-21 21:35 UTC (for 10 hours)
  • "Failed Payload Job" has been observed since 2019-11-22 01:35 UTC (for 6 hours)
  • "Short Pilot" has been observed since 2019-11-20 13:35 UTC (for 1 hours)

  • "Short Pilot" has been observed since 2019-11-18 13:35 UTC
  • "Pilot Submission Failure" has been observed since 2019-11-13 13:30 UTC (for 1 hours)
  • "Short Pilot" has been observed since 2019-11-12 13:30 UTC (for 1 hours)
  • "Short Pilot" has been observed since 2019-11-11 09:30 UTC (for 5 hours)  
  • "Short Pilot" has been observed since 2019-11-10 03:30 UTC (for 3 hours)
  • Downtime 2020-01-09 14:38 - 2020-01-10 16:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2230

ARC.LMU.de

  • This is a test site. Do not need to report any issue.

ARC.LMU2.de

  • Banned as currently no resource behind the CE

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-239

ARC.Melbourne.au

  • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 2 hours)
  • Following JIRA tickets submitted: BIIDCO-2158
  •  Downtime 2019-12-02 22:00- 2019-12-03 03:00 (UTC) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2158
     
  • Job status check: Application finished with errors observed on 2019-11-20.

ARC.MPPMU.de

  • "Failed Payload Job" has been observed since 2020-01-05 02:34 UTC (for 5 hours)
  • Following JIRA tickets submitted: BIIDCO-2052 , BIIDCO-1386 , BIIDCO-128
  •  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2052
  • "Failed Payload Job" has been observed since 2019-09-30 14:27 UTC (for 8 hours)
  • "Failed Payload Job" has been observed since 2019-09-29 15:27 UTC (for 7 hours)
  • Job submission check : Pilot submission failure has been found since 00:26:00 UTC on 2019/04/21.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1386
  • Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-128

ARC.SIGNET.si

  • Aborted pilots on CE pikolit.ijs.si since 2020-01-05 01:34 UTC: BIIDCO-2224
  • "Aborted Pilot" has been observed since 2020-01-05 01:34 UTC (for 6 hours)
  • Following JIRA tickets submitted: BIIDCO-2172 , BIIDCO-1383 , BIIDCO-2112 , BIIDCO-1350 , BIIDCO-1420
  • "Short Pilot" has been observed since 2020-01-04 02:34 UTC (for 4 hours)
  • "Failed Payload Job" has been observed since 2020-01-04 02:34 UTC (for 4 hours)
  • Following JIRA tickets submitted: BIIDCO-2172 , BIIDCO-1383 , BIIDCO-2112 , BIIDCO-1350 , BIIDCO-1420
  • "Short Pilot" has been observed since 2019-12-24 18:28 UTC (for 4 hours)
  • Increasing rate of stalled job, no jobs completed anymore since 10:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2172
  •  "Aborted Pilot" has been observed since 2019-12-03 21:35 UTC 
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1383

    Solved and verified  GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=144366 has been submitted
  • "Short Pilot" has been observed since 2019-11-11 09:30 UTC (for 5 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
     
  • "Short Pilot" has been observed since 2019-11-10 05:30 UTC (for 1 hours)
  • Health checker info. : "Not enough disk space on " has been found since 21:20:00 UTC on 2019/09/22. (details)
  • Health checker info. : "Failed pilot jobs" has been found since 20:20:00 UTC on 2019/08/28.(details)
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1350
  • Health checker info. : "Short pilot jobs" has been found since 13:20:00 UTC on 2019/08/01.
  • Health checker info. : "Aborted pilot jobs" has been found since 01:20:00 UTC on 2019/06/03
  • "Failed pilot jobs" has been found at 15:20:00 UTC on 2019/05/22.(details)
  • "Short pilot jobs" has been found at 15:20:00 UTC on 2019/05/22.(details)
  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/05/21.(details)
  • Job status check: many Stalled jobs on 2019/05/14 at 7:00 UTC.
  • Health checker info. : "Short pilot jobs" has been found at 07:20:00 UTC on 2019/05/14.(details)
  • Health checker info. : "Short pilot jobs" has been found at 15:20:00 UTC 2019/04/05 and at 14:20:00 UTC on 2019/04/12.
  • Job status check: Application finished with errors (5% of the jobs) at 11:15 UTC on 2018/12/21.

  • "Failed to install DIRAC on " has been found since 20:20:00 UTC on 2018/11/03.

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1420

  • Health checker info. : "Aborted pilot jobs" has been found since 20:20:00 UTC on 2018/10/20.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1383
  • Health checker info. : "Failed pilot jobs" has been found since 06:20:00 UTC on 2018/10/03.(details)
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1350

CLOUD.CC1_Krakow.pl

  • Not used in production yet. Seeing no jobs (no plot) is not a problem

CLOUD.DESY.de

  • Newly commissioned site. Problems should be reported. (With a separate ticket from BIIDCO-2270)
  •   Being configured (BIIDCO-2270). No report necessary.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2270

DIRAC.Beihang.cn

  • Site is banned.
  • "Pilot Submission Failure" has been observed since 2020-01-05 16:34 UTC (for 29 hours)
  • Following JIRA tickets submitted: BIIDCO-1312 , BIIDCO-2202 , BIIDCO-1807 , BIIDCO-1812 , BIIDCO-1534 , BIIDCO-289 , BIIDCO-43 , BIIDCO-38
  • "Pilot Submission Failure" has been observed since 2019-12-30 19:34 UTC (for 59 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1312
  • "Pilot Submission Failure" has been observed since 2019-12-30 19:34 UTC (for 11 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1312
  • "Pilot Submission Failure" has been observed since 2019-12-23 05:28 UTC (for 7 hours). 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2202
  • "Pilot Submission Failure" has been observed since 2019-12-23 05:28 UTC (for 1 hours)
  • Job submission check : Pilot submission failure has been found since 04:21:00 UTC on 2019/07/04.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1312
  • Health checker info. : "Short pilot jobs" has been found since 16:20:00 UTC on 2019/06/30.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1807
  • "Failed Payload Job" has been observed since 2019-04-19 11:15 UTC  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1812
  • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/04/18.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1807
  • Health checker info. : "Short pilot jobs" has been found since 06:20:00 UTC on 2019/04/17.
  • "Application finished with errors" (100% currently) on 2019/04/10 00:15 UTC. Problem reported since (at least) 2019/04/07 07:00 UTC.
  • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2018/12/08.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1534
  • Job status check: "application finished with errors" (100% currently) on 2018/10/26.
  • Job submission check : Pilot submission failure has been found since 09:24:00 UTC on 2018/09/21. (details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1312
  • The number of jobs limited.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-289
  • All the upload trials are failing against all the SEs configured: OutputSE (KMI-TMP-SE, PNNL-TMP-SE), Fail-over SEs(DESY-TMP-SE, Napoli-TMP-SE, PNNL-TMP-SE, KIT-TMP-SE) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-43
  • Large % of failed jobs in DIRAC status plot (Added 2016-11-03 22:45:00 UTC) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-38

DIRAC.BINP.ru

  • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 10 hours)
  • "Short Pilot" has been observed since 2019-11-10 05:30 UTC (for 1 hours)
  • Job status check: "Application Finished With Errors" (39% of the jobs over the last 24h) at 7:00 UTC on 2019/05/15.
  • Job status check: Application finished with errors (27% of the jobs over the last 24h) at 8:00 UTC on 2018/12/22.
  • Health checker info. : "Failed to install DIRAC on " has been found at 22:20:00 UTC on 2018/09/15

DIRAC.BINP-VM.ru

  • "Pilot Submission Failure" has been observed since 2020-01-17 05:53 UTC (for 17 hours)
  • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
  • Following JIRA tickets submitted: BIIDCO-2208 , BIIDCO-1607 , BIIDCO-749 , BIIDCO-2185
  • Job status check: Job stalled 100%, then no jobs running since 2019-12-24 06:00:00 (for 24 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2208
  • Health checker info. : "Aborted pilot jobs" has been found at 06:20:00 UTC on 2019/02/21
  • Job submission check : Pilot submission failure has been found since 10:23:00 UTC on 2019/01/14.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1607
  • Job status plots, "Application Finished With Errors" (2018-02-11 but lasting for at least a month)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-749
  • Downtime: maintenance on dirac.binp-vm.ru till 2019-12-21.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2185

DIRAC.CINVESTAV.mx

  • "Pilot Submission Failure" has been observed since 2020-02-12 11:59 UTC (for 11 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1277
  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/04/14.

  • Job submission check : Pilot submission failure has been found at 13:27:00 UTC on 2019/03/19.

DIRAC.DESY.de

  • Test site. Not in use in MC production

DIRAC.IITG.in   

  • "Aborted Pilot" has been observed since 2020-02-07 02:59 UTC (for 36 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2070
  • "Aborted Pilot" has been observed since 2020-02-06 05:59 UTC (for 10 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2070
  • "Failed Payload Job" has been observed since 2019-12-19 21:28 UTC (for 1 hours)
  • "Aborted Pilot" has been observed since 2019-11-29 04:35 UTC (for 265 hours)
  • Following JIRA tickets submitted: BIIDCO-2070 , BIIDCO-1686 , BIIDCO-1823
  • "Aborted Pilot" has been observed since 2019-11-19 15:35 UTC (for 39 hours) (details).
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2070
  • "Aborted Pilot" has been observed since 2019-11-18 06:35 UTC (for 1 hours) (details).
  • "Aborted Pilot" has been observed since 2019-11-02 20:30 UTC: JIRA ticket updated
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2070
  • AID: "Aborted Pilot" has been observed since 2019-10-16 22:38 UTC: JIRA ticket created

    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2070

  • "Aborted pilot jobs" has been found since 12:20:00 UTC on 2019/07/10. (screenshot)
  • Job status check: "Application finished with errors" on 2019/07/10 at 00:00 UTC (screenshot)
  • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/05/16.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1686
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1768
  • Job status check: many "Application finished with errors" (overall 66% during past 24 hours) on 2019/05/15 at 7:00 UTC.
  • Job status check: many "Application finished with errors" on 2019/05/14 at 7:00 UTC.
  • Job status plots, 100% "Application Finished With Errors", 10:00:00 UTC on 2019/04/08. Still unchanged as of 2019/04/26. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1823

DIRAC.IITH.in

  • "Aborted pilot jobs" has been found at 22:20:00 UTC on 2019/06/03.(details)
  • Health checker info. : "Aborted pilot jobs" has been found at 22:20:00 UTC on 2019/06/02.(details)
  • Health checker info. : "Short pilot jobs" has been found at 07:20:00 UTC on 2019/03/29.(details
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1768

DIRAC.LMU.de

  • Not in use in MC production
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-26
  • Banned for now.

DIRAC.MIPT.ru

  • "Short Pilot" has been observed since 2020-01-16 21:53 UTC (for 1 hours)
  • "Failed Payload Job" has been observed since 2020-01-16 21:53 UTC (for 1 hours)
  • "Failed Payload Job" has been observed since 2020-01-15 02:53 UTC (for 14 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2242
  • "Short Pilot" has been observed since 2020-01-15 05:53 UTC (for 1 hours) (details).


  • "Failed Payload Job" has been observed since 2020-01-15 02:53 UTC (for 4 hours) (details)

  • "Short Pilot" has been observed since 2020-01-14 00:53 UTC (for 6 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2238
  • "Short Pilot" has been observed since 2020-01-04 07:34 UTC (for 62 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2225
  • "Failed Payload Job" has been observed since 2020-01-04 07:34 UTC (for 62 hours)
  • Following JIRA tickets submitted: BIIDCO-1816 , BIIDCO-2112 , BIIDCO-1768
  • "Aborted Pilot" has been observed since 2019-12-25 03:28 UTC (for 3 hours)
  • "Aborted Pilot" has been observed since 2019-12-21 03:28 UTC (for 3 hours) 
  • "Aborted Pilot" has been observed since 2019-12-20 10:28 UTC (for 5 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1816
  • "Short Pilot" has been observed since 2019-11-11 11:30 UTC (for 3 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
     
  • "Short Pilot" has been observed since 2019-11-10 04:30 UTC (for 2 hours)
  • "Failed Payload Job" has been observed since 2019-11-10 05:30 UTC (for 1 hours)
  • Health checker info. : "Short pilot jobs" has been found since 12:20:00 UTC on 2019/05/25.(details)
  • Health checker info. : "Aborted pilot jobs" has been found since 13:20:00 UTC on 2019/04/20.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1816
  • Health checker info. : "Aborted pilot jobs" has been found since 11:20:00 UTC on 2019/04/06 and since 05:20:00 UTC on 2019/04/12. and since 20:20:00 UTC on 2019/04/17.
  • Health checker info. : Short pilot jobs" has been found at 23:20:00 UTC on 2019/04/10 and 15:20:00 UTC on 2019/04/14.
  • Health checker info. : "Short pilot jobs" has been found at 07:20:00 UTC on 2019/03/29.(details
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1768

DIRAC.Nagoya.jp

  • "Failed Payload Job" has been observed since 2020-01-22 12:53 UTC (for 10 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2251

  • "Short Pilot" has been observed since 2020-01-04 21:34 UTC (for 9 hours)
  • "Failed Payload Job" has been observed since 2020-01-04 23:34 UTC (for 7 hours)
  • Following JIRA tickets submitted: BIIDCO-2178 , BIIDCO-2112
  • Job stalling since 2019-12-14 10:00(UTC) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2178
  • "Short Pilot" has been observed since 2019-11-19 05:35 UTC (for 1 hours)
  • "Short Pilot" has been observed since 2019-11-11 11:30 UTC (for 3 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  • "Short Pilot" has been observed since 2019-11-10 05:30 UTC (for 1 hours)
  • Health checker info. : "Short pilot jobs" has been found since 09:20:00 UTC on 2019/10/09.

DIRAC.Nara-WU.jp

  • Job submission check : Pilot submission failure has been found at 06:23:00 UTC at 2019/10/03.
  • Under commissioning from 2018-11-13
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1432

DIRAC.NDU.jp

  • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 10 hours)
  • Following JIRA tickets submitted: BIIDCO-2112
  • "Short Pilot" has been observed since 2020-01-04 02:34 UTC (for 4 hours)
  • Following JIRA tickets submitted: BIIDCO-2112
  • "Short Pilot" has been observed since 2019-11-11 10:30 UTC (for 4 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112

DIRAC.Niigata.jp

  • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
  • "Short Pilot" has been observed since 2019-11-19 05:35 UTC (for 1 hours)

  • Health checker info. : "Short pilot jobs" has been found at 15:20:00 UTC on 2019/10/09.

  • Job submission check : Pilot submission failure has been found since 19:26:00 UTC on 2019/05/26. (details)

  • Health checker info. : "Aborted pilot jobs" has been found since 12:20:00 UTC on 2019/05/18.
  • Job submission check : Pilot submission failure has been found since 13:30:00 UTC on 2019/05/14. (details)
  • Health checker info. : "Aborted pilot jobs" has been found at 06:20:00 UTC on 2019/04/21.

DIRAC.Niigata2.jp       

  • Date, Issue, Tickets..

DIRAC.Osaka-CU.jp

  • Site is banned
  • "Pilot Submission Failure" has been observed since 2020-01-03 09:34 UTC (for 84 hours)
  • Following JIRA tickets submitted: BIIDCO-1434 , BIIDCO-2205
  • "Pilot Submission Failure" has been observed since 2020-01-03 09:34 UTC (for 21 hours)
  • Following JIRA tickets submitted: BIIDCO-1434 , BIIDCO-2205
  • "Pilot Submission Failure" has been observed since 2020-01-01 00:34 UTC (for 30 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1434
  • "Pilot Submission Failure" has been observed since 2020-01-01 00:34 UTC (for 6 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1434
  • "Pilot Submission Failure" has been observed since 2019-12-31 20:34 UTC (for 2 hours)
  • "Pilot Submission Failure" has been observed since 2019-12-30 16:34 UTC (for 14 hours)  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1434
  • "Pilot Submission Failure" has been observed since 2019-12-25 04:28 UTC (for 2 hours)
  • "Pilot Submission Failure" has been observed since 2019-12-24 04:28 UTC (for 9 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2205
  • "Pilot Submission Failure" has been observed since 2019-12-24 04:28 UTC (for 2 hours)
  • Job submission check : Pilot submission failure has been found since 06:21:00 UTC on 2019/04/02
  • Job submission check : Pilot submission failure has been found since 07:23:00 UTC on 2018/12/04. (details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1434
  • Health checker info. : "Short pilot jobs" has been found since 22:20:00 UTC on 2018/03/17.
    → Ask site admin to check the status 2018-03-17 10:00 JST. (DB access failure again from DIRAC.Osaka-CU.jp to PNNL from 2018-03-16 11:00 UTC)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-290

DIRAC.PAU.in

  • "Pilot Submission Failure" has been observed since 2020-01-22 20:53 UTC (for 2 hours)

DIRAC.PNNL.us

  • Site to be decommissioned
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-919

DIRAC.PNNL2.us

  • Site to be decommissioned
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-920

DIRAC.PNNL-CASCADE.us

  • Seeing no jobs (no plot) is not a problem

DIRAC.PNNL-PIC.us

  • Seeing no jobs (no plot) is not a problem

DIRAC.RCNP.jp

  • "Failed Payload Job" has been observed since 2020-01-05 02:34 UTC (for 4 hours)
  • "Short Pilot" has been observed since 2020-01-04 02:34 UTC (for 4 hours)

DIRAC.LocalTest.jp

  • Downtime:
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2182
  • "Short Pilot" has been observed since 2019-11-11 10:30 UTC (for 4 hours). 
  • Health checker info. : "Short pilot jobs" has been found since 09:20:00 UTC on 2019/10/09

DIRAC.SSU.kr

  • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
  • Following JIRA tickets submitted: BIIDCO-2112 , BIIDCO-2106 , BIIDCO-1982
  • "Short Pilot" has been observed since 2019-11-11 10:30 UTC (for 4 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  • Health checker info. : "Short pilot jobs" has been found at 13:20:00 UTC on 2019/10/03.(details
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2106
  • Health checker info. : "Short pilot jobs" has been found since 23:20:00 UTC on 2019/08/27
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1982
  • "Short pilot jobs" has been found at 01:20:00 UTC on 2019/08/26. 
  • Health checker info. : "Short pilot jobs" has been found at 04:20:00 UTC on 2019/08/21.(details)
  • Health checker info. : "Short pilot jobs" has been found at 23:20:00 UTC on 2019/08/21.(details)

DIRAC.TIFR.in

  • "Short Pilot" has been observed since 2020-01-12 04:53 UTC (for 337 hours)
  • "Failed Payload Job" has been observed since 2020-01-14 09:53 UTC (for 284 hours)
  • "Short Pilot" has been observed since 2020-01-12 04:53 UTC (for 314 hours)
  • "Failed Payload Job" has been observed since 2020-01-14 09:53 UTC (for 261 hours)
  • Following JIRA tickets submitted: BIIDCO-2235 , BIIDCO-2234 , BIIDCO-2112 , BIIDCO-1132 , BIIDCO-714
  • "Short Pilot" has been observed since 2020-01-12 04:53 UTC (for 290 hours)
  • "Failed Payload Job" has been observed since 2020-01-14 09:53 UTC (for 237 hours)
  • Following JIRA tickets submitted: BIIDCO-2235 , BIIDCO-2234 , BIIDCO-2112 , BIIDCO-1132 , BIIDCO-714
  • "Short Pilot" has been observed since 2020-01-12 04:53 UTC (for 74 hours)
  • "Failed Payload Job" has been observed since 2020-01-14 09:53 UTC (for 21 hours)
  • "Failed Payload Job" has been observed since 2020-01-12 05:53 UTC (for 11 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2235
  • "Short Pilot" has been observed since 2020-01-12 04:53 UTC (for 11 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2234

  • "Short Pilot" has been observed since 2019-11-11 10:30 UTC (for 4 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  • "Short Pilot" has been observed since 2019-11-10 04:30 UTC (for 2 hours)
  • Following JIRA tickets submitted: BIIDCO-1132 , BIIDCO-714
  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/10/06.
  • Job submission check : Pilot submission failure has been found at 14:25:00 UTC on 2019/05/11. (details)
  • Health checker info. : "Short pilot jobs" has been found since 14:20:00 UTC on 2019/05/10.(details)
  • Health checker info. : "Short pilot jobs" has been found since 13:20:00 UTC on 2018/10/22.
  • Job status plots, "Application Finished With Errors" has been found at about 00:00:00 JST on 2018/07/06. (details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1132
  • Health checker info. : "Short pilot jobs" -- Already reported: 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-971
  •  RunningLimit is set for MCProduction=1
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1006
  • Job stalled at input data resolution
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-714

DIRAC.TMU.jp

  • "Short Pilot" has been observed since 2020-01-24 02:53 UTC (for 28 hours)
  • Following JIRA tickets submitted: BIIDCO-2112 , BIIDCO-1522
  • "Short Pilot" has been observed since 2020-01-24 02:53 UTC (for 4 hours)
  • Following JIRA tickets submitted: BIIDCO-2112 , BIIDCO-1522
  • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
  • Following JIRA tickets submitted: BIIDCO-2112 , BIIDCO-1522
  • "Short Pilot" has been observed since 2019-11-11 11:30 UTC (for 3 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  • "Pilot Submission Failure" has been observed since 2019-10-27 12:30 UTC (for 2 hours)

  • Health checker info. : "Failed to install DIRAC on " has been found since 14:20:00 UTC on 2019/04/24.
  • Job submission check : Pilot submission failure has been found at 13:27:00 UTC on 2019/03/19.
  • Health checker info. : "Short pilot jobs" has been found since 10:20:00 UTC on 2018/11/02
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1522

DIRAC.Tokyo.jp

  • Decommissioned
  • Date, Issue, Tickets..

DIRAC.UAS.mx

  • "Short Pilot" has been observed since 2019-11-11 11:30 UTC (for 3 hours)  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  • "Short Pilot" has been observed since 2019-11-10 05:30 UTC (for 1 hours)
  • Following JIRA tickets submitted: BIIDCO-1772 , BIIDCO-1508
  • Health checker info. : "Belle II software could not be installed on " has been found since 15:20:00 UTC on 2019/04/25
  • Job submission check : Pilot submission failure has been found since 00:21:00 UTC on 2019/04/04. (details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1772
  • Health checker info. : "Belle II software could not be installed on " has been found since 01:20:00 UTC on 2019/02/20.
  • Job submission check: 100% failed with errors from 22:00 2019/01/08 till 04:00 2019/01/09 (UTC)
  • Health checker info. : "Belle II software could not be installed on " has been found since 04:20:00 UTC on 2018/12/17. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1508
  • Health checker info. : "Belle II software could not be installed on " has been found since 16:20:00 UTC on 2018/11/14.
  • Job submission check : Pilot submission failure has been found since 01:26:00 UTC on 2018/09/21. (details)

DIRAC.UVic.ca

  • Downtime 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2233
  • "Failed Payload Job" has been observed since 2020-01-05 00:34 UTC (for 6 hours)
  • Following JIRA tickets submitted: BIIDCO-2188 , BIIDCO-2137 , BIIDCO-2124
  • Downtime

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2188

  • 2019-11-27 Downtime 2019-11-27 17:00-18:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2137
  • Many stalled jobs 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2124

DIRAC.UVic-local.ca

  • Downtime 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2233
  • "Failed Payload Job" has been observed since 2020-01-05 00:34 UTC (for 6 hours)
  • Following JIRA tickets submitted: BIIDCO-2140
  • Downtime
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2189
  • Downtime
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2228
  •  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2150

    GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=144312 has been submitted
  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/10/06.
  • User jobs failed on the site:
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1975
  • Job status check: "Input Data Resolution" issues (13% overall, 100% in past hours) on 2019/05/16 at 7:00 UTC.
  • Health checker info. : "Short pilot jobs" has been found since 04:20:00 UTC on 2019/05/16.(details)
  • Health checker info. : "Belle II software could not be installed on " has been found since 04:20:00 UTC on 2019/05/13.

DIRAC.Yamagata.jp

  • "Short Pilot" has been observed since 2019-11-19 13:35 UTC (for 1 hours)

  • "Short Pilot" has been observed since 2019-11-18 11:35 UTC
  • "Short Pilot" has been observed since 2019-11-18 06:35 UTC (for 1 hours) (details).

  • Health checker info. : "Short pilot jobs" has been found since 21:20:00 UTC on 2019/06/05.(details)

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1861

  • Health checker info. : "Short pilot jobs" has been found at 22:20:00 UTC on 2019/03/13.(details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1761

DIRAC.Yonsei.kr

  • "Short Pilot" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
  • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 1 hours)
  • Following JIRA tickets submitted: BIIDCO-2112
  • "Failed Payload Job" has been observed since 2019-12-30 21:34 UTC (for 1 hours)
  • "Short Pilot" has been observed since 2019-11-11 11:30 UTC (for 3 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
     
  • Date, Issue, Tickets..

DIRAC.LocalTest.jp

  • Date, Issue, Tickets..

LCG.CESNET.cz

  • "Failed Payload Job" has been observed since 2020-01-04 21:34 UTC (for 10 hours)
  • Following JIRA tickets submitted: BIIDCO-2112 , BIIDCO-771
  • "Failed Pilot" has been observed since 2019-12-24 16:28 UTC (for 6 hours)
  • No jobs running since 2019-12-23 23:00:00 UTC, approximately.
  • "Short Pilot" has been observed since 2019-11-21 21:35 UTC (for 11 hours)
  • "Short Pilot" has been observed since 2019-11-20 18:35 UTC (for 12 hours) (details). 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  • Health checker info. : "Failed pilot jobs" has been found at 06:20:00 UTC on 2019/05/15.(details)
  • Job submission check : Pilot submission failure has been found at 06:26:00 UTC on 2019/05/15. (details)
  • Health checker info. : "Failed pilot jobs" has been found since 20:20:00 UTC on 2019/05/13.(details)
  •   Need some intervention to run Merge jobs
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-771

LCG.COSENZA.IT

  • "Failed Payload Job" has been observed since 2020-01-05 04:34 UTC (for 3 hours)
  • "Short Pilot" has been observed since 2019-11-22 04:35 UTC (for 3 hours)
  • "Failed Payload Job" has been observed since 2019-11-21 21:35 UTC (for 10 hours)
  • Downtime 2019-11-12 13:00 - 2019-11-19 19:00 (UTC)

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2128

  • "Short Pilot" has been observed since 2019-11-11 10:30 UTC (for 4 hours)

LCG.CNAF.it

  • Downtime 2020-01-22 23:00 to 2020-01-23 19:00 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2250
  • "Failed Payload Job" has been observed since 2020-01-04 21:34 UTC (for 10 hours)
  • Following JIRA tickets submitted: BIIDCO-2140
  • Downtime 2019-12-09 08:00 to 2019-12-09 10:30 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2167
  • 2019-11-27 Downtime 2019-11-27 9:00-16:00 UTC 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2136

  • "Short Pilot" has been observed since 2019-11-11 09:30 UTC (for 5 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
     
  • "Short Pilot" has been observed since 2019-11-10 05:30 UTC (for 1 hours)
  • Health checker info. : "Short pilot jobs" has been found since 11:20:00 UTC on 2019/10/03.(details)

LCG.CYFRONET.pl

  • "Pilot Submission Failure" has been observed since 2020-02-08 04:59 UTC (for 25 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2278
  • "Failed Payload Job" has been observed since 2019-12-23 10:28 UTC (for 12 hours)
  • The job status plot of B2PlotDisplay shows jobs stalled for two hours, as of 2019-12-23 10:35:17 UTC. Before this incident, there was no job at all on this site. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2201
  • "Short Pilot" has been observed since 2019-11-21 21:35 UTC (for 11 hours)
  • "Failed Payload Job" has been observed since 2019-11-21 21:35 UTC (for 11 hours)
  • Following JIRA tickets submitted: BIIDCO-2112 , BIIDCO-1920 , BIIDCO-1246
  • "Failed Payload Job" has been observed since 2019-11-20 13:35 UTC (for 1 hours)

  • "Short Pilot" has been observed since 2019-11-11 11:30 UTC (for 3 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  •  "Short Pilot" has been observed since 2019-11-10 05:30 UTC (for 1 hours)
  • Following JIRA tickets submitted: BIIDCO-1920 , BIIDCO-1246
  • Downtime:  2019-07-11 10:00  (UTC) -  2019-12-10 22:00 (UTC)
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1920
  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/10/06.
  • Job submission check : Pilot submission failure has been found since 14:23:00 UTC on 2019/07/31.
  • Health checker info. : "Short pilot jobs" has been found since 13:20:00 UTC on 2018/12/13.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1246

LCG.DESY.de

  • The site to be retired 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1240
     – No more jobs to be submitted.

LCG.Frascati.it

  •  Site is currently Banned due to hardware problem since 2019-07-05

  • Downtime 2019-12-15 06:41 UTC - 2019-12-16 07:41 UTC:
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2180
  • Downtime 2019-12-16 07:30 (UTC) - 2019-12-18 17:00 (UTC)

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2186

  • Job submission check : Pilot submission failure has been found since 14:24:00 UTC on 2019/05/24. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1882
    • GGUS 141688 ticket submitted.
  • Health checker info. : "BLAH ERROR" has been found since 15:20:00 UTC on 2019/05/21.(details)

LCG.HEPHY.at

  • Downtime 2020-02-06 15:00 (UTC) - 2020-02-10 15:00 (UTC)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2274
  • "Failed Payload Job" has been observed since 2020-01-05 05:34 UTC (for 2 hours)
  • Health checker info. : "Failed pilot jobs" has been found at 13:20:00 UTC on 2019/10/03.(details)
  • Health checker info. : "Failed pilot jobs" has been found at 15:20:00 UTC on 2019/05/22.(details)
  • Health checker info. : "Short pilot jobs" has been found at 15:20:00 UTC on 2019/04/12.
  • Health checker info. : "Failed pilot jobs" has been found at 02:20:00 UTC on 2019/01/30.(details) and at 02:20:00 UTC on 2019/01/31.(details)
  • submission check : Pilot submission failure has been found at 14:22:00 UTC on 2018/12/27.

LCG.IPHC.fr

  • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 11 hours)
  • Following JIRA tickets submitted: BIIDCO-2191
  • Downtime:
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2191
  • Health checker info. : "Failed pilot jobs" has been found at 00:20:00 UTC on 2018/06/18.(details)

LCG.KEK.jp

  • Job status: Large number of jobs finished with errors (61.0%) in last 24 hour period, from approx. 2020-01-01 00:00 - 02:00 UTC
  • Downtime:

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2181

  • Health checker info. : "Short pilot jobs" has been found since 05:20:00 UTC on 2019/10/09

  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/10/06.
  • SiteDirector "Failed to check the availability" 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1934
  • Pilot job failure with Sandbox upload error
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2179

LCG.KEK2.jp

  • Downtime:
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2181
  • Health checker info. : "Short pilot jobs" has been found at 16:20:00 UTC on 2019/10/09.

  • Still all jobs failing with InputDataResolution on 2019/07/25.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1542
  • GGUS ticket : "KEK SE: PrepareToGet ETIMEDOUT for a specific file path"(140328) has been submited at 21:26:29 UTC on 2019/03/21.
  • Health checker info. : "Short pilot jobs" has been found at 11:20:00 UTC on 2019/03/22.(details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1741

  • Health checker info. : "Failed pilot jobs" has been found since 13:20:00 UTC on 2018/12/21.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1559
  • all jobs are in "Input data resolution" status since 12.00 2018/12/18 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1542

LCG.KEK-merge.jp

  • Downtime:
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2181
  • Health checker info. : "Short pilot jobs" has been found at 00:20:00 UTC on 2019/08/26.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1978
  •   Most jobs failing with InputDataResolution
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1776
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1777
  • "Belle II software could not be installed on cb268.cc.kek.jp" has been found since 14:20:00 UTC on 2019/04/05
  • Health checker info. : "Short pilot jobs" has been found since 20:20:00 UTC on 2019/04/02
  •   being commissioned...

LCG.KISTI.kr

  • Downtime:
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2247
  • "Short Pilot" has been observed since 2019-11-20 13:35 UTC (for 17 hours) 

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112

  • Jobs slots are disabled for SE maintenance from 2018-10-19 to 2018-10-23

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1380

  • Health checker info. : "BLAH ERROR" has been found since 06:20:00 UTC on 2018/10/19.(details)

  • "Short pilot jobs" has been found at 06:20:00 UTC on 2018/10/09.(details)
  • BLAH error seems to be happen if jobs exceed the allocated # of queues, not a problem (Site specific feature)  
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1259
  • A large number of Merge jobs in waiting status
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-773

LCG.KMI.jp

  • "Pilot Submission Failure" has been observed since 2020-01-31 11:59 UTC (for 2 hours) (details) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2263

    "Short Pilot" has been observed since 2020-01-05 02:34 UTC (for 5 hours)
  • "Failed Payload Job" has been observed since 2020-01-04 22:34 UTC (for 9 hours)
  • Following JIRA tickets submitted: BIIDCO-2112 , BIIDCO-1533
  • "Failed Payload Job" has been observed since 2019-11-19 12:35 UTC (for 2 hours)
  • "Short Pilot" has been observed since 2019-11-11 11:30 UTC (for 3 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  • Job status check: Application finished with errors (7% of the jobs in last 24 hours) on 2018/12/21 at 8:48 UTC.
  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2018/12/13.(details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1533

  • Health checker info. : "Belle II software could not be installed on pwn22.local" has been found since 21:20:00 UTC on 2018/11/22.
  • Job submission check : Pilot submission failure has been found since 21:24:00 UTC on 2018/10/02. (details)

LCG.LAL.fr

  • 2020-01-02 13:20 UTC File transfer failure from KMI-TMP-SE and from KEK-Disk-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219
  • 2020-01-02 09:15 UTC File transfer failure from KMI-TMP-SE to LAL-DATA-SE
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2219
  • Downtime:
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2190
  • Downtime 2019-12-02 15:00 - 2019-12-06 15:00 (UTC) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2157
  • Downtime 2019-11-21 09:00-12:00 (UTC) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2133
  • "Failed Payload Job" has been observed since 2019-11-18 05:35 UTC (for 2 hours)
  • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/05/01.(details)

LCG.Legnaro.it

  • Date, Issue, Tickets...

LCG.Napoli.it

  • Downtime from 01-28 17:30 to 2020-01-29 16:28 

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2239


  • schedule downtime 2020-01-15 11:00 - 2020-01-15 12:00 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2239
  • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 11 hours)
  • Following JIRA tickets submitted: BIIDCO-2140
  • "Failed Payload Job" has been observed since 2020-01-04 02:34 UTC (for 4 hours)
  • Following JIRA tickets submitted: BIIDCO-2140
  • "BLAH Error" has been observed since 2019-12-25 22:27:58 UTC (for 7 hours)

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2210

  • "Aborted Pilot" has been observed since 2019-12-02 09:35 UTC (for 5 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2155
  • Downtime 2019-11-12 13:00 - 2019-11-19 19:00 (UTC)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2128
  • "Pilot Submission Failure" has been observed since 2019-11-16 06:35 UTC (for 32 hours)
  • Health checker info. : "Failed pilot jobs" has been found at 14:20:00 UTC on 2019/10/06.
  • Job submission check : Pilot submission failure has been found since 12:27:00 UTC on 2019/10/02.

  •  t2-recas-ce01.na.infn.it shows pilot submission error and this CE should  be banned till 2019 September.

  • Stalled jobs
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1255

LCG.NTU.tw

  • "BLAH Error" has been observed since 2020-02-13 14:53:07 UTC (for 24 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2276
  • "BLAH Error" has been observed since 2020-02-07 06:53:04 UTC (for 351 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2276
  • "BLAH Error" has been observed since 2020-01-26 05:52:37 UTC (for 62 hours)
  • Following JIRA tickets submitted: BIIDCO-2140
  • Following JIRA tickets submitted: BIIDCO-2140
  • "BLAH Error" has been observed since 2020-01-22 22:52:36 UTC (for 1 hours)
  • "Pilot Submission Failure" has been observed since 2020-01-14 17:53 UTC (for 13 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2241
  • "Failed Payload Job" has been observed since 2020-01-05 01:34 UTC (for 6 hours)
  • "Pilot Submission Failure" has been observed since 2019-12-26 09:34 UTC(for 5 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2211

    Solved and verified GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=144666 has been submitted
  • "Pilot Submission Failure" has been observed since 2019-12-24 15:28 UTC (for 7 hours)
  • "Pilot Submission Failure" has been observed since 2019-12-23 20:28 UTC (for 2 hours)(details).
  • "Failed Payload Job" has been observed since 2019-12-23 21:28 UTC (for 1 hours)
  • According to the JobStatus of B2PlotDisplay, jobs were finished with errors as of 2019-12-23 13:32:55 for 6 hours.
  • "CRL has expired" for "node35-0,node19-0,node36-0,node33-0,node38-0,node37-0,node39-0" has been observed since 2019-12-18 23:27:52 UTC (for 6 hours)  

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2192

  • Solved and verified: GGUS ticket https://ggus.eu/index.php?mode=ticket_info&ticket_id=144612
  • "Failed Payload Job" has been observed since 2019-12-10 05:28 UTC (for 1 hours)
  • Downtime  2019-12-06 08:00 (UTC)  - 2019-12-09 04:00 (UTC)  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2165
  • 2019-11-27 Downtime 2019-11-22 20:00 (UTC) - 2019-11-29 20:00 (UTC)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2138
  • "CRL has expired" for "node38-0,node32-0,node33-0,node37-0,node39-0,node34-0,node35-0" has been observed since 2019-11-17 14:30:48 UTC (for 53 hours)

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2123

  • Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2140

    Solved and verified: GGUS ticket https://ggus.eu/?mode=ticket_info&ticket_id=144166

  • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/04/18
  • Downtime 2020-01-10 09:00  - 2020-01-13 05:00   (UTC)  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2231

LCG.Pisa.it

  • "Pilot Submission Failure" has been observed since 2020-02-03 00:59 UTC (for 254 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1791

  • "Failed Pilot" has been observed since 2020-02-08 13:59 UTC (for 1 hour)
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1619

  • "Failed Pilot" has been observed since 2020-02-06 13:59 UTC (for 17 hours) 
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1619
  • "Pilot Submission Failure" has been observed since 2020-02-03 00:59 UTC (for 102 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1791
  • Following GGUS tickets are submitted:
  • "Failed Pilot" has been observed since 2020-01-05 03:34 UTC (for 4 hours)
  • "Pilot Submission Failure" has been observed since 2019-10-28 03:30 UTC (for 1660 hours)
  • "Failed Payload Job" has been observed since 2020-01-05 02:34 UTC (for 5 hours)
  • Following JIRA tickets submitted: BIIDCO-1791 , BIIDCO-1591 , BIIDCO-1157
  • "Failed Pilot" has been observed since 2020-01-01 00:34 UTC (for 54 hours)
  • "Failed Pilot" has been observed since 2020-01-01 00:34 UTC (for 30 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1791
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1619

  • "Failed Pilot" has been observed since 2020-01-01 00:34 UTC (for 6 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1791
     
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1619

  •  "Failed Pilot" has been observed since 2019-12-31 21:34 UTC (for 1 hours)
  • "Pilot Submission Failure" has been observed since 2019-10-28 03:30 UTC (for 1338 hours)
  • "Pilot Submission Failure" has been observed since 2019-10-28 03:30 UTC 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1791
  • "Failed Pilot" has been observed since 2019-10-28 03:30 UTC
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1619
       GGUS ticket:  https://ggus.eu/?mode=ticket_info&ticket_id=144073
  • Job submission check : Pilot submission failure has been found since 10:25:00 UTC on 2019/04/12 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1791
  • Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1591

    "Failed to install DIRAC on so1wn8.pi.infn.it,n2wn13.pi.infn.it,n2wn18.pi.infn.it,so1wn6" has been found since 01:20:00 UTC on 2019/01/02.
  • Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1157

    "Short pilot jobs" has been found since 02:20:00 UTC on 2018/09/21.(details)

LCG.Roma3.it

  • Health checker info. : "Failed pilot jobs" has been found at 14:20:00 UTC on 2019/04/20.
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1538

LCG.TAU.il

  • "Failed Payload Job" has been observed since 2020-01-05 00:34 UTC (for 7 hours)
  • Following JIRA tickets submitted: BIIDCO-2187 , BIIDCO-2112 ,
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2229
  • Downtime
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2187
  • "Failed Pilot" has been observed since 2020-01-09 13:53 UTC (for 9 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2229
  • "Short Pilot" has been observed since 2019-11-22 02:35 UTC (for 5 hours)
  • "Short Pilot" has been observed since 2019-11-11 12:30 UTC (for 2 hours)

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112

  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/10/06.
  • Health checker info. : "Failed pilot jobs" has been found since 19:20:00 UTC on 2019/05/24.(details)

LCG.Torino.it

  • "Aborted Pilot" has been observed since 2020-02-12 13:59 UTC (for 9 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2285
  • "BLAH Error" has been observed since 2020-02-09 05:53:05 UTC (for 9 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2279
  • "BLAH Error" has been observed since 2020-02-08 14:53:04 UTC (for 1 hours)
  • "Pilot Submission Failure" has been observed since 2020-01-26 02:53 UTC (for 3 hours)
  • Following JIRA tickets submitted: BIIDCO-2215
  • "Pilot Submission Failure" has been observed since 2020-01-22 21:53 UTC (for 1 hours)
  • "Pilot Submission Failure" has been observed since 2020-01-05 02:34 UTC (for 5 hours)
  • Following JIRA tickets submitted: BIIDCO-2215
  • "Failed Payload Job" has been observed since 2020-01-04 04:34 UTC (for 2 hours)
  • Following JIRA tickets submitted: BIIDCO-2215
  • "Pilot Submission Failure" has been observed since 2019-12-28 18:34 UTC 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2215
  • Job status check: 4.5% of jobs identified as stalled by the watchdog since 2019-12-25 22:00:00 UTC (for 12 hours)
  • "Pilot Submission Failure" has been observed since 2019-12-23 13:28 UTC (for 9 hours)
  • "Short Pilot" has been observed since 2019-11-10 05:30 UTC (for 1 hours) 
  • Job submission check : Pilot submission failure has been found at 13:26:00 UTC on 2019/05/20. (details)
  • Job submission check : Pilot submission failure has been found at 14:25:00 UTC on 2019/05/11. (details)
  • Job submission check : Pilot submission failure has been found at 14:25:00 UTC on 2019/05/09

  • Job submission check : Pilot submission failure has been found at 06:25:00 UTC on 2019/03/23.

LCG.ULAKBIM.tr

  • schedule downtime fro 2020-01-14 07:00 to 2020-01-16 13:00 (UTC)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2237
  • The queue 'belle7' to be disabled. use only 'belle'
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1896
  • Health checker info. : "Aborted pilot jobs" has been found since 01:20:00 UTC on 2019/08/01.

OSG.BNL.us

  • Pilot submission failure observed since 2020-01-28 01:53 UTC  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2177
    .
  • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 11 hours)
  • Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2140
  • Job status: 81.0% of jobs finished with error between 2020-01-01 02:00 - 2020-01-02 02:00 UTC.
  • "Failed Payload Job" has been observed since 2019-12-20 14:28 UTC 
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2195
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2193

    Solved and verified GGUS ticket : https://ggus.eu/index.php?mode=ticket_info&ticket_id=144665 has  been submitted
  • "Belle II software could not be installed" for "bgk01.sdcc.bnl.gov" has been observed since 2019-12-18 15:27:52 UTC
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2194
  • All pilot submission failure to bgk02.sdcc.bnl.govsince 2019-12-11
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2177
  • "Pilot Submission Failure" has been observed since 2019-12-05 05:28 UTC
  • "Short Pilot" has been observed since 2019-11-21 21:35 UTC

  • "Short Pilot" has been observed since 2019-11-11 10:30 UTC (for 4 hours)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
     
  • Health checker info. : "Short pilot jobs" has been found since 13:20:00 UTC on 2019/08/01.
  • Health checker info. : "Belle II software could not be installed on " has been found since 19:20:00 UTC on 2019/02/14.
  • Job submission check: Jobs fail with errors or input data resolution the last 24h (6:00 UTC, 2019/01/09) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1596
  • Production jobs: UNAVAILABLE files
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1302
  • Number of concurrent MCProduction jobs restricted
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1256
  •  MCProduction jobs are mostly stalled
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1253

OSG.CORI.us

  • OSG.CORI.us resource has been removed because CY18 allocation was not approved

OSG.UMiss.us

  • "Short Pilot" has been observed since 2020-01-14 11:53 UTC (for 28 hours)

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2243

    "Short Pilot" has been observed since 2020-01-12 22:53 UTC (for 32 hours)

  • "Short Pilot" has been observed since 2019-11-22 05:35 UTC (for 3 hours)
  • Following JIRA tickets submitted: BIIDCO-2112 , BIIDCO-1863 , BIIDCO-1856 , BIIDCO-1768
  • "Short Pilot" has been observed since 2019-11-21 04:35 UTC (for 2 hours) (details).
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  • "Short Pilot" has been observed since 2019-11-11 11:30 UTC (for 3 hours) 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2112
  • Health checker info. : "Short pilot jobs" has been found since 21:20:00 UTC on 2019/07/10. (screenshot)
  • Health checker info. : "Short pilot jobs" has been found since 21:20:00 UTC on 2019/07/08. (screenshot)
  • Health checker info. : "Short pilot jobs" has been found at 06:20:00 UTC on 2019/07/03.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1863
  • Health checker info. : "Short pilot jobs" has been found since 09:20:00 UTC on 2019/06/27
  • Health checker info. : "Short pilot jobs" has been found since 17:20:00 UTC on 2019/06/04.(details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1863
  • Health checker info. : "Short pilot jobs" has been found at 23:20:00 UTC on 2019/06/03.(details)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1863
  • Health checker info. : "Aborted pilot jobs" has been found at 22:20:00 UTC on 2019/06/02.
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1856
  • Health checker info. : "Short pilot jobs" has been found at 14:20:00 UTC on 2019/05/20.(details)
  • Job status check: 100% of issues of Input Data Resolution on 2019/05/14 at 7:00 UTC.
  • Health checker info. : "Short pilot jobs" has been found at 07:20:00 UTC on 2019/05/14.(details)
  • Health checker info. : "Short pilot jobs" has been found since 22:20:00 UTC on 2019/05/12.(details)
    Updated
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1768
  • Job submission check : Pilot submission failure has been found since 12:27:00 UTC on 2019/05/04. (details)
  • Health checker info. : "Short pilot jobs" has been found since 07:20:00 UTC on 2019/05/11.(details)
  • Health checker info. : "Short pilot jobs" has been found since 14:20:00 UTC on 2019/04/11 and  at 17:20:00 UTC on 2019/04/14.
  • Job status check: 34.7% appl. finshed with errors on 2019/04/08.

SSH.KMI.jp

  • "Short Pilot" has been observed since 2019-12-24 08:28 UTC (for 5 hours)  
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2206
  • Job status plot: input data resolution problems (for 7 hours) since 2019-12-24 00:00 UTC, approximately.
  • "Short Pilot" has been observed since 2019-12-24 05:28 UTC (for 1 hours)
  • Job status check: Application finished with errors (12% of the jobs in last 24 hours) on 2018/12/22 at 11:30 UTC.
  • Health checker info. : "Short pilot jobs" has been found at 20:20:00 UTC on 2018/08/13.

Test.KIT.de

  • "Failed Pilot" has been observed since 2020-01-19 21:53 UTC (for 1 hours)
  • "Aborted Pilot" has been observed since 2020-01-16 21:53 UTC (for 1 hours)
  • Downtime on 2020-01-16 09:00 to 2020-01-16 09:30
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2244
  • Downtime on 2020-01-15 13:00 to 2020-01-15 15:00 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2240
  • Downtime on 2019-01-13  8:00 UTC  to 2020-01-13 16:00 UTC 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2236
  • "Failed Payload Job" has been observed since 2020-01-04 20:34 UTC (for 11 hours)
  • Following JIRA tickets submitted: BIIDCO-2140
  • Downtime on 2019-12-11  8:00 UTC  to 2020-01-24 22:00 UTC 
    Jira
    serverDESY JIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2173
  • "Pilot Submission Failure" has been observed since 2019-12-05 05:28 UTC 
  • "Aborted Pilot" has been observed since 2019-11-23 05:35 UTC (for 1 hours)
  • Test site for the opportunistic resources at KIT. No need to report problems.LCG.Pisa.it
  • Downtime 2020-01-09 14:38 - 2020-01-10 16:00 UTC. 
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2230

Test.ULAKBIM.tr

  • schedule downtime from 2020-01-14 07:00 to 2020-01-16 13:00 (UTC)
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2237
  • Test site for the SL7 resources at ULAKBIM. No need to report problems.
  • No activities expected currently.

VCYCLE.Napoli.it

  • "Failed Payload Job" has been observed since 2020-01-05 04:34 UTC (for 3 hours)
  • Following JIRA tickets submitted: BIIDCO-2143 , BIIDCO-1613 , BIIDCO-1612
  • "Failed Payload Job" has been observed since 2019-11-21 20:35 UTC (for 12 hours)
  • "Failed Payload Job" has been observed since 2019-11-29 00:35 UTC (for 6 hours)

    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-2143

  • Following JIRA tickets submitted: BIIDCO-1613 , BIIDCO-1612
  • "Failed Payload Job" has been observed since 2019-11-20 13:35 UTC (for 1 hours)

  • Opportunistic site (Empty plot is not a problem)
  •  Ban lifted
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1613
  • "Sudo CE Error: sudo execution fails with return code 1"
    Jira
    serverDESY JIRA
    serverIdd254f614-e16c-3f52-8b79-663658704a33
    keyBIIDCO-1612

VCYCLE.HNSC01.it, VCYCLE.HNSC02.it

  • Opportunistic site (Empty plot is not a problem)


Links