Page tree




File structure 2018

Extra trainids OPEN

Run 459 (pulse capacitor) of beamteime 201830/p900019, effects all channels

Channels have different number of sequences inside a run OPEN

For example: run 257 of commissioning beamtime 201830/p900019

2 sequences:

Channel 0, 1, 7, 8, 9, 11, 12, 13, 14, 15

3 sequences:

Channel 2, 3, 4, 5, 6, 10, 

% /gpfs/exfel/exp/SPB/201830/p900019/raw/r0257
% ls
RAW-R0257-AGIPD00-S00000.h5
RAW-R0257-AGIPD00-S00001.h5
RAW-R0257-AGIPD01-S00000.h5
RAW-R0257-AGIPD01-S00001.h5
RAW-R0257-AGIPD02-S00000.h5
RAW-R0257-AGIPD02-S00001.h5
RAW-R0257-AGIPD02-S00002.h5
RAW-R0257-AGIPD03-S00000.h5
RAW-R0257-AGIPD03-S00001.h5
RAW-R0257-AGIPD03-S00002.h5
RAW-R0257-AGIPD04-S00000.h5
RAW-R0257-AGIPD04-S00001.h5
RAW-R0257-AGIPD04-S00002.h5
RAW-R0257-AGIPD05-S00000.h5
RAW-R0257-AGIPD05-S00001.h5
RAW-R0257-AGIPD05-S00002.h5
RAW-R0257-AGIPD06-S00000.h5
RAW-R0257-AGIPD06-S00001.h5
RAW-R0257-AGIPD06-S00002.h5
RAW-R0257-AGIPD07-S00000.h5
RAW-R0257-AGIPD07-S00001.h5
RAW-R0257-AGIPD08-S00000.h5
RAW-R0257-AGIPD08-S00001.h5
RAW-R0257-AGIPD09-S00000.h5
RAW-R0257-AGIPD09-S00001.h5
RAW-R0257-AGIPD10-S00000.h5
RAW-R0257-AGIPD10-S00001.h5
RAW-R0257-AGIPD10-S00002.h5
RAW-R0257-AGIPD11-S00000.h5
RAW-R0257-AGIPD11-S00001.h5
RAW-R0257-AGIPD12-S00000.h5
RAW-R0257-AGIPD12-S00001.h5
RAW-R0257-AGIPD13-S00000.h5
RAW-R0257-AGIPD13-S00001.h5
RAW-R0257-AGIPD14-S00000.h5
RAW-R0257-AGIPD14-S00001.h5
RAW-R0257-AGIPD15-S00000.h5
RAW-R0257-AGIPD15-S00001.h5

Data acquisition part is shifted inside of run part different for each channel OPEN

DAQ starts with different train numbers for differently channels → the data acquired lies on different positions in each file

run 445 in the p02042

sequence 0

sequence 5

Empty sequences CLEARIFIED BUT MIGHT BE PROBLEMATIC FOR SOFTWARE

E.g

r0438 in beamtime 201701/p002042

last 3 sequences out of 5 do not contain data but only metadata (INDEX)

or

r0457 in beamtime 201701/p002042

last 11 sequences out of 17 do not contain data but only metadata (INDEX)

% /gpfs/exfel/exp/SPB/201701/p002042/raw/r0438
% ls -s 
8278528 RAW-R0438-AGIPD00-S00000.h5
8081920 RAW-R0438-AGIPD00-S00001.h5
    512 RAW-R0438-AGIPD00-S00002.h5
    512 RAW-R0438-AGIPD00-S00003.h5
    512 RAW-R0438-AGIPD00-S00004.h5
8294912 RAW-R0438-AGIPD01-S00000.h5
8065536 RAW-R0438-AGIPD01-S00001.h5
    512 RAW-R0438-AGIPD01-S00002.h5
    512 RAW-R0438-AGIPD01-S00003.h5
    512 RAW-R0438-AGIPD01-S00004.h5
8294912 RAW-R0438-AGIPD02-S00000.h5
8065536 RAW-R0438-AGIPD02-S00001.h5
    512 RAW-R0438-AGIPD02-S00002.h5
    512 RAW-R0438-AGIPD02-S00003.h5
    512 RAW-R0438-AGIPD02-S00004.h5
8278528 RAW-R0438-AGIPD03-S00000.h5
8081920 RAW-R0438-AGIPD03-S00001.h5
    512 RAW-R0438-AGIPD03-S00002.h5
    512 RAW-R0438-AGIPD03-S00003.h5
    512 RAW-R0438-AGIPD03-S00004.h5
8294912 RAW-R0438-AGIPD04-S00000.h5
8065536 RAW-R0438-AGIPD04-S00001.h5
    512 RAW-R0438-AGIPD04-S00002.h5
    512 RAW-R0438-AGIPD04-S00003.h5
    512 RAW-R0438-AGIPD04-S00004.h5
8278528 RAW-R0438-AGIPD05-S00000.h5
8081920 RAW-R0438-AGIPD05-S00001.h5
    512 RAW-R0438-AGIPD05-S00002.h5
    512 RAW-R0438-AGIPD05-S00003.h5
    512 RAW-R0438-AGIPD05-S00004.h5
8278528 RAW-R0438-AGIPD06-S00000.h5
8081920 RAW-R0438-AGIPD06-S00001.h5
    512 RAW-R0438-AGIPD06-S00002.h5
    512 RAW-R0438-AGIPD06-S00003.h5
    512 RAW-R0438-AGIPD06-S00004.h5
8294912 RAW-R0438-AGIPD07-S00000.h5
8065536 RAW-R0438-AGIPD07-S00001.h5
    512 RAW-R0438-AGIPD07-S00002.h5
    512 RAW-R0438-AGIPD07-S00003.h5
    512 RAW-R0438-AGIPD07-S00004.h5
8311296 RAW-R0438-AGIPD08-S00000.h5
8049152 RAW-R0438-AGIPD08-S00001.h5
    512 RAW-R0438-AGIPD08-S00002.h5
    512 RAW-R0438-AGIPD08-S00003.h5
    512 RAW-R0438-AGIPD08-S00004.h5
8294912 RAW-R0438-AGIPD09-S00000.h5
8065536 RAW-R0438-AGIPD09-S00001.h5
    512 RAW-R0438-AGIPD09-S00002.h5
    512 RAW-R0438-AGIPD09-S00003.h5
    512 RAW-R0438-AGIPD09-S00004.h5
8311296 RAW-R0438-AGIPD10-S00000.h5
8049152 RAW-R0438-AGIPD10-S00001.h5
    512 RAW-R0438-AGIPD10-S00002.h5
    512 RAW-R0438-AGIPD10-S00003.h5
    512 RAW-R0438-AGIPD10-S00004.h5
8311296 RAW-R0438-AGIPD11-S00000.h5
8049152 RAW-R0438-AGIPD11-S00001.h5
    512 RAW-R0438-AGIPD11-S00002.h5
    512 RAW-R0438-AGIPD11-S00003.h5
    512 RAW-R0438-AGIPD11-S00004.h5
8311296 RAW-R0438-AGIPD12-S00000.h5
8049152 RAW-R0438-AGIPD12-S00001.h5
    512 RAW-R0438-AGIPD12-S00002.h5
    512 RAW-R0438-AGIPD12-S00003.h5
    512 RAW-R0438-AGIPD12-S00004.h5
8294912 RAW-R0438-AGIPD13-S00000.h5
8065536 RAW-R0438-AGIPD13-S00001.h5
    512 RAW-R0438-AGIPD13-S00002.h5
    512 RAW-R0438-AGIPD13-S00003.h5
    512 RAW-R0438-AGIPD13-S00004.h5
8294912 RAW-R0438-AGIPD14-S00000.h5
8065536 RAW-R0438-AGIPD14-S00001.h5
    512 RAW-R0438-AGIPD14-S00002.h5
    512 RAW-R0438-AGIPD14-S00003.h5
    512 RAW-R0438-AGIPD14-S00004.h5
8294912 RAW-R0438-AGIPD15-S00000.h5
8065536 RAW-R0438-AGIPD15-S00001.h5
    512 RAW-R0438-AGIPD15-S00002.h5
    512 RAW-R0438-AGIPD15-S00003.h5
    512 RAW-R0438-AGIPD15-S00004.h5
% /gpfs/exfel/exp/SPB/201701/p002042/raw/r0457
% ls -s
7852032 RAW-R0457-AGIPD00-S00000.h5
8393216 RAW-R0457-AGIPD00-S00001.h5
8393216 RAW-R0457-AGIPD00-S00002.h5
8393216 RAW-R0457-AGIPD00-S00003.h5
8393216 RAW-R0457-AGIPD00-S00004.h5
7720960 RAW-R0457-AGIPD00-S00005.h5
    512 RAW-R0457-AGIPD00-S00006.h5
    512 RAW-R0457-AGIPD00-S00007.h5
    512 RAW-R0457-AGIPD00-S00008.h5
    512 RAW-R0457-AGIPD00-S00009.h5
    512 RAW-R0457-AGIPD00-S00010.h5
    512 RAW-R0457-AGIPD00-S00011.h5
    512 RAW-R0457-AGIPD00-S00012.h5
    512 RAW-R0457-AGIPD00-S00013.h5
    512 RAW-R0457-AGIPD00-S00014.h5
    512 RAW-R0457-AGIPD00-S00015.h5
    512 RAW-R0457-AGIPD00-S00016.h5
7868416 RAW-R0457-AGIPD01-S00000.h5
8393216 RAW-R0457-AGIPD01-S00001.h5
8393216 RAW-R0457-AGIPD01-S00002.h5
8393216 RAW-R0457-AGIPD01-S00003.h5
8393216 RAW-R0457-AGIPD01-S00004.h5
7704576 RAW-R0457-AGIPD01-S00005.h5
    512 RAW-R0457-AGIPD01-S00006.h5
    512 RAW-R0457-AGIPD01-S00007.h5
    512 RAW-R0457-AGIPD01-S00008.h5
    512 RAW-R0457-AGIPD01-S00009.h5
    512 RAW-R0457-AGIPD01-S00010.h5
    512 RAW-R0457-AGIPD01-S00011.h5
    512 RAW-R0457-AGIPD01-S00012.h5
    512 RAW-R0457-AGIPD01-S00013.h5
    512 RAW-R0457-AGIPD01-S00014.h5
    512 RAW-R0457-AGIPD01-S00015.h5
    512 RAW-R0457-AGIPD01-S00016.h5
7868416 RAW-R0457-AGIPD02-S00000.h5
8393216 RAW-R0457-AGIPD02-S00001.h5
8393216 RAW-R0457-AGIPD02-S00002.h5
8393216 RAW-R0457-AGIPD02-S00003.h5
8393216 RAW-R0457-AGIPD02-S00004.h5
7704576 RAW-R0457-AGIPD02-S00005.h5
    512 RAW-R0457-AGIPD02-S00006.h5
    512 RAW-R0457-AGIPD02-S00007.h5
    512 RAW-R0457-AGIPD02-S00008.h5
    512 RAW-R0457-AGIPD02-S00009.h5
    512 RAW-R0457-AGIPD02-S00010.h5
    512 RAW-R0457-AGIPD02-S00011.h5
    512 RAW-R0457-AGIPD02-S00012.h5
    512 RAW-R0457-AGIPD02-S00013.h5
    512 RAW-R0457-AGIPD02-S00014.h5
    512 RAW-R0457-AGIPD02-S00015.h5
    512 RAW-R0457-AGIPD02-S00016.h5
7868416 RAW-R0457-AGIPD03-S00000.h5
8393216 RAW-R0457-AGIPD03-S00001.h5
8393216 RAW-R0457-AGIPD03-S00002.h5
8393216 RAW-R0457-AGIPD03-S00003.h5
8393216 RAW-R0457-AGIPD03-S00004.h5
7704576 RAW-R0457-AGIPD03-S00005.h5
    512 RAW-R0457-AGIPD03-S00006.h5
    512 RAW-R0457-AGIPD03-S00007.h5
    512 RAW-R0457-AGIPD03-S00008.h5
    512 RAW-R0457-AGIPD03-S00009.h5
    512 RAW-R0457-AGIPD03-S00010.h5
    512 RAW-R0457-AGIPD03-S00011.h5
    512 RAW-R0457-AGIPD03-S00012.h5
    512 RAW-R0457-AGIPD03-S00013.h5
    512 RAW-R0457-AGIPD03-S00014.h5
    512 RAW-R0457-AGIPD03-S00015.h5
    512 RAW-R0457-AGIPD03-S00016.h5
7868416 RAW-R0457-AGIPD04-S00000.h5
8393216 RAW-R0457-AGIPD04-S00001.h5
8393216 RAW-R0457-AGIPD04-S00002.h5
8393216 RAW-R0457-AGIPD04-S00003.h5
8393216 RAW-R0457-AGIPD04-S00004.h5
7704576 RAW-R0457-AGIPD04-S00005.h5
    512 RAW-R0457-AGIPD04-S00006.h5
    512 RAW-R0457-AGIPD04-S00007.h5
    512 RAW-R0457-AGIPD04-S00008.h5
    512 RAW-R0457-AGIPD04-S00009.h5
    512 RAW-R0457-AGIPD04-S00010.h5
    512 RAW-R0457-AGIPD04-S00011.h5
    512 RAW-R0457-AGIPD04-S00012.h5
    512 RAW-R0457-AGIPD04-S00013.h5
    512 RAW-R0457-AGIPD04-S00014.h5
    512 RAW-R0457-AGIPD04-S00015.h5
    512 RAW-R0457-AGIPD04-S00016.h5
...

Explanation

Starting and stopping a run was decoupled form the data acquisition. If for some reason the run was not stopped after the acquisition was stopped, this results in an still ongoing run producing "empty" files.

Data for whole channels is missing OPEN

E.g. r0257 in beamtime 201830/p900019

Files for channels 2, 3, 4, 10, 14, 15 do not contain data at all (only metadata)

% /gpfs/exfel/exp/SPB/201830/p900019/raw/r0257
% ls -s
8393216 RAW-R0257-AGIPD00-S00000.h5
8393216 RAW-R0257-AGIPD00-S00001.h5
8393216 RAW-R0257-AGIPD01-S00000.h5
8393216 RAW-R0257-AGIPD01-S00001.h5
    512 RAW-R0257-AGIPD02-S00000.h5
    512 RAW-R0257-AGIPD02-S00001.h5
    512 RAW-R0257-AGIPD02-S00002.h5
    512 RAW-R0257-AGIPD03-S00000.h5
    512 RAW-R0257-AGIPD03-S00001.h5
    512 RAW-R0257-AGIPD03-S00002.h5
    512 RAW-R0257-AGIPD04-S00000.h5
    512 RAW-R0257-AGIPD04-S00001.h5
    512 RAW-R0257-AGIPD04-S00002.h5
8393216 RAW-R0257-AGIPD05-S00000.h5
8393216 RAW-R0257-AGIPD05-S00001.h5
  16896 RAW-R0257-AGIPD05-S00002.h5
8393216 RAW-R0257-AGIPD06-S00000.h5
8393216 RAW-R0257-AGIPD06-S00001.h5
  16896 RAW-R0257-AGIPD06-S00002.h5
8393216 RAW-R0257-AGIPD07-S00000.h5
8393216 RAW-R0257-AGIPD07-S00001.h5
8393216 RAW-R0257-AGIPD08-S00000.h5
8393216 RAW-R0257-AGIPD08-S00001.h5
8393216 RAW-R0257-AGIPD09-S00000.h5
8393216 RAW-R0257-AGIPD09-S00001.h5
    512 RAW-R0257-AGIPD10-S00000.h5
    512 RAW-R0257-AGIPD10-S00001.h5
    512 RAW-R0257-AGIPD10-S00002.h5
8393216 RAW-R0257-AGIPD11-S00000.h5
8393216 RAW-R0257-AGIPD11-S00001.h5
    512 RAW-R0257-AGIPD12-S00000.h5
    512 RAW-R0257-AGIPD12-S00001.h5
8393216 RAW-R0257-AGIPD13-S00000.h5
8393216 RAW-R0257-AGIPD13-S00001.h5
    512 RAW-R0257-AGIPD14-S00000.h5
    512 RAW-R0257-AGIPD14-S00001.h5
    512 RAW-R0257-AGIPD15-S00000.h5
    512 RAW-R0257-AGIPD15-S00001.h5

Files which contain only one train OPEN

E.g.: run r0257 of beamtime 201830/p900019

Missing TrainIDs OPEN

Files with this happening can also trigger problems when reading the metadata:

start_analyse.py

testing_h5read.py

% python testing_h5read.py

===== as int =====

/gpfs/exfel/exp/SPB/201830/p900019/raw/r0321/RAW-R0321-AGIPD00-S00000.h5 INDEX/trainId
trainid [       0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
 53732137 53732138 53732139 53732140 53732141 53732142 53732143 53732144
 53732145 53732146 53732147 53732148 53732149 53732150 53732151 53732152
 53732153 53732154 53732155 53732156 53732157 53732158]
Calling /home/kuhnm/agipd/calibration/test/h5read_problems/read_data.py --as_int
/gpfs/exfel/exp/SPB/201830/p900019/raw/r0321/RAW-R0321-AGIPD00-S00000.h5 INDEX/trainId
trainid [      64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
       64       64       64       64       64       64       64       64
 53732137 53732138 53732139 53732140 53732141 53732142 53732143 53732144
 53732145 53732146 53732147 53732148 53732149 53732150 53732151 53732152
 53732153 53732154 53732155 53732156 53732157 53732158]

===== not as int =====

Calling /home/kuhnm/agipd/calibration/test/h5read_problems/read_data.py
/gpfs/exfel/exp/SPB/201830/p900019/raw/r0321/RAW-R0321-AGIPD00-S00000.h5 INDEX/trainId
trainid [       0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
        0        0        0        0        0        0        0        0
 53732137 53732138 53732139 53732140 53732141 53732142 53732143 53732144
 53732145 53732146 53732147 53732148 53732149 53732150 53732151 53732152
 53732153 53732154 53732155 53732156 53732157 53732158]

TrainID starting numbers differ in run between modules OPEN

still happening, even after cables with which the wings are connected were changed (now have the same length and specs)

File structure 2017

TrainID starting numbers differ in run between modules OPEN

e.g. p900009, run 709 (pulse capacitor run)

reading in "/INSTRUMENT/SPB_DET_AGIPD1M-1/DET/{channel}CH0:xtdf/header/trainId"

channel 0, seq 0: [1532337107 1532337108 1532338073 1532338074 1532338075]
channel 8, seq 0: [1532337107 1532337108 1532338072 1532338073 1532338074]

→ the first 2 entries are not usable but afterwards channel 0 starts with ...73 but channel 8 with ...72

This is consistent per wing:

channels with shifted first train id:  [0 1 2 3 4 5 6 7]
channel 00: 1532338073
channel 01: 1532338073
channel 02: 1532338073
channel 03: 1532338073
channel 04: 1532338073
channel 05: 1532338073
channel 06: 1532338073
channel 07: 1532338073
channel 08: 1532338072
channel 09: 1532338072
channel 10: 1532338072
channel 11: 1532338072
channel 12: 1532338072
channel 13: 1532338072
channel 14: 1532338072
channel 15: 1532338072

Also happens for LiTi, run 770:

channel 00: 1541401370
channel 01: 1541401370
channel 02: 1541401370
channel 03: 1541401370
channel 04: 1541401370
channel 05: 1541401370
channel 06: 1541401370
channel 07: 1541401370
channel 08: 1541401369
channel 09: 1541401369
channel 10: 1541401369
channel 11: 1541401369
channel 12: 1541401369
channel 13: 1541401369
channel 14: 1541401369
channel 15: 1541401369

Propagates to the data and to the other trains

train 2, memory cell 8

train 3, memory cell 8

train 4, memorycell 8


PulseCount length is not stable CLEARIFIED

e.g. p900009, run 709

length of pulseCount:

Channel 00 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 01 - seq 0: 256, seq 1: 256, seq 2: 112
Channel 02 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 03 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 04 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 05 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 06 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 07 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 08 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 09 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 10 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 11 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 12 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 13 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 14 - seq 0: 256, seq 1: 256, seq 2: 128
Channel 15 - seq 0: 256, seq 1: 256, seq 2: 128

→ this also affects trainid and all other ids


Explanation

All metadata and data is stored in blocks of 16. If either the number of trains or the number of pulses is not a multiple of 16 "empty" space is filled up with zeros.

In this example the filling up took place but because one train was lost in channel 1 this now became a multiple of 16 and thus was not filled up anymore. Thus the difference in the length of pulseCount of 15.

Zeros at end of sequence CLEARIFIED

e.g. p900009, run 709 (pulse capacitor run)

seq 0
[1532337107 1532337108 1532338073
 ...
 1532338317 1532338318 1532338319 1532338320          0          0
          0          0          0          0]
seq 1
[1532338321 1532338322 1532338323
 ...
 1532338567 1532338568 1532338569 1532338570          0          0
          0          0          0          0]
seq 2
[1532338571 1532338572 1532338573
 ...
 1532338674 1532338675 1532338676 1532338677 1532338678 1532338679
 1532338680 1532338681 1532338682 1532338683]

-> the transition between two sequences is filled up with zeros although the numbering is continues.

Like this:

  • last real number in seq 0: 1532338320
  • then multiple zeros
  • first entry in seq 1: 1532338321 (which is just the one from seq 0 + 1)


Explanation

The DAQ performance has been increased by altering the HDF5 block size to 16 on the image dimension, leading to less DAQ instabilities. This however means that files will contain image multiples of 16, being 0-filled for the last images to match this criterion.

Missing train numbers: OPEN

e.g. p900009, run 709 (pulse capacitor run), channel 1:

... 1532338601 1532338602 1532338604 1532338605 ...

→ trainid 1532338603 is missing


Missing trailing zeros if missing train number CLEARIFIED

.g. p900009, run 709 has missing train in channel 1, seq 2

Channel 00: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 01: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 0}]
Channel 02: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 03: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 04: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 05: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 06: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 07: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 08: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 09: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 10: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 11: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 12: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 13: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 14: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]
Channel 15: [{'seq': 0, 'n_zeros': 6}, {'seq': 1, 'n_zeros': 6}, {'seq': 2, 'n_zeros': 15}]


channel 01, last 30 entries:: 

[1532338669 1532338670 1532338671 1532338672 1532338673 1532338674
 1532338675 1532338676 1532338677 1532338678 1532338679 1532338680
 1532338681 1532338682 1532338683          0          0          0
          0          0          0          0          0          0
          0          0          0          0          0          0]

channel 00, last 30 entries:

[1532338654 1532338655 1532338656 1532338657 1532338658 1532338659
 1532338660 1532338661 1532338662 1532338663 1532338664 1532338665
 1532338666 1532338667 1532338668 1532338669 1532338670 1532338671
 1532338672 1532338673 1532338674 1532338675 1532338676 1532338677
 1532338678 1532338679 1532338680 1532338681 1532338682 1532338683]


Explanation

All metadata and data is stored in blocks of 16. If either the number of trains or the number of pulses is not a multiple of 16 "empty" space is filled up with zeros.

In this example the filling up took place but because one train was lost in channel 1 this now became a multiple of 16 and thus was not filled up anymore. Thus the difference in the length of pulseCount of 15.

Unmatching pulseCount sum and dimension of data CLEARIFIED

PulseCount and data shape do not match (pulseCount sum vs data shape)

Channel 00, sequence 0 (18500 vs 18512)
Channel 00, sequence 1 (18500 vs 18512)
Channel 00, sequence 2 (8362 vs 8368)
Channel 01, sequence 0 (18500 vs 18512)
Channel 01, sequence 1 (18500 vs 18512)
Channel 02, sequence 0 (18500 vs 18512)
Channel 02, sequence 1 (18500 vs 18512)

...

Explanation

Data is also filled up with zeros if the dimension is not a multiple of 16.

The following observation happen after fill up was considered

Train shifting OPEN

train shift by 16 train

run 819 (dark):

channel 00: 1542237342
channel 01: 1542237342
channel 02: 1542237342
channel 03: 1542237342
channel 04: 1542237342
channel 05: 1542237342
channel 06: 1542237342
channel 07: 1542237342
channel 08: 1542237358
channel 09: 1542237358
channel 10: 1542237358
channel 11: 1542237358
channel 12: 1542237358
channel 13: 1542237358
channel 14: 1542237358
channel 15: 1542237358

The number of total trains is the same for all modules. Channel 08-15 just take 16 trains more at the end (last trainID channel 0: 1542238339 vs channel 08: 1542238355)

train shifted by 5 trains

run 490 (pcdrs):

channel 00: 1497185314
channel 01: 1497185314
channel 02: 1497185314
channel 03: 1497185314
channel 04: 1497185314
channel 05: 1497185314
channel 06: 1497185314
channel 07: 1497185314
channel 08: 1497185309
channel 09: 1497185309
channel 10: 1497185309
channel 11: 1497185309
channel 12: 1497185309
channel 13: 1497185309
channel 14: 1497185309
channel 15: 1497185309


Unmatching number of trains between wing 1 and wing 2 OPEN

total number of trains differ between wing 1 and 2 (train loss at the end or something wrong with the signal?)

run 626:

There are no problems with this run (no train loss, no train shifting at the beginning, ...) except that not all modules have the same number of trains for seq 6

Channel 00: 36 (trainid: 1508879646 ... 1508879681)
Channel 01: 36 (trainid: 1508879646 ... 1508879681)
Channel 02: 36 (trainid: 1508879646 ... 1508879681)
Channel 03: 36 (trainid: 1508879646 ... 1508879681)
Channel 04: 36 (trainid: 1508879646 ... 1508879681)
Channel 05: 36 (trainid: 1508879646 ... 1508879681)
Channel 06: 36 (trainid: 1508879646 ... 1508879681)
Channel 07: 36 (trainid: 1508879646 ... 1508879681)
Channel 08: 35 (trainid: 1508879646 ... 1508879680)
Channel 09: 35 (trainid: 1508879646 ... 1508879680)
Channel 10: 35 (trainid: 1508879646 ... 1508879680)
Channel 11: 35 (trainid: 1508879646 ... 1508879680)
Channel 12: 35 (trainid: 1508879646 ... 1508879680)
Channel 13: 35 (trainid: 1508879646 ... 1508879680)
Channel 14: 35 (trainid: 1508879646 ... 1508879680)
Channel 15: 35 (trainid: 1508879646 ... 1508879680)


trainId:

Channel 7, seq 6:

[1508879646 1508879647 1508879648 1508879649 1508879650 1508879651
1508879652 1508879653 1508879654 1508879655 1508879656 1508879657
1508879658 1508879659 1508879660 1508879661 1508879662 1508879663
1508879664 1508879665 1508879666 1508879667 1508879668 1508879669
1508879670 1508879671 1508879672 1508879673 1508879674 1508879675
1508879676 1508879677 1508879678 1508879679 1508879680 1508879681]

Channel 8, seq 6:

[1508879646 1508879647 1508879648 1508879649 1508879650 1508879651
1508879652 1508879653 1508879654 1508879655 1508879656 1508879657
1508879658 1508879659 1508879660 1508879661 1508879662 1508879663
1508879664 1508879665 1508879666 1508879667 1508879668 1508879669
1508879670 1508879671 1508879672 1508879673 1508879674 1508879675
1508879676 1508879677 1508879678 1508879679 1508879680]

Data shape:

Channel 7, seq 6: 2880 x 2 x 512 x 128

Channel 8, seq 6: 2800 x 2 x 512 x 128

File size differs as well:

% ls -lh *6.h5
-r--r----- 1 xdata xdata 722M Oct 13 12:51 RAW-R0626-AGIPD00-S00006.h5
-r--r----- 1 xdata xdata 722M Oct 13 12:51 RAW-R0626-AGIPD01-S00006.h5
-r--r----- 1 xdata xdata 722M Oct 13 12:51 RAW-R0626-AGIPD02-S00006.h5
-r--r----- 1 xdata xdata 722M Oct 13 12:51 RAW-R0626-AGIPD03-S00006.h5
-r--r----- 1 xdata xdata 722M Oct 13 12:51 RAW-R0626-AGIPD04-S00006.h5
-r--r----- 1 xdata xdata 722M Oct 13 12:51 RAW-R0626-AGIPD05-S00006.h5
-r--r----- 1 xdata xdata 722M Oct 13 12:51 RAW-R0626-AGIPD06-S00006.h5
-r--r----- 1 xdata xdata 722M Oct 13 12:51 RAW-R0626-AGIPD07-S00006.h5
-r--r----- 1 xdata xdata 702M Oct 13 12:51 RAW-R0626-AGIPD08-S00006.h5
-r--r----- 1 xdata xdata 702M Oct 13 12:51 RAW-R0626-AGIPD09-S00006.h5
-r--r----- 1 xdata xdata 702M Oct 13 12:51 RAW-R0626-AGIPD10-S00006.h5
-r--r----- 1 xdata xdata 702M Oct 13 12:51 RAW-R0626-AGIPD11-S00006.h5
-r--r----- 1 xdata xdata 702M Oct 13 12:51 RAW-R0626-AGIPD12-S00006.h5
-r--r----- 1 xdata xdata 702M Oct 13 12:51 RAW-R0626-AGIPD13-S00006.h5
-r--r----- 1 xdata xdata 702M Oct 13 12:51 RAW-R0626-AGIPD14-S00006.h5
-r--r----- 1 xdata xdata 702M Oct 13 12:51 RAW-R0626-AGIPD15-S00006.h5

Massive train loss in run 627 CLEARIFIED

120 pulses

1264 trains lost per module, 1267 trains collected in data -> total trains requested 2531 (?)

Channel 00: 1264
Channel 01: 1264
Channel 02: 1264
Channel 03: 1264
Channel 04: 1264
Channel 05: 1264
Channel 06: 1264
Channel 07: 1264
Channel 08: 1264
Channel 09: 1264
Channel 10: 1264
Channel 11: 1264
Channel 12: 1264
Channel 13: 1264
Channel 14: 1264
Channel 15: 1264
Number of total trains lost: 20224

Looking at the trainID it seems that every other train is missing:

seq 0: Train loss found at indices:
idx 0: ... 1508880311 1508880313 1508880315 ...
idx 1: ... 1508880311 1508880313 1508880315 1508880317 ...
idx 2: ... 1508880313 1508880315 1508880317 1508880319 ...
seq 1: Train loss found at indices:
idx 0: ... 1508880807 1508880809 1508880811 ...
idx 1: ... 1508880807 1508880809 1508880811 1508880813 ...
idx 2: ... 1508880809 1508880811 1508880813 1508880815 ...
seq 2: Train loss found at indices:
idx 0: ... 1508881307 1508881309 1508881311 ...
idx 1: ... 1508881307 1508881309 1508881311 1508881313 ...
idx 2: ... 1508881309 1508881311 1508881313 1508881315 ...
seq 3: Train loss found at indices:
idx 0: ... 1508881807 1508881809 1508881811 ...
idx 1: ... 1508881807 1508881809 1508881811 1508881813 ...
idx 2: ... 1508881809 1508881811 1508881813 1508881815 ...
seq 4: Train loss found at indices:
idx 0: ... 1508882307 1508882309 1508882311 ...
idx 1: ... 1508882307 1508882309 1508882311 1508882313 ...
idx 2: ... 1508882309 1508882311 1508882313 1508882315 ...
seq 5: Train loss found at indices:
idx 0: ... 1508882807 1508882809 1508882811 ...
idx 1: ... 1508882807 1508882809 1508882811 1508882813 ...
idx 2: ... 1508882809 1508882811 1508882813 1508882815 ...

The output is the same for all channels

Explanation

Currently the readout of the detector firmware is not fast enough to handle this number of pulses (will be fixed in the future)

Train loss coupled to pattern? CLEARIFIED

runs taken with 176 pulses loose 332 trains for each module

dark runs

671:

Channel 00: 332
Channel 01: 332
Channel 02: 332
Channel 03: 332
Channel 04: 332
Channel 05: 332
Channel 06: 332
Channel 07: 332
Channel 08: 332
Channel 09: 332
Channel 10: 332
Channel 11: 332
Channel 12: 332
Channel 13: 332
Channel 14: 332
Channel 15: 332
Total number of trains lost: 5312

indices (for all channels the same):

seq 0: Train loss found at indices:
idx 1: ... 1529290169 1529290170 1529290172 1529290173 ...
idx 4: ... 1529290173 1529290174 1529290176 1529290177 ...
idx 7: ... 1529290177 1529290178 1529290180 1529290181 ...
seq 1: Train loss found at indices:
idx 2: ... 1529290501 1529290502 1529290504 1529290505 ...
idx 5: ... 1529290505 1529290506 1529290508 1529290509 ...
idx 8: ... 1529290509 1529290510 1529290512 1529290513 ...
seq 2: Train loss found at indices:
idx 1: ... 1529290833 1529290834 1529290836 1529290837 ...
idx 4: ... 1529290837 1529290838 1529290840 1529290841 ...
idx 7: ... 1529290841 1529290842 1529290844 1529290845 ...
seq 3: Train loss found at indices:
idx 0: ... 1529291166 1529291168 1529291169 ...
idx 3: ... 1529291169 1529291170 1529291172 1529291173 ...
idx 6: ... 1529291173 1529291174 1529291176 1529291177 ...

672:

Channel 00: 332
Channel 01: 332
Channel 02: 332
Channel 03: 332
Channel 04: 332
Channel 05: 332
Channel 06: 332
Channel 07: 332
Channel 08: 332
Channel 09: 332
Channel 10: 332
Channel 11: 332
Channel 12: 332
Channel 13: 332
Channel 14: 332
Channel 15: 332
Total number of trains lost: 5312

indices (for all channels the same):

seq 0: Train loss found at indices:
idx 1: ... 1529310114 1529310115 1529310117 1529310118 ...
idx 4: ... 1529310118 1529310119 1529310121 1529310122 ...
idx 7: ... 1529310122 1529310123 1529310125 1529310126 ...
seq 1: Train loss found at indices:
idx 2: ... 1529310446 1529310447 1529310449 1529310450 ...
idx 5: ... 1529310450 1529310451 1529310453 1529310454 ...
idx 8: ... 1529310454 1529310455 1529310457 1529310458 ...
seq 2: Train loss found at indices:
idx 1: ... 1529310778 1529310779 1529310781 1529310782 ...
idx 4: ... 1529310782 1529310783 1529310785 1529310786 ...
idx 7: ... 1529310786 1529310787 1529310789 1529310790 ...
seq 3: Train loss found at indices:
idx 0: ... 1529311111 1529311113 1529311114 ...
idx 3: ... 1529311114 1529311115 1529311117 1529311118 ...
idx 6: ... 1529311118 1529311119 1529311121 1529311122 ...

xray runs

673:

Channel 00: 332
Channel 01: 332
Channel 02: 332
Channel 03: 332
Channel 04: 332
Channel 05: 332
Channel 06: 332
Channel 07: 332
Channel 08: 332
Channel 09: 332
Channel 10: 332
Channel 11: 332
Channel 12: 332
Channel 13: 332
Channel 14: 332
Channel 15: 332
Total number of trains lost: 5312

indices (for all channels the same):

seq 0: Train loss found at indices:
idx 1: ... 1529374569 1529374570 1529374572 1529374573 ...
idx 4: ... 1529374573 1529374574 1529374576 1529374577 ...
idx 7: ... 1529374577 1529374578 1529374580 1529374581 ...
seq 1: Train loss found at indices:
idx 2: ... 1529374901 1529374902 1529374904 1529374905 ...
idx 5: ... 1529374905 1529374906 1529374908 1529374909 ...
idx 8: ... 1529374909 1529374910 1529374912 1529374913 ...
seq 2: Train loss found at indices:
idx 1: ... 1529375233 1529375234 1529375236 1529375237 ...
idx 4: ... 1529375237 1529375238 1529375240 1529375241 ...
idx 7: ... 1529375241 1529375242 1529375244 1529375245 ...
seq 3: Train loss found at indices:
idx 0: ... 1529375566 1529375568 1529375569 ...
idx 3: ... 1529375569 1529375570 1529375572 1529375573 ...
idx 6: ... 1529375573 1529375574 1529375576 1529375577 ...

674:

Channel 00: 332
Channel 01: 332
Channel 02: 332
Channel 03: 332
Channel 04: 332
Channel 05: 332
Channel 06: 332
Channel 07: 332
Channel 08: 332
Channel 09: 332
Channel 10: 332
Channel 11: 332
Channel 12: 332
Channel 13: 332
Channel 14: 332
Channel 15: 332
Total number of trains lost: 5312

indices (for all channels the same):

seq 0: Train loss found at indices:
idx 1: ... 1529382939 1529382940 1529382942 1529382943 ...
idx 4: ... 1529382943 1529382944 1529382946 1529382947 ...
idx 7: ... 1529382947 1529382948 1529382950 1529382951 ...
seq 1: Train loss found at indices:
idx 2: ... 1529383271 1529383272 1529383274 1529383275 ...
idx 5: ... 1529383275 1529383276 1529383278 1529383279 ...
idx 8: ... 1529383279 1529383280 1529383282 1529383283 ...
seq 2: Train loss found at indices:
idx 1: ... 1529383603 1529383604 1529383606 1529383607 ...
idx 4: ... 1529383607 1529383608 1529383610 1529383611 ...
idx 7: ... 1529383611 1529383612 1529383614 1529383615 ...
seq 3: Train loss found at indices:
idx 0: ... 1529383936 1529383938 1529383939 ...
idx 3: ... 1529383939 1529383940 1529383942 1529383943 ...
idx 6: ... 1529383943 1529383944 1529383946 1529383947 ...

Explanation

Currently the readout of the detector firmware is not fast enough to handle this number of pulses (will be fixed in the future)


  • No labels