General Concept

A large number of parameters which are essential for ALL Experiments is continuously saved (pulse energy, arrival time, Beamline settings ...) in the PHOTONDIAGNOSTIC FLASH DAQ system.
Other experiment related information is ONLY saved by the users ON DEMAND in the FLASH DAQ system. To record this type of data we have 3 User DAQs at FLASH1 (FLASH1_USER1,FLASH1_USER2 and FLASH1_USER3) and one at FLASH 2 (FLASH2_USER1)
How to save this experiment related "User-Data" will be explained briefly in this page:


Taking User data: Starting and stopping runs (Users)

In order to save the experiment related data from ADCs, Images, delayline detectors etc. the User DAQ has to be started to record the experiment data. The options which data can be saved, have to be discussed way before the beamtime. For regular beamtime operation the DAQ also was setup by experts before the beamtime. If things do not look like described here look in the troubleshooting section below.


DAQ control panel

To get to the User DAQ control one has to go to Experiment control -> User DAQ tab -> FLxUSERy DAQ CTRL
jddd_DAQ-control-manual-how_to_get.png

The following window opens. Here the User DAQ can be started and stopped as well as the incoming data can be visualized, information about data rates and saved properties is presented as well as the panel is automatically printed in to the logbook:

jddd_DAQ-control-manual2.png




Checklist for proper saving of the data

If not all points of the list are o.k. there may be some problem with the writing of the data. Look to the troubleshooting section and / or talk to beamline stuff.


More infos on the tools to check the proper saving of the data


DAQ monitor

The DAQ monitor shows a predefined selection of parameters as they are received in the DAQ before writing to a file. Here one can check if the desired data arrives in the DAQ and if the ranges are set correctly (e.g. sufficient acquisition time). The parameters to look at are defined before the beamtime with the local contact / DAQ group (details see below)


Offline DAQ monitor

This tool looks if a predefined selection of parameters is saved in the raw file. When a raw.-file reached its maximum size ( ~ 1 GB) it is closed and a new file is opened. Once closed, the Offline DAQ monitor reads the file and checks if there are entris for the selected parameters. It shows the fraction of events containing data. For “fast” data (data written with 10 Hz and bunch ID synchronous like images, ADC traces) this should be close to 100%. For “slow” data (saved about every second e.g. pressures , temperatures...) this is about 10-20%. However it is to note that this program just check if there is anything saved. It does not check if it is the proper data.


DAQ Data GUI

To ultimately check if the correct data is saved one can use the DAQ data GUI to look at the saved raw. files.

DAQgui1.jpg DAQgui2.jpg daqgui3.png

The DAQ Data GUI is a general tool to select and visualise data that was saved in the DAQ . It has in addition simple analysis options like histogram, mean, min or max values as function of time etc. Also correlations between different parameters can be analyzed in the tool. More details can be found How to Use the FLASH DAQ Data GUI? and here (DAQdataGUI link collection)

The tool (separate Java program) can be started DAQdataGUI or in the DAQ control panel ( lower right) on your local (Desy) PC. In some cases the access via Windows does not work. An option which should always work is to log onto flashlxuser1 or flashlxuser2 and start DAQdataGUI from the command line.



To access the desired data one has to:


How to get the data out for analysis (short summary)

The DAQdataGui is a powerful tool to visualize the saved data and to do very preliminary analysis. however for more detailed analysis the data has to be read into analysis programs ( matlab, python, Origin ...) here are different options one can use (what option is the best and how to set it up has to be discussed before hand with the FLASH DAQ experts ...)


The HDF5 files (online and summary) are saved in the "gpfs" system . it can be accessed from Windows and Linux computers at DESY by the persons ( logins) which are registered for the beamtime in DOOR ( functional accounts

The path to your data is structured the following way:

More details and links can be found in the DAQ and controls overview.




Setting up the DAQ - DAQ Run Control GUI (local contacts, FLASH stuff)

This section is intended for the local contacts / FLASH staff to set up the DAQ according to the needs of users before the beamtime. For troubleshooting this may also be helpful for users ...


Configuring the DAQ


RCGUI_2.jpg

Using the jddd DAQ control panel one can start and stop runs but in order to configure the DAQ which data to save one has to use a a separate *DAQ run control * java application. Since it needs write access to DOOCS internal file systems, which can not be made available elsewhere one has to start the system on our DOOCS control computers flashlxuser1.desy.de or flashlxuser2.desy.de.
Log in with the beamline account (bl1user, bl2user, ..., fl24user...) either via X-Win32 from windows or via ssh from Linux machines.
Once logged in to flashlxuser1 or flashlxuser2 the DAQ run control can be started. Depending on the DAQ you want to use the commands are:


Unix commandDAQ
RCGUIFL1USER1FLASH1_USER1 DAQ
RCGUIFL1USER2FLASH1_USER2 DAQ
RCGUIFL1USER3FLASH1_USER3 DAQ
RCGUIFL2USER1FLASH2_USER1 DAQ

Which DAQ to use is decided by the FLASH DAQ team according to the beamtime schedule.


RCGUI_properties.jpg


Using the Add and Rem(ove) buttons one can move the desired "subsystems" in the included side - meaning that they are saved. The "subsystems" contain typically several DOOCS parameters. E.g. EXPERIMENT_MHZ_ADC_BL1 contains all relevant DOOCS properties of the 4 MHZ ADC channels available at BL1. To get a detailled information about the saved parameters one can have a look to the "Show Properties in Subsystems" button . In addition, a list of all possible subsystems and their description can be found here.


Configuring the DAQ monitor

The DAQ monitor ( and the Offline monitor) can be configured using DAQdataGUI.
Load a test run containing all relevant properties. Select all properties you want to monitor. Use Tools -> Channels to DAQMonitor and choose your DAQ. This sends a list with the selected properties to the DAQ monitor of your system.
daq_monitor_config.png



Start and Stop a run

In order to "activate" and test your configuration you have to start a run with the new configuration. NOTE: Whenever the configuration was changed with the RCGUI the new run HAS to be started with the RCGUI (starting with the jddd DAQ panel would only start the last configuration that has been running before !!!


To START a run - meaning starting to save the experiment related data - one has to press the "start" button in the Run Control GUI. This opens a confirmation window asking if you really want to start the run ("in default mode"). Press yes. Now it takes about 20 seconds for the system to start up and start recording data. Indication is the "Run Control: RUN" and "DAQ: RUN" greenish indicators on the upper left side of the run control.

To STOP a run one has to press the "stop" button. This opens a window for comments. This contains already the actual Experiment name by default. Please leave this information in. Add a comment for your run. This comment will be printed into the logbook and can later on be used to identify individual runs ... note you are required to type in something otherwise the run will not stop.

The stopping also needs about 20 seconds.*_IT IS IMPORTANT_ not to start and stop the DAQ while it is still ramping up or down. So please wait before restarting the DAQ.*

After the run use the DAQ data GUI to check if the data intended is saved properly.


Load/Save the configuration


whitelist.jpg

Once a list of subsystems is defined and tested this parameter combination should then be saved in a so-called white list. By loading this white list one can then easily configure the DAQ after changes / restarts ... to the initially defined settings to save the correct data.
The whitelist can be saved/loaded with the Run Control GUI by: File -> Group White List -> Save/Load. Note, while the top left rectangle's background of time and date is yellow, don't try any further controls of the GUI, you might confuse the program. It is busy reading the configuration from the run control database. The background changes back to grey when it has finished.

The filename convention for the White Lists are arranged in the way:
DAQ name - Date of creation - name of the PI / Experiment - optional comment on the type of data

The whitelists are stored in separate folders for each beamline.


Choosing the file size

The DAQ writes into a file (in .raw format) up to a configurable value. If the limit is reached the file will be closed and a new file will be started. Thus a run can contain tens to hundreds of files.

The size can be configured to be 100 MB, 350 MB or 1000 MB (default)

Few things to consider (in case of doubt talk to the expert (Erland ...)):

To choose the desired filesize one has to choose a run mode configuration in the RunControlGUI

This will however the clear the definition of selected subsystems and one has to reload the whitelist of the experiment again.





Assigning the DAQ to the beamline in jddd

To assign which DAQ is used at what beamline one has to use the dropdown menu in the parameter monitor. In each Beamline overview tab (BL1,2,3) there is a button (upper right) that starts the parameter monitor panel. In the upper middle is a DAQ field in which the appropriate DAQ can be assigned to the Beamline.


Setting the logbook to print in / putting the Whitelist name in jddd

The jddd DAQ Control should be set up that it prints by default the relevant run information in the logbook associated to your beamline Logbooks

Define logbook / and whitelist for display in the jddd DAQ control (jddd otherwise does not "know" which White list was used ... this has to be put in by hand ...up to now)
setting_parameters.jpg





In case something goes wrong: Troubleshooting


No START or STOP buttons are visible in the DAQ control panel

* The jddd GUI only shows the Start and stop bunnon if the DAQ is in the "Idle" state. sometimes the DAQ "just" gets stuck in the stopping phase. Then stopping again or starting form the not "Idle" state can help. For this goto the "Advanced" tab of the jddd DAQ control . there the START and STOP buttons are always available ... independent of the state of the DAQ. If this does not help you may have to restart the DAQ ...


The DAQ does not start a run




Data seems not properly saved


RC_logfile.jpg When checking the saved run with the DAQdataGUI it may happen that the property shows up in the data tree but no events are saved. Then (most likely) something is wrong with the sending of the data.




Restarting the DAQ





Here in more detail:





UserDAQ_status.jpg










Some more details for experts ...

Re-read the data base

If there are new Properties included by Vladimir et al one has to reread the data base.

This can be done easiest by going to the run modes and choose a different "stored setting" (changing e.g. from filesize 100 MB to 350 MB) and going back to subsystems. At this point the new database entries are read in. then one can do the same backwards to switch back to the initial file size. Doing this the configured list of subsystems is lost and one has to reload the whitelist and start a run to get the loaded whitelist in the configuration.


OR one can do the hard way:

On flashlxuser1 or 2 start RCGUIFL1USER1 or appropriate to the DAQ you want to reconfigure. Select the tab "Run Modes". To then reload or re-init the run control database you need the expert menu entry:

Options -> "Show Expert Menu"

Then you go to "Expert" -> "Run Control Commands" -> "Remove SHM" (occasionally twice).

After you see a message in the RCGUI messages window that the SHM was removed you do:

"expert" -> "Run Control Commands" -> "Re-init SHM"

Now close the RCGUI and start it again. Switch to the tab "subsystems", wait for the subsystems to be reloaded and load the whitelist you want to use.


Tracking DAQ restarts

A system was implemented (March 2020) where each restart of a user DAQ is recorded in a log file

One can see all log files in :

/home/doocsadm/bm/rcdb/restartslog/

data base errors

The red flags next to the  different stream indicators tell something about data base errors

upper flag:  There was a problem to read run catalogue

lower flag: There was a problem to read dccp catalogue.

In both cases there are counters of failures to read a catalogue. If this counter is not incrementing with every attempt to read a catalogue, then it's no problem. It can be monitored and reset in the panel that one gets by pressing on "green" area.