This troubleshooting guide aims to provide basic REFM-OPT recovery instructions.
Open the REFM-OPT overview panel (MainTaskbar → Diagnostics → Laser-Based Synchronization → REFM-OPT Overview)
Observe the panel for a few seconds to check whether any station is frozen. Check for any red status flags. Unfold the plot on the bottom. Check for jumps of the phase shifters.
In case of problems, please inform email@example.com and print information to the elogbook.
If a station is offline in the overview panel:
Make sure the REFM-OPT server is running. REFM-OPT servers are installed on the LLRF crate with the same name (e.g. xfelcpulla2m).
Start troubleshooting the topmost (upstream) broken station first. Then continue with the next lower (downstream) station.
Always remember: The protection mode is supposed to disable the phase feedback and freeze the REFM-OPT phase shifter in the last known working point. This state is safe for XFEL operation. However the phase feedback is disabled in this state and should be restored as soon as possible.
Check the overview sub-panel of a REFM-OPT:
The protection recovery will reset some protection modules. The modules are re-enabled at the end of the routine. The maximum allowed phase jump during recovery is defined in operation → protection recovery → acceptable phase deviation. If the routine fails, the phase shifter is restored to its old value.
This routine provides steps to restore basic operation without active feedback. It should be used when no or insufficient RF output power is provided, when a phase jump has occurred or when feedback recovery failed. The procedure is meant for general disaster recovery.
First recover RF output power. Daisy chained REFM-OPTs can be cross-checked with the next station in line (a block diagram is on the REFM-OPT overview panel). Check the RF input power on these stations. Be aware that there might be additional RF components like REFMs between two REFM-OPTs.
Second recover the correct RF phase. If the REFM-OPT is in protection mode you must first leave this mode in order to gain manual control:
To recover the correct RF phase you should first go to the overview panel.There is a fixed phase offset between both RF channels. This offset needs to be taken into account if the REFM-OPT RF amplifiers where switched recently. You can click onto the RF switches to check their history. Also check the phase shifter history to see if there where any jumps and to find your reference value.
If the server is running but not updating, check the x2timer status first. The trigger source on the system panel must be set to external (x2timer). The macropulse # below should update regularly. The x2timer panel is linked for further analysis under system → x2timer.
The logfile is accessible under system → logreader. Use it to gain further information.
The Ethernet connection status is monitored under system → TMCB. The timeout is defined by the operating system of the server host and in the order of 15 minutes. Try to ping the TMCB (e.g. "
ping xfelrfoptcba2m", adapt the hostname). If its reachable you can try to recover the connection using the resume button.
If you think that just the server is malfunctioning but the REFM-OPT is still in operation you can leave it in this state for expert recovery. You can for instance check with the next REFM-OPT in line that the RF power is still OK and no phase jump occurred.
If you absolutely need access to the REFM-OPT, for example to adjust a wrongly set phase shifter and you think the FPGA crashed such that the server can not connect to it anymore you can power cycle the TMCB as a last resort. The procedure is laid out under FRED operation. This is emergency only and will cause a phase jump.
The MZM temperature is or has been outside its predefined temperature window if the MZM temperature protection module is triggered. In this case
The FRED server allows to monitor the FRED status but by intention offers only rudimentary control over the FRED. The FRED Panel is linked under modules → FRED. You can switch power on and power off for the complete REFM-OPT at once. You can also power cycle only the TMCB by switching to on, TMCB off and then back to power on. All this will cause a phase jump as the DAC for the phase shifter will also be cycled. Don't switch anything off unless you absolutely have to.
For more advanced control you need to directly connect to the FRED via telnet:
telnet xfelrfoptfra2m 10001", adapt the hostname).
help" to see a list of supported commands. For example check the FRED state ("
TMCBHardReset"). Use with caution, this will cause a phase jump.
ctrl + ]"), press connect on the FRED panel to reconnect the server.