On Thurday June 20th of 2019 at ~08.09am a power glitch affected several components of the P10 beamline. These were:
Frontend & Optics Hutch:
- All valves of the vauum interlock closed (easily reopened via vil web page). All vacuum pumps and pressures were fine.
- The monohromator Bragg axis lost power (i.e. it lost its position and fell into its brake) and the pmac showed a line fault. The encoder system was not effected, so that the monochromator could be easily brought online again with the help of FS-BT (J. Heuer). Deactivating the brake and a short move was all that was required to bring the system back into the operating state. Finally, the monochromator energy could be restored by moving 'fmbenergy' back to its originally posistion.
- We discovered also, that several ZMX axis on the lower crate were turned off (first 8 or so), we switched these back on.
Experimental hutch 1:
- The first 5 ZMX crates lost power (i.e all activated axis between 1-80 turned off. Critically, was of course the loss of power of the first ZMX crate which operates the stages of the first Q-SYS optical table. This table dropped into its brakes and we lost the positions of 'ot1z1', 'ot1z2', 'ot1z3' & the virtual motor 'ot1z'. We leveled the table by moving 'ot1z1' up by ~0.35mm (this end always drops faster). On top of this, the levelled table had to be lifted up by an additional ~0.6mm on all 3 axis. We used the 'udio' and the unchanged position of 'uslt2' to restore 'ot1' to its previous position.
- Again, the second ZMX crate had trouble to start. The lights of the limit switches of all 16 stages or the stages 1-4 & 9-12 were turning on and off for a 'nice' light show. Finally, after several tries to restart the crate, a fast turn off /on switch recovered the crate to its normal state.
- The power output of the HAMEG power supplies (EH1 and EH2) went off (i.e. all our vacuum valves closed). Switching the output back on worked.
Experimental hutch 2:
- The first ZMX crates lost power (i.e all activated axis between 1-16 turned off. Again, this first ZMX crate operates the stages of the 2nd Q-SYS optical table. Also, this table dropped into its brakes and we lost the positions of 'ot2z1', 'ot2z2', 'ot2z3' & the virtual motor 'ot2z'. We leveled the table by moving 'ot2z1' up by ~0.20mm. On top of this, the ot2 table had to be lifted up by an additional ~0.6mm on all 3 axis. We used the beam through the unchanged position of 'uslt2' to restore 'ot2' to its previous position. Nevertheless a fine correction of the lenses was needed.
- The hase4mtrans pc went down and had to be restarted.
- The GINIX system seemed unaffected.
- The EIGER and PILATUS seemed unaffected.
Total loss of beam time: ~4 hours