From c795012435c96ebbcb90fe8c7d82afb909054632 Mon Sep 17 00:00:00 2001 From: hadaq Date: Sat, 30 Mar 2019 17:42:25 +0100 Subject: [PATCH] updated Operator Guide --- hadesoperator/calibration.tex | 4 +-- hadesoperator/daqcontrol.tex | 41 ++++++++++++++++++++----------- hadesoperator/expertknowledge.tex | 12 +++------ hadesoperator/main.tex | 3 +-- hadesoperator/powercycles.tex | 14 ++++++----- 5 files changed, 40 insertions(+), 34 deletions(-) diff --git a/hadesoperator/calibration.tex b/hadesoperator/calibration.tex index 08f5279..8aa0473 100644 --- a/hadesoperator/calibration.tex +++ b/hadesoperator/calibration.tex @@ -14,8 +14,8 @@ There is an icon in the Tactical Overview checking the status of calibration nea To do a calibration, use the "TDC Calibration" buttons in the DAQ control window. First, make sure that all boards are running and no front-ends (especially from RICH, ECAL, Start) are missing. If some are missing, follow the usual procedures to recover them. -After clicking "Start TDC Cal", the Tactical Overview will show the (approximate) progress of the process in the lowest row. Once it is completed, use "Stop TDC Cal" to return to the normal data taking mode. +After clicking "Start TDC Cal", the Tactical Overview will show the (approximate) progress of the process in the lowest row. Once it is completed, use "Stop TDC Cal" to return to the normal data taking mode. Be patient now, the EB need some time to actually finish the process. Make sure that the Tactical Overview now shows the new time of the last calibration. If there was a DAQ failure (either restart or failing boards\footnote{Only from ECAL, RICH and Start. MDC is not affected by this run} while the calibration took place, the process has to be repeated. -If the run takes longer than 5 minutes, there might be problems with noise or other data from the detectors. Try switching off beam and/or setting thresholds higher if nothing else helps. +If the run takes longer than 5 minutes, there might be problems with noise or other data from the detectors. Try switching off beam and/or setting thresholds higher if nothing else helps. The EB monitor can give some hints on where the problem is located. diff --git a/hadesoperator/daqcontrol.tex b/hadesoperator/daqcontrol.tex index d428a8a..2e65fda 100644 --- a/hadesoperator/daqcontrol.tex +++ b/hadesoperator/daqcontrol.tex @@ -14,13 +14,20 @@ Most control tasks can be done using the DAQ Control GUI, only few things need a \begin{description} \item[Start DAQ] The main script to restart the DAQ. Standard solution if some problems appear (red blinking). \item[Start Stop Trigger] Interrupt sending triggers - \item[Reprogram FPGA] Reload designs to old TRB2 based boards + \item[Reprogram FPGA] Reload designs to old TRB2 based boards (RPC, TOF, FW, CTS only) \item[Reboot OEP] Reload designs of MDC front-ends (e.g. to be used if error "oeps too old" appears - let it follow a DAQ-restart!) \item[Reboot MDC Hub] Reload designs for MDC hub boards \item[ReSync Missing OEP] Tries to get a failed MDC front-end running again - \item[Reboot TRB3] The equivalent of 'Reprogram FPGA' for new subsystems. + \item[Resync DiRich] Tries to recover failing RICH front-ends. Might need a DAQ restart afterwards + \item[Reboot CentralHubs] Reloads the firmware for all central hub boards (1) + \item[Reboot DiRich] Reloads the firmware for RICH front-ends (1) + \item[Reboot Ecal] Reloads the firmware for all ECal boards (1) + \item[Reboot RICH Combiner] Reloads the firmware for the first layer of RICH network hubs (1) + \item[Reboot RICH hubs] Reloads the firmware for the second layer of RICH network hubs (1) \end{description} +The tools marked with (1) can only be used succesfully, if all boards are responding and are not reported as missing. If there are missing boards, first try a DAQ restart and then the reboot. + \subsection{Monitor Tools} \begin{description} \item[CTS Mon] The monitor for CTS settings and scalers. One instance should be open for experts, but usually not needed by the operator @@ -30,6 +37,7 @@ Most control tasks can be done using the DAQ Control GUI, only few things need a \item[DAQ Rawmon] Opens a browser pointed at \url{https://hades33/rawmon} for low level data analysis. Expert tool \item[Daq VNC] This VNC is typically used by experts to remotely control the system. Not used for operating the system from the control room \item[Chat] A terminal based chat window, not used during beam time + \item[Hmon Snapshot] Creates a copy of all plots and windows in Hmon for documentation \item[mumble] Conferencing tool, an instance should always be running on the left DAQ computer \end{description} @@ -55,24 +63,24 @@ Most control tasks can be done using the DAQ Control GUI, only few things need a \end{description} \subsection{Expert Tools} -{\it This section is visible only when the ''Show Expert Tools'' box on the upper left corner is checked. -It contains actions which are not necessary for regular DAQ operation.} +{\it This section is visible only when the ''Show Expert Tools'' box on the upper left corner is checked.} \begin{description} \item[MDC HighVoltageRamp] - \item[BLR Reinit FPGA] + \item[BLR Reinit FPGA] \item[BLR Reload Settings] \item[CTS Scope] - \item[MDC LV Turn On All Relais] - \item[Relaod RICH] + \item[MDC LV Turn On All Relais] Switches on power to all MDC front-ends. Use if a larger amount of boards is missing after a power cycle + \item[Reload RICH] \item[Reload RPC Thresholds] \item[Restart BNET Control] Restart the DABC master process on lxhadeb07 that controls the building network (BNET). This process also delivers the eventbuilder monitoring at \url{https://hades33/eb} \item[Restart Hmon] use to activate changes in main monitoring \item[Restart Rawdata Mon] Restart the DABC analysis of the raw data contents. Use this button if \url{https://hades33/rawmon} is not working - \item[Restart Sequencer] + \item[Restart Sequencer] Restart automatic HV control procedures in EPICS + \item[Set ECAL Thr] Sets thresholds for all ECAl front-ends \item[Set RICH HV] Switches RICH HV on if all channels are off, otherwise switches RICH HV off - \item[Set RICH Thr] Sets thresholds in RICH to default values from file + \item[Set RICH Thr] Sets thresholds in RICH to default values from file (can be done while DAQ is running) \item[Start Speech Output] \item[Stop Speech Output] \end{description} @@ -83,20 +91,23 @@ This section offers buttons to cycle the low voltage supplies of several detecto shortcuts to avoid operations on the full detector control system GUI. A power cycle may be necessary in some situations to reset the DAQ front-end boards. {\bf Note that after a power cycle it is mostly mandatory to perform -a ''Reprogram FPGA'', ''Reboot OEP'' and ''Reboot MDCHub'' by +a ''Reprogram FPGA'', ''Reboot OEP'' and/or ''Reboot MDCHub'' by clicking these buttons in the DAQ-Operator section!} \begin{description} \item[Powercycle all] LV power cycle of everything. Requires ''Reprogram FPGA'', ''Reboot OEP'', and ''Reboot MDCHub'' afterwards - \item[RICH Powercycle] LV power cycle of RICH. + \item[RICH Powercycle] LV power cycle of RICH. Requires a ''Set RICH Thr'' afterwards. \item[RICH Fan Powercycle] LV power cycle of RICH Fans. - \item[ECAL LV5 Powercycle] LV power cycle of ECAL. - \item[Wall Powercycle] LV power cycle of forward wall. Requires ''Reprogram FPGA''afterwards. + \item[ECAL RPC 48V Powercycle] LV powercycle for all DAQ boards mounted in the ECAL frame + \item[ECAL Padiwa Powercycle] LV power cycle of ECAL. + \item[ECAL Rack Powercycle] LV power cycle for some auxilliary boards in the ECAL frame + \item[Wall Powercycle] LV power cycle of forward wall. Requires ''Reprogram FPGA'' afterwards. \item[Start Powercycle] LV power cycle of Start detector - \item[HUBS Powercycle] LV power cycle of trbnet hub nodes. Requires ''Reprogram FPGA''afterwards. + \item[MdcTofCentral Powercycle] LV power cycle of some central boards (MDCHub, TOF, CTS...). Requires ''Reprogram FPGA'' afterwards + \item[HUBS Powercycle] LV power cycle of trbnet hub nodes. \item[MDC LV Powercycle] LV power cycle of MDC. Requires ''Reboot OEP'' and ''Start DAQ'' afterwards. To be used, if DAQ- or MDCHub-restart did not work sufficiently. -\item[RPC special PwrCyc] LV power cycle of RPC. Requires ''Reprogram FPGA''afterwards. +\item[RPC special PwrCyc] Don't use. \end{description} diff --git a/hadesoperator/expertknowledge.tex b/hadesoperator/expertknowledge.tex index c461494..9a05ad4 100644 --- a/hadesoperator/expertknowledge.tex +++ b/hadesoperator/expertknowledge.tex @@ -21,7 +21,7 @@ Only cooling fans. Don't change. \item[Forward Wall] haepcp06 \\ Powers the TRB2 and Hub of Forward Wall. FPGA reprogram needed. \item[Pexor PC] haepcp02 \\ Hard Reset for PC. \item[White Rabbit PC] haepcp10 \\ Hard Reset for PC. -\item[RPC something] haepcp15 +\item[RPC something] haepcp15 - unknown purpose \end{description} @@ -34,11 +34,5 @@ Easiest way: \end{enumerate} \section{Full Power Cycle} -To do a full power-cycle (by experts only!) -\begin{enumerate} -\item "HUBS Powercycle", "MDC LV Powercycle", "Start Powercycle", "MdcTofCentral Powercycle", "ECAL Powercycle", 'Wall Powercycle" -\item "ECAL LV5 Powercycle" - only when ECAL front-ends (Padiwa) failed -\item "Reprogram FPGA" - for all TRB2 based boards and AddOns. Boards need ~1 minute to boot after power cycle -\item "RICH Powercycle". It might be necessary to clear temperature interlock in EPICS first. Can be done during "Reprogram FPGA" -\item "Reboot OEP" - reload MDC front-ends (does a reset) -\end{enumerate} +To do a full power-cycle (by experts only!): Follow the ``Daily DAQ reboot'' guide. + diff --git a/hadesoperator/main.tex b/hadesoperator/main.tex index 1a5083f..40270e4 100644 --- a/hadesoperator/main.tex +++ b/hadesoperator/main.tex @@ -107,8 +107,7 @@ \tableofcontents \part{General Advice} - -If one board is missing, do not panic! Do not risk many seconds of possible data taking for one missing board. Also take into account: if MDC is not running on HV, some chambers do not see any noise at all, so there will be some motherborads without data. In this case do not panic, if the ''MBO wo data'' button on the tactical overview shows some errors. +Please read the additional page with notes for the DAQ Operator. \cleardoublepage \part{Control} diff --git a/hadesoperator/powercycles.tex b/hadesoperator/powercycles.tex index fcf2c99..d8b6642 100644 --- a/hadesoperator/powercycles.tex +++ b/hadesoperator/powercycles.tex @@ -1,5 +1,5 @@ The main power supplies can be controlled. A full list for the experts can be found in section \ref{powersupplies}. - +Before doing a power cycle, please stop trigger using ``Start Stop Trigger'' in the control window. \begin{figure}[htp] @@ -10,17 +10,19 @@ The main power supplies can be controlled. A full list for the experts can be fo \end{figure} \subsection{MDC Chamber} If a few MDC boards fail (with a "FEE error" or "MBO w/o data") and a DAQ restart doesn't help (or is needed frequently), a power cycle of one MDC chamber can be performed: Read the address of the failing board (e.g. 2045) from the Tactical Overview. Use the Icon "MDC LV Powercycle" and select the corresponding button (here 204x). -Afterwards, the correct FPGA design needs to be loaded to the OEP. Click the ``RebootOEP'' button. After 10 seconds all OEP are rebooted and DAQ can be started again. If rebooting of the OEP fails, first run a DAQ start-up until the script complains ``OEP design too old''. Close the window and try rebooting OEP again. +Afterwards, the correct FPGA design needs to be loaded to the OEP. Click the ``RebootOEP'' button. After 10 seconds all OEP are rebooted and DAQ can be started again. If rebooting of the OEP fails, first run a DAQ start-up until the script complains ``OEP design too old''. Close the window and try ``Reboot OEP'' again. \subsection{Full MDC} -If it's unclear which part of MDC causes an error, or if there is a larger number of boards failing, you can do a complete MDC power cycle using "EPICS Power" and "MDC Group" instead of using the "MDC LV Powercycle". +If it's unclear which part of MDC causes an error, or if there is a larger number of boards failing, you can do a complete MDC power cycle using "EPICS old" and "MDC Group" instead of using the "MDC LV Powercycle". \subsection{RICH} -All boards of the RICH subsystem (addresses 7xxx and 82xx) are powered by the supplies in the "RICH Group". Use the button "RICH Group ON | OFF" to first switch them off (takes about 10 seconds), wait a few seconds and switch them on again. When all 9 power supplies are active again, do a DAQ restart. +All boards of the RICH subsystem (addresses 7xxx and 82xx) are powered by the supplies in the "RICH Group". Use the button "RICH Powercycle" to switch them off and on again. When all power supplies are active again (see the list in ``EPICS old'' -> ``LV crates''), do a DAQ restart. \subsection{ECal / RPC} -There are two separate power supplies for ECal: The front-ends (Padiwa) can be power cycled using "ECAL 5V Powercycle". After that a DAQ restart is needed to load the correct settings again. The TRB boards share a common power supply with RPC (available in EPICS), so that a "Reprogram FPGA" is needed afterwards. +There are two separate power supplies for ECal: The front-ends (Padiwa) can be power cycled using "ECAL Padiwa Powercycle". After that a DAQ restart is needed to load the correct settings again. + +The TRB boards share a common power supply with RPC (``ECAL RPC 48V Powercycle''), so that a "Reprogram FPGA" is needed afterwards. \subsection{Full Powercycle} -A full power cycle should only be done after contact to an expert. A list of necessary steps is found in the Expert Section of this manual. +A full powercycle should only be done after contact with an expert. Follow the guide for the ``Daily Powercycle''. -- 2.43.0