\label{section:daqcontrolgui}
-
+(updated 2023-10-27)
Most control tasks can be done using the DAQ Control GUI, only few things need access to other tools.
\begin{figure}[htp]
\centering
- \includegraphics[width=\textwidth]{figures/DaqControl.png}
+ \includegraphics[width=\textwidth]{figures/DaqControl2023.png}
\caption{DAQ Control Window}
\label{fig:daqcontrol}
\end{figure}
-\subsection*{DAQ-Operator}
+\subsection*{DAQ Operation}
\begin{description}
\item[Start DAQ] The main script to restart the DAQ. Standard solution if some problems appear (red blinking).
\item[Start Stop Trigger] Interrupt sending triggers to stop data taking for a while
- \item[Reprogram FPGA] Reload designs to old TRB2 based boards (CTS only)
- \item[Reboot OEP] Reload designs of MDC front-ends (e.g. to be used if error "oeps too old"
+% \item[Reprogram FPGA] Reload designs to old TRB2 based boards (CTS only)
+% \item[Reboot OEP] Reload designs of MDC front-ends (e.g. to be used if error "oeps too old"
appears)
- \item[Reboot MDC Hub] Reload designs for MDC hub boards
- \item[ReSync Missing OEP] Tries to get a failed MDC front-end running again
+% \item[Reboot MDC Hub] Reload designs for MDC hub boards
+% \item[ReSync Missing OEP] Tries to get a failed MDC front-end running again
\item[Resync DiRich] Tries to recover failing RICH front-ends
- \item[Reboot CentralHubs] Reloads the firmware for all central hub boards (1)
- \item[Reboot DiRich] Reloads the firmware for RICH front-ends (1)
- \item[Reboot Ecal] Reloads the firmware for all ECal boards (1)
- \item[Reboot RICH Combiner] Reloads the firmware for the first layer of RICH network hubs (1)
- \item[Reboot RICH Hubs] Reloads the firmware for the second layer of RICH network hubs (1)
-\end{description}
-
-The tools marked with (1) can only be used succesfully, if all boards are responding and are not reported as missing. If there are missing boards, first try a DAQ restart and then the reboot.
-
-\subsection*{Monitor Tools}
-\begin{description}
- \item[CTS Mon] The monitor for CTS settings and scalers. One instance should be open for experts, but usually not needed by the operator
- \item[Hmon] Opens a browser pointed at \url{https://hades33/mon}
- \item[Epics Power] Opens a VNC with the EPICS control for power supplies
- \item[Eventbuilder] Opens a browser pointed at \url{https://hades33/eb} to monitor eventbuilders -
-mainly for experts
-% \item[Daq VNC] This VNC is typically used by experts to remotely control the system. Not used for
-% operating the system from the control room
- \item[Chat] A terminal based chat window, not used during beam time
- \item[Hmon Snapshot] Creates a copy of all plots and windows in Hmon for documentation
-\end{description}
-
-\subsection*{Eventbuilder Tools}
-\begin{description}
+% \item[Reboot CentralHubs] Reloads the firmware for all central hub boards (1)
+% \item[Reboot DiRich] Reloads the firmware for RICH front-ends (1)
+% \item[Reboot Ecal] Reloads the firmware for all ECal boards (1)
+% \item[Reboot RICH Combiner] Reloads the firmware for the first layer of RICH network hubs (1)
+% \item[Reboot RICH Hubs] Reloads the firmware for the second layer of RICH network hubs (1)
\item[BEAM files] Start recording files triggered with accelerator beam. These appear with
prefix ''BE'' in eventbuilder monitor and are accounted in the runtime database (logbook).
\item[COSMICS files] Start recording files triggered without beam by cosmic radiation. These
\item[NO FILE] Stop recording any kind of file. This will close previous files. Note that
eventbuilders are not stopped by this, but continue running without writing files.
However, the QA and raw DAQ monitoring clients are still delivered with data in this state.
-
+\end{description}
+
+% The tools marked with (1) can only be used succesfully, if all boards are responding and are not reported as missing. If there are missing boards, first try a DAQ restart and then the reboot.
+
+\subsection*{Monitoring}
+\begin{description}
+ \item[CTS Monitor] The monitor for CTS settings and scalers. One instance should be open for experts, but usually not needed by the operator
+ \item[Epics] Opens a VNC with the EPICS control for power supplies and links to other detector controls
+% \item[Eventbuilder] Opens a browser pointed at \url{https://hades33/eb} to monitor eventbuilders -
+mainly for experts
+ \item[Daq Expert VNC] This VNC is typically used by experts to remotely control the system. In the control room only used to see actions by the expert.
+ \item[Chat] A terminal based chat window, not used during beam time
+ \item[Hmon Snapshot] Creates a copy of all plots and windows in Hmon for documentation
+\end{description}
+
+\subsection*{Eventbuilder}
+\begin{description}
+ \item[Reset EB] Resets the eventbuilder, e.g. empties all buffers.
\item[Restart EB] Start eventbuilders again (with most recent settings). It needs to be followed by
clicking ''BEAM file'' (or ''TEST file'' or whatever) to start recording data again.
\item[Set N EB] Sets the number of running instances of eventbuilders and restart the building
-network with the new configuration.
- Change only on request. E.g. 1 EB for cosmics, 8 or 12 EB for beam data.
-
+network with the new configuration. Change only on request. E.g. 1 EB for cosmics, 16 EB for beam data.
\end{description}
\subsection*{Settings}
\begin{description}
-
+ \item[Beam Abort OnOff] Toggles the beam-abort (in-)active
+ \item[Beam Abort Reset] Reset the beam abort error message in the tactical overview
\item[BLR Reinit FPGA] Reconfigures the analog trigger collection board from TOF / RPC. Use when
rates from TOF/RPC or M2 are wrong.
- \item[Thresholds XXX] Reload thresholds for the named detector. Use if something seems to be wrong
+ \item[XXX Thresh] Reload thresholds for the named detector. Use if something seems to be wrong
with frontend settings
\end{description}
-
+\subsection*{Reboot}
+ \begin{description}
+
+ \item[SYSTEM] Reboots FPGA boards of the names system. Only affects boards that are currently online (not ``missing endpoints''), takes just a few seconds.
+ \item[Reprogram CTS] Reprograms the CTS with its firmware
+ \item[MDC OEP] Special reboot procedure for MDC front-ends - takes a while, should only be used if necessary
+ \end{description}
+
+\subsection*{Powercycle}
+This section offers buttons to cycle the low voltage supplies of several detector systems. These are convenience
+shortcuts to avoid operations on the full detector control system GUI.
+A power cycle may be necessary in some situations to reset the DAQ front-end boards. \\
+{\textbf MDC LV} opens a new window to cycle the electronics of an individual chamber.
+
\subsection*{Expert Tools}
{\it This section is visible only when the ''Show Expert Tools'' box on the upper left corner is checked.}
This button has to be pressed when all DAQ input subsystems as shown
in the eventbuilder input nodes have turned from blue to green color. On pressing, the ''TC''
files are stopped and the new calibration is used for the next data taking runs. For more info see sec.~\ref{sec:tdc_calibration_procedure}.
+ \item[Start TDC Caltest] Records a file with calibration triggers to test current calibration. Should be stopped after ~ 30 seconds.
+ \item[Stop TDC Caltest] Return to normal data taking, calculate calibration status
\item[Stop EB] Stop eventbuilders. For expert and testing purpose only!
% \item[MDC HighVoltageRamp] ! NOT working at the moment
\item[Restart BNET Control] Restart the DABC master process that controls the building
\item[Set RICH HV] Switches on all RICH HV lines and sets all HV modules to the correct supply voltage setting.
If only one module is already switched on, this button will ramp down all HV modules instead.
During ramp up keep an eye on \url{hades33/mon/monitor.cgi?6-RichRatev2} maximum rate should not exceed 1 MHz for longer times
- \item[Start / Stop Speech Output] Restart / Silence speech output of DAQ issues
+ \item[Stop Speech Output] Silence speech output of DAQ issues
\end{description}
-
-
-\subsection*{Power Tools}
-This section offers buttons to cycle the low voltage supplies of several detector systems. These are convenience
-shortcuts to avoid operations on the full detector control system GUI.
-A power cycle may be necessary in some situations
-to reset the DAQ front-end boards.
-{\it These scripts might or might no be up to date. Needs some work.}
-\begin{description}
-
-
-% \item[Powercycle all] LV power cycle of everything. Requires ''Reprogram FPGA'', ''Reboot OEP'', and ''Reboot MDCHub''
-% afterwards
- \item[RICH Powercycle] LV power cycle of RICH.
- \item[RICH Fan Powercycle] LV power cycle of RICH Fans.
- \item[ECAL RPC 48V Powercycle] LV powercycle for all DAQ boards mounted in the ECAL frame
- \item[ECAL Padiwa Powercycle] LV power cycle of ECAL frontends
- \item[ECAL Rack Powercycle] LV power cycle for some auxilliary boards in the ECAL frame
- \item[Wall Powercycle] LV power cycle of forward wall.
- \item[Start Powercycle] LV power cycle of Start detector
- \item[MdcTofCentral Powercycle] LV power cycle of some central boards (MDCHub, TOF, CTS...). Requires ''Reprogram FPGA'' afterwards
- \item[HUBS Powercycle] LV power cycle of trbnet hub nodes.
- \item[MDC LV Powercycle] LV power cycle of MDC. Requires ''Reboot OEP'' and ''Start DAQ''
-afterwards. To be used, if DAQ- or MDCHub-restart did not work sufficiently.
- \item[MDC LV Turn On All Relais] Switches on power to all MDC front-ends. Use if a larger amount
-of
-boards is missing after a power cycle
-
-\item[RPC special PwrCyc] Don't use.
-
-
-\end{description}
\part{Daily News for the Operator}
-\textbf{2022/02/19}
-\begin{itemize*}
- \item Always {\bf keep an eye on the rate plots} (``Main Screen 4'' on the web page). If you
-notice any asymmetry between sectors / parts of detectors, try to do a reload of thresholds for the
-detector. If this doesn't help, talk to the detector expert, e.g. a power cycle of the frontends
-might be necessary.
-\begin{itemize*}
-\item e.g. in ECal a row of 8 green dots appear (-> ECal Padiwa Powercycle)
-\item e.g. in TOF one sector has a lower/higher rate (-> Tresholds TOF)
-\item e.g. in STS a group of 8 blue channels appear (-> Reset STS Thresholds)
-\end{itemize*}
-Special attention is needed after a power cycle!
-
- \item After any of the ``Reboot Hubs`` and DAQ restart: If Eventbuilders do not work, do
-a ''Fix missing IP`` and (if it prints something apart from 'Done') another DAQ restart.
-
- \item After a break, {\bf Eventbuilders need some time to recover}. Wait a bit and don't try to
-restart them more than once per minute. Also try to use ''Reset`` from the Web browser EB interface
-for a 'softer' reset.
-
- \item In most cases a {\bf reboot of a subsystem is the better option} compared to a power cycle.
-If all boards of a subsystem are present, do a reboot, but not a power cycle. Check the missing
-endpoints - {\bf if the boards you want to reboot are missing, it won't work} (apart from MDC).
-
-
- \item There are a couple of recent issues with MDC, where power cycles or reboots are not
-necessary:
-\begin{itemize*}
-\item If 2110 and 2112 are missing after a DAQ restart: Restart again.
-\end{itemize*}
+\textbf{2024/01/01}
+Happy New Year!
-
-\item Many actions can be done {\bf without a DAQ restart}:
-\begin{itemize*}
-\item All actions from the ''Settings`` column can be done while DAQ is running
-\item ''Reset Thresholds STS`` require you to start/stop trigger before and after.
-\item All ''Powercycle Padiwa`` require you to stop trigger before and load thresholds after.
-\end{itemize*}
-
-
-\end{itemize*}
\clearpage
\part{General Advice}
\hyperref[section:daqcontrolgui]{Control GUI}.
\item Monitor the status of the whole system, mainly using the
\hyperref[sec:tacticaloverview]{Tactical Overview}.
- \item Monitor some detailed performance data using the \hyperref[sec:hmon]{Hmon Plots}.
- \item Run the \hyperref[sec:onlineqa]{Online QA} with analysis of the actual detector performance.
+ \item Monitor some detailed performance data using the \hyperref[sec:hmon]{Hmon Plots} (see ''the main windows``).
+% \item Run the \hyperref[sec:onlineqa]{Online QA} with analysis of the actual detector performance.
\item Make entries to the logbook about configuration changes and important issues.
\end{itemize*}
\section{DAQ Shifts}
\begin{itemize*}
- \item Please make sure your first shift is one of the day shifts, not a night shift. During the
-day availability of experts is better, so you can get more help.
- \item Before your first shift, please arrange with one of the other operators to spend a couple of
-hours with them (preferably while having beam in the cave) to learn a bit about the operation of
-the system.
- \item If you do a remote shift, make sure you have access to the DAQ VNC (hades70:1) and the main
+ \item Your first shift should be one of the day shifts, not a night shift. During the
+day availability of experts is better, so you can get more help.
+ \item Before your first shift, you are asked to stay in the counting house for one shift as a ''trainee``, together with another operator. This is necessary to learn how the system looks like, how errors show up and how to get to the right procedure to solve them.
+ \item There will always be a DAQ expert or DAQ professional on call. We'll try to arrange shifts in a way that a knowledgeable operator is ''close by`` (e.g. online) for inexperienced operators.
+ \item If you do a remote shift (e.g. during a cosmics run), make sure you have access to the DAQ VNC (hades70:1) and the main
web server (\url{http://hades33/mon}). Access from outside GSI needs an SSH tunnel made with your
account. \href{https://jspc29.x-matter.uni-frankfurt.de/xmatterpad/p/HADES_GSI_Network_Access}{Some
hints are listed here}.
\end{itemize*}
-\section{General Remarks}
+\section{General To-Do's}
\begin{itemize*}
\item The DAQ operator should always be present in the \textbf{Mumble channel} ``Operators Corner''
- DAQi will provide some audible (sometimes even recognizable) hints if something is wrong.
-\item Don't forget to monitor the \textbf{QA plots} to see whether data quality is ok.
+% \item Don't forget to monitor the \textbf{QA plots} to see whether data quality is ok.
\item If you notice something noteworthy in the Hmon monitoring, use the \textbf{``Hmon Snapshot''}
button to store the current plots for later analysis.
-\item Don’t hesitate to \textbf{call the expert} if there is a problem you can’t solve or you don't
+\item Don’t hesitate to \textbf{call the professional / expert} if there is a problem you can’t solve or you don't
fully understand!
\item Almost \textbf{everything can be accessed} from the web page (\url{http://hades33/mon} and
still active after switching back to "beam" files.
\item \textbf{If one board is missing, do not panic!} Do not risk many seconds of possible data
-taking for one missing board. As a rough guide 2 MDC and/or 2 RICH boards missing is acceptable. If
+taking for one missing board. As a rough guide 2 MDC and/or 4 RICH boards missing is acceptable. If
a board goes missing after an hour of data taking, you can restart the DAQ, but don't do it too
often in case of frequent failures.
\item Please don’t do any power cycle without asking an expert – unless it is explicitly stated
in the manual or help texts.
-\item At least once a day, when there is a break >15 minutes, a \textbf{full restart} of the system
+\item Whenever possible, but at least once a day, when there is a break >15 minutes, a \textbf{full restart} of the system
should be performed. Talk to the DAQ expert to get this done!
+\section{More Remarks}
+\begin{itemize*}
+ \item Please {\bf keep an eye on the rate plots} (``Main Screen 4'' on the web page). If you
+notice any asymmetry between sectors / parts of detectors, try to do a reload of thresholds for the
+detector. If this doesn't help, talk to the detector expert, e.g. a power cycle of the frontends
+might be necessary.
+\begin{itemize*}
+\item e.g. in ECal a row of 8 green dots appear (-> ECal Padiwa Powercycle)
+\item e.g. in TOF one sector has a lower/higher rate (-> Tresholds TOF)
+% \item e.g. in STS a group of 8 blue channels appear (-> Reset STS Thresholds)
+\end{itemize*}
+Special attention is needed after a power cycle!
+
+% \item After any of the ``Reboot Hubs`` and DAQ restart: If Eventbuilders do not work, do
+% a ''Fix missing IP`` and (if it prints something apart from 'Done') another DAQ restart.
+
+ \item After a break, {\bf Eventbuilders need some time to recover}. Wait a bit and don't try to
+restart them more than once per minute. Also try to use ''Reset`` from the Web browser EB interface or the GUI
+for a 'softer' reset.
+
+ \item In most cases a {\bf reboot of a subsystem is the better option} compared to a power cycle.
+If all boards of a subsystem are present, do a reboot, but not a power cycle. Check the missing
+endpoints - {\bf if the boards you want to reboot are missing, it won't work} (apart from MDC).
+
+
+% \item There are a couple of recent issues with MDC, where power cycles or reboots are not
+% necessary:
+% \begin{itemize*}
+% \item If 2110 and 2112 are missing after a DAQ restart: Restart again.
+% \end{itemize*}
+
+
+\item Some actions can be done {\bf without a DAQ restart}:
+\begin{itemize*}
+\item All actions from the ''Settings`` column can be done while DAQ is running
+% \item ''Reset Thresholds STS`` require you to start/stop trigger before and after.
+\item All ''Powercycle Padiwa`` requires you to stop trigger before, load thresholds after and start trigger again.
+\end{itemize*}
+
+
+\end{itemize*}
\end{itemize*}
jspc29 / daqdocu.git / commitdiff
