\subsection{Eventbuilder Tools}
\begin{description}
- \item[Restart EB] Start eventbuilders again (with new settings or after stop)
- \item[Stop EB] Stop eventbuilders
- \item[Restart BNET N Builder] Sets the number of running instances of eventbuilders. Change only on request. E.g. 1 Builder for cosmics, 6/8 for beam data.
- \item[Start/Stop TDC CAL] Do a calibration run for TDCs. See the corresponding section of the manual
- \item[Prefix XX] Change the file prefix. 'co' for cosmics, 'be' for beam data, 'te' during breaks
-\end{description}
+ \item[StartTDC CAL] Start calibration run for the trb3 TDCs. This will disable the regular CTS trigger inputs and send special calibration events with a 500 Hz pulser.
+ At the same time, the eventbuilders will record files of prefix ''TC''.
+ \item[Stop TDC CAL] Stop the calibration run for trb3 TDCs that was started with the above button. This button has to be pressed when all DAQ input subsystems as shown
+ in the eventbuilder input nodes have turned from blue to green color. On pressing, the ''TC'' files are stopped and the new calibration is used for the next data taking runs.
+ \item[BEAM files] Start recording files triggered with accelerator beam. These appear with prefix ''BE'' in eventbuilder monitor and are accounted in the runtime database (logbook).
+ \item[COSMICS files] Start recording files triggered without beam by cosmic radiation. These appear with prefix ''CO'' in eventbuilder monitor and are accounted in the runtime database
+ (logbook).
+ \item[Test file] Start recording any kind of test files. These appear with prefix ''TE'' in eventbuilder monitor and are NOT accounted in the runtime database.
+ Test files are not stored permanently!
+ \item[NO FILE] Stop recording any kind of file. This will close previous files. Note that eventbuilders are not stopped by this, but continue running without writing files.
+ However, the QA and raw DAQ monitoring clients are still delivered with data in this state.
+
+ \item[Restart EB] Start eventbuilders again (with most recent settings)
+ \item[Stop EB] Stop eventbuilders. For expert and testing purpose only!
+ \item[Set N EB] Sets the number of running instances of eventbuilders and restart the building network with the new configuration.
+ Change only on request. E.g. 1 EB for cosmics, 8 or 12 EB for beam data.
+
+ \end{description}
\subsection{Expert Tools}
+{\it This section is visible only when the ''Show Expert Tools'' box on the upper left corner is checked.
+It contains actions which are not necessary for regular DAQ operation.}
+
\begin{description}
\item[MDC HighVoltageRamp]
+ \item[BLR Reinit FPGA]
+ \item[BLR Reload Settings]
\item[CTS Scope]
- \item[ECal 5V Entrance Powercycle]
- \item[ECal LV5 Powercycle]
- \item[MDC LV Powercycle]
\item[MDC LV Turn On All Relais]
- \item[Pexor Repair]
- \item[Reboot RICH]
+ \item[Reload RICH]
\item[Reload RPC Thresholds]
- \item[Restart BNET Control]
+ \item[Restart BNET Control] Restart the DABC master process on lxhadeb07 that controls the building network (BNET).
+ This process also delivers the eventbuilder monitoring at \url{https://hades33/eb}
\item[Restart Hmon] use to activate changes in main monitoring
- \item[Restart Rawdata Mon] Use if \url{https://hades33/rawmon} is not working
+ \item[Restart Rawdata Mon] Restart the DABC analysis of the raw data contents. Use this button if \url{https://hades33/rawmon} is not working
\item[Restart Sequencer]
- \item[RICH Powercycle]
- \item[RICH special PwrCyc]
\item[Set RICH HV] Switches RICH HV on if all channels are off, otherwise switches RICH HV off
\item[Set RICH Thr] Sets thresholds in RICH to default values from file
\item[Start Speech Output]
\item[Stop Speech Output]
- \item[Wall Powercycle]
\end{description}
+\subsection{Power Tools}
+This section offers buttons to cycle the low voltage supplies of several detector systems. These are convenience
+shortcuts to avoid operations on the full detector control system GUI.
+A power cycle may be necessary in some situations
+to reset the DAQ front-end boards. {\bf Note that after a power cycle it is mostly mandatory to perform
+a ''Reprogram FPGA'', ''Reboot OEP'' and ''Reboot MDCHub'' by
+clicking these buttons in the DAQ-Operator section!}
+
+\begin{description}
+ \item[Powercycle all] LV power cycle of everything. Requires ''Reprogram FPGA'', ''Reboot OEP'', and ''Reboot MDCHub''
+ afterwards
+ \item[RICH Powercycle] LV power cycle of RICH.
+ \item[RICH Fan Powercycle] LV power cycle of RICH Fans.
+ \item[ECAL LV5 Powercycle] LV power cycle of ECAL.
+ \item[Wall Powercycle] LV power cycle of forward wall. Requires ''Reprogram FPGA''afterwards.
+ \item[Start Powercycle] LV power cycle of Start detector
+ \item[HUBS Powercycle] LV power cycle of trbnet hub nodes. Requires ''Reprogram FPGA''afterwards.
+ \item[MDC LV Powercycle] LV power cycle of MDC. Requires ''Reboot OEP'' afterwards.
+\item[RPC special PwrCyc] LV power cycle of RPC. Requires ''Reprogram FPGA''afterwards.
+
+
+\end{description}
\ No newline at end of file
{\tt http://hades33/eb/}. The default GUI displays the state of the BNET as shown in
in figure \ref{fig:evtbuild:bnetgui}. All nodes of the BNET show up as separate rows with respective properties (hub inputs, file outputs) monitored. Clicking on
an element will allow further inspection and trending plots.
+Please see section \ref{sec:ebmonguide:dabc} for more details.
\begin{figure}[htb]
\centering
- \includegraphics[width=1\textwidth]{../bnetgui_plain_cut.png}
+ \includegraphics[width=1\textwidth]{figures/BNET_GUI.png}
\caption[BNET master web control gui]{BNET master web control gui}
\label{fig:evtbuild:bnetgui}
\end{figure}
--- /dev/null
+\section{Eventbuilder Operator Guide}
+\label{sec:eboperatorguide}
+
+
+\subsection{DABC Webserver monitor }
+\label{sec:ebmonguide:dabc}
+In 2018 the HADES event building system was changed in topology. Before, all $n$ DAQ hubs sending their subevents to all $m$ eventbuilders
+each of which combines all subevents to full event packets. By switching the subevent destionation of the hubs in
+a ``barrel-shift'' manner, the eventbuilding load was distributed to all active eventbuilder entities.
+In the new scheme, each DAQ hub is configured to always send to a dedicated input node
+of the DABC builder network (BNET).
+
+Moreover, run number synchronisation between parallel event builder files is no longer done by the EPICS ''master'' ioc of EB1.
+Instead, all controls of the BNET is handled by a central {\em Master} process located on the server lxhadeb03. This master processes runs independent of the other BNET processes
+and will re-connect to their command channels whenever the eventbuilders are restarted.
+
+{\bf NOTE: The BNET master process itself can be re-started by button ''Restart BNET Master'' in the expert section of the DAQ control gui, see section \ref{sec:shortguide:daqcontrolwindow}.}
+Usually this is necessary only once before data taking, or after a DABC software upgrade of the master process.
+
+The BNET master offers a monitoring and control GUI on a webserver at the address {\tt http://lxhadeb07:8099/}. It is also available as part of the hmon webserver at \newline
+{\tt http://hades33/eb/}. When this address is opened in any web browser (e.g. firefox, chrome), the default GUI displays the state of the BNET as shown in figure \ref{fig:ebmonitorguide:dabc1}.
+This screenshot explains the main graphical elements and the functionality of the most important buttons. Besides the control buttons,
+Clicking on any active object gui elements (i.e. the text is underlined as a hyperlink), an appropriate plot or text information will
+show up in the main display area. Just holding the mouse over a display element will after few scondes usually open a hover window showing additional information, or a tooltip help.
+Note that the checkbox for ''Monitoring display'' has to be enabled to update the display frequently every 2 seconds.
+
+Figure \ref{fig:ebmonitorguide:dabc2} describes the GUI elements related to data taking and file writing. The respective buttons on top allow to select the next run type
+and to start and stop datataking at any time. Note that the BNET processes are not restarted when changing the run type as with the old eventbuilders, but continue even when
+writing to files is stopped. In this case, just the filename display will become empty and the builder node color turns yellow.
+Clicking on any of the rate and file size numbers allows to watch the trending graph of the assigned value in the display area.
+
+\newpage
+Figure \ref{fig:ebmonitorguide:dabc3} illustrates the details of the GbE hub input display.
+For each BNET input node row, the UDP receiver ports are listed with the trbnet
+address of the sending hub (if any data has yet been received!). The color of each hub shows the actual state of data receiving:
+
+\begin{description}
+ \item [Green]: Data is received properly (TDC calibration runs: Calibration is sufficient)
+ \item [Red]: No data is currently received (TDC calibration runs: Calibration is not ready, or calibration file not yet existing)
+ \item [Blue]: TDC calibration runs only: Calibration is ongoing, but statistics not yet sufficient)
+\end{description}
+
+If Calibration of trb3 inputs is not yet existing (red hub although full eventbuilding is running), a calibration procedure has to be
+done\footnote{Files of type ''Tc'' must be started. The hub inputs without calibration will turn blue until a sufficient TDC fine time
+calibration has been accumulated. Stopping the calibration runs with the button will store the new calibration as files for each tdc address into the {\em /home/hadaq/oper}
+directory on the eventbuilder server. The following runs will use thes calibration files and all hubs should show green color.}.
+Every single GbE hub input may be inspected by holding the mouse over it to see a hover info window. Clicking on it will display further information:
+The current receiver statistics is printed in a text field. For trb3 systems that are configured for online TDC calibration, the TDC sub-subevents
+with their addresses will be displayed. The color of each tdc will indicate if the statistics is sufficent for a TDC calibration
+(red: low statistics, blue: calibration ongoing, green: statistics is OK for doing new calibration).
+Clicking on the central hub box will display a comparison of count statistics for each TDC. If DABC is configured for debugging with additional histograms, clicking
+on each TDC box displays comparison of channel statistics. This mode is disabled for HADES production beamtime because of performance reasons!
+
+Figure \ref{fig:ebmonitorguide:dabc4} is an example of an input node that is not a trb3 hub with TDC calibration. In this case, clicking on the hub will offer buttons to display the
+trigger type statistics, or the subevent size histogram. {\bf Please note that the trigger type information are available only in the CTS hub 0x8800}.
+
+
+Clicking on the {\em hostname:port} title of a BNET node row will display the eventbuilder ''terminal''
+offering an ASCII monitor of the event building at the local inputs (see figure \ref{fig:ebmonitorguide:dabc5}).
+This is the DABC emulation of the old {\em hadaq} and {\em netmem} eventbuilder output, which is also known from standalone DABC datataking systems.
+Depending on the type of BNET node, parts of the generic terminal information is missing: For BNET input nodes, there is no final data file output.
+For builder nodes, there is no UDP receiver information, since the input rows reflect the DABC socket connections to the BNET input nodes.
+
+Figure \ref{fig:ebmonitorguide:dabc5} contains a kind of tutorial how the user can set up an individual display of monitored values.
+Chosing the webserver address {\tt http://lxhadeb03:8090/?browser=fix} displays the usually hidden hierarchy browser
+of the DABC master node. Here all known parameters, commands, and entities of the BNET are available in a hierarchic tree view.
+Clicking on any icon will display it in the currently active display area (usually the main display area of the default GUI).
+However, it is possible to change the default GUI and arrange the GUI elements in an arbitrary manner.
+Using the ''clear'' link in the DABC object browser will erase the default elements. The user may chose any of the given view partitionings and pad modes
+in the selection menu (e.g. a 4x4 grid of pads, different tabs, or flexible desktop-like subwindows). All newly clicked icons will then draw with respect to the selected style
+(the example in figure \ref{fig:ebmonitorguide:dabc5} shows the ''flex'' option).
+The right mouse button menu of the browser icons offers further draw options, e.g. a gauge instrument instead of trending graphs for rates.
+
+Note that frequent monitoring of the elements must be enabled explicitly in the marked checkbox on the top left.
+
+Once the monitoring entities of interest have been arranged in such way, the user can save it for later usage by means of the globe icon (''Global'')
+on top of the tree view. Right mouse button menu on such icon offers a ''Direct link'' option which will copy the current view to another browser window.
+The URL of this new browser window (or tab depending on browser set up) contains all required information, including the monitoring mode of the plots.
+It can be stored as bookmark (favourite address) in the browser, so it can be recalled later to show the user defined display.
+
+
+% \begin{figure}[htb]
+% \centering
+% \includegraphics[width=1\textwidth]{bnetgui_plain_cut.png}
+% \caption[Default BNET master control gui]{Default BNET master control gui}
+% \label{fig:ebmonitorguide:bnetguiplain}
+% \end{figure}
+%
+\clearpage
+
+
+
+\begin{figure}[!htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=1,angle=90]{figures/bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: overview]{BNET web GUI: overview}
+ \label{fig:ebmonitorguide:dabc1}
+\end{figure}
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=2,angle=90]{figures/bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: file controls]{BNET web GUI: file and run control}
+ \label{fig:ebmonitorguide:dabc2}
+\end{figure}
+
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=3,angle=90]{figures/bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: input monitor]{BNET web GUI: hub and trb3 monitor}
+ \label{fig:ebmonitorguide:dabc3}
+\end{figure}
+
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=4,angle=90]{figures/bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: show triggertype]{BNET web GUI: how to show trigger type statistics}
+ \label{fig:ebmonitorguide:dabc4}
+\end{figure}
+
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=5,angle=90]{figures/bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: eventbuilder terminals for each node]{BNET web GUI: how to show eventbuilder terminals for each node}
+ \label{fig:ebmonitorguide:dabc5}
+\end{figure}
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=6,angle=90]{figures/bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: Expert browser and user defined GUI]{BNET web GUI: how to set up a user defined GUI display}
+ \label{fig:ebmonitorguide:dabc6}
+\end{figure}
+
+
+\clearpage
+%
+\subsection{Eventbuilder monitoring in hmon}
+This is new and documentation still to do.
+
+
+
+
\clearpage
\part{Handling Procedures}
\section{TDC Cablibration Run}
+\input{tdccalibration}
\section{Power Cycles}
\input{powercycles}
\clearpage
\part{Eventbuilder}
-
-
+\input{eboperatorguide}
\clearpage
\part{QA}
--- /dev/null
+\subsection{TDC calibration in DABC software}
+The calibration of the trb3 TDC finetime counters is currently handled in the eventbuilder input nodes with the DABC software.
+The eventbuilder software has to aquire calibration information for each trb3 channel with special ''calibration runs''.
+In these runs the CTS will operate a pulser trigger of a few 100 Hz with a dedicated {\em calibration trigger} of type D.
+The trb3 boards will then produce special calibration data that is used to find out the channel-specific corrections.
+This calibration information is stored by DABC and will be used in all subsequent runs to mark the raw data from trb3 systems
+with correction factors for each hit message. The calibration meta information extracted from the calibration run is kept as file for
+each input subsystem on the eventbuilder server. It is also archived automatically to the tsm and /lustre storage
+into the subfolder {\em cal}
+
+\subsection{Operator calibration procedure}
+Since the TDC fine time calibration may change due to temperature changes or other long term drifts, it is necessary to repeat
+the calibration procedure frequently. The uppermost left field in the BNET control GUI tells the time since last successful
+calibration (see Fig. \ref{fig:evtbuild:bnetgui} where the time is shown as ''7h04m''). If this display turns red, usually a calibration
+run has to be performed.
+
+The calibration procedure is as follows:
+
+\begin{enumerate}
+ \item Be sure that the beam is off and no other data source may interfere with the calibration pulser!
+
+ \item Click the Button {\bf Start TDC Cal} in the DAQ operator GUI. This will disable all inputs of the CTS and put it into the
+ special calibration mode (trigger type D, 500~Hz pulser). The eventbuilders will start writing files with prefix ''TC''.
+
+
+ \item Observe the eventbuilder BNET GUI (Fig. \ref{fig:evtbuild:bnetgui}). The ''Input nodes'' box shows for each eventbuilder server
+ the subsystems (''HUBs'') which send data to them from the cave. These are labeled with their trbnet hub address, e.g. ''0x83c1''.
+ During calibration, the trb3 systems will turn to a blue color. The ''Run control'' caption of the webpage will show the state ''Calibrating''.
+
+ \item When all of the input hubs in the BNET GUI have turned to green color, the calibration statistics is sufficient.
+ In this state the ''Run control'' state will show ''Ready''. It can take several minutes until this is reached, depending
+ on the configured calibration precision in the setup.
+
+
+ \item If BNET GUI (or corresponding hmon display) shows that calibration is OK, stop the calibration procedure by
+ pressing the {\bf Stop TDC Cal} button in the DAQ control GUI ({\bf do not stop the run with the ''NO FILE'' button,
+ and do not start immediately recording any other files types by the corresponding control GUI buttons}).
+ The eventbuilders will close the calibration run files and will archive the most recent calibration information.
+ The trigger settings of the CTS will be restored to the values just before the calibration procedure.
+
+ \item If BNET GUI should show that calibration is not successful after 5 minutes, find out which subsystems have
+not turned to green and stay blue instead. Please check thresholds and detector setup of these systems, it might be that there are noise
+hits in addition to the calibration pulser hits. The calibration procedure can be stopped
+at any time anyway and will use the calibration infos although not complete. Repeat the calibration procedure after the reason
+for the missing statistics has been fixed as soon as possible.
+
+ \item Be sure that the beam or the desired trigger source is switched on before starting beam or cosmics files again!
+
+\end{enumerate}
+
+
jspc29 / daqdocu.git / commitdiff
