+\newpage
\section{Eventbuilder Monitor Guide}
\label{sec:ebmonguide}
-\begin{figure}[hp]
+
+
+\subsection{DABC Webserver monitor }
+\label{sec:ebmonguide:dabc}
+In 2018 the HADES event building system was changed in topology. Before, all $n$ DAQ hubs sending their subevents to all $m$ eventbuilders
+each of which combines all subevents to full event packets. By switching the subevent destionation of the hubs in
+a ``barrel-shift'' manner, the eventbuilding load was distributed to all active eventbuilder entities.
+In the new scheme, each DAQ hub is configured to always send to a dedicated input node
+of the DABC builder network (BNET).
+
+Moreover, run number synchronisation between parallel event builder files is no longer done by the EPICS ''master'' ioc of EB1.
+Instead, all controls of the BNET is handled by a central {\em Master} process located on the server lxhadeb03. This master processes runs independent of the other BNET processes
+and will re-connect to their command channels whenever the eventbuilders are restarted.
+
+{\bf NOTE: The BNET master process itself can be re-started by button ''Restart BNET Master'' in the expert section of the DAQ control gui, see section \ref{sec:shortguide:daqcontrolwindow}.}
+Usually this is necessary only once before data taking, or after a DABC software upgrade of the master process.
+
+The BNET master offers a monitoring and control GUI on a webserver at the address {\tt http://lxhadeb03:8090/}. It is also available as part of the hmon webserver at \newline
+{\tt http://hades33/eb/}. When this address is opened in any web browser (e.g. firefox, chrome), the default GUI displays the state of the BNET as shown in figure \ref{fig:ebmonitorguide:dabc1}.
+This screenshot explains the main graphical elements and the functionality of the most important buttons. Besides the control buttons,
+Clicking on any active object gui elements (i.e. the text is underlined as a hyperlink), an appropriate plot or text information will
+show up in the main display area. Just holding the mouse over a display element will after few scondes usually open a hover window showing additional information, or a tooltip help.
+Note that the checkbox for ''Monitoring display'' has to be enabled to update the display frequently every 2 seconds.
+
+Figure \ref{fig:ebmonitorguide:dabc2} describes the GUI elements related to data taking and file writing. The respective buttons on top allow to select the next run type
+and to start and stop datataking at any time. Note that the BNET processes are not restarted when changing the run type as with the old eventbuilders, but continue even when
+writing to files is stopped. In this case, just the filename display will become empty and the builder node color turns yellow.
+Clicking on any of the rate and file size numbers allows to watch the trending graph of the assigned value in the display area.
+
+\newpage
+Figure \ref{fig:ebmonitorguide:dabc3} illustrates the details of the GbE hub input display.
+For each BNET input node row, the UDP receiver ports are listed with the trbnet
+address of the sending hub (if any data has yet been received!). The color of each hub shows the actual state of data receiving:
+
+\begin{description}
+ \item [Green]: Data is received properly (TDC calibration runs: Calibration is sufficient)
+ \item [Red]: No data is currently received (TDC calibration runs: Calibration is not ready, or calibration file not yet existing)
+ \item [Blue]: TDC calibration runs only: Calibration is ongoing, but statistics not yet sufficient)
+\end{description}
+
+If Calibration of trb3 inputs is not yet existing (red hub although full eventbuilding is running), a calibration procedure has to be
+done\footnote{Files of type ''Tc'' must be started. The hub inputs without calibration will turn blue until a sufficient TDC fine time
+calibration has been accumulated. Stopping the calibration runs with the button will store the new calibration as files for each tdc address into the {\em /home/hadaq/oper}
+directory on the eventbuilder server. The following runs will use thes calibration files and all hubs should show green color.}.
+Every single GbE hub input may be inspected by holding the mouse over it to see a hover info window. Clicking on it will display further information:
+The current receiver statistics is printed in a text field. For trb3 systems that are configured for online TDC calibration, the TDC sub-subevents
+with their addresses will be displayed. The color of each tdc will indicate if the statistics is sufficent for a TDC calibration
+(red: low statistics, blue: calibration ongoing, green: statistics is OK for doing new calibration).
+Clicking on the central hub box will display a comparison of count statistics for each TDC. If DABC is configured for debugging with additional histograms, clicking
+on each TDC box displays comparison of channel statistics. This mode is disabled for HADES production beamtime because of performance reasons!
+
+Figure \ref{fig:ebmonitorguide:dabc4} is an example of an input node that is not a trb3 hub with TDC calibration. In this case, clicking on the hub will offer buttons to display the
+trigger type statistics, or the subevent size histogram. {\bf Please note that the trigger type information are available only in the CTS hub 0x8800}.
+
+
+Clicking on the {\em hostname:port} title of a BNET node row will display the eventbuilder ''terminal''
+offering an ASCII monitor of the event building at the local inputs (see figure \ref{fig:ebmonitorguide:dabc5}).
+This is the DABC emulation of the old {\em hadaq} and {\em netmem} eventbuilder output, which is also known from standalone DABC datataking systems.
+Depending on the type of BNET node, parts of the generic terminal information is missing: For BNET input nodes, there is no final data file output.
+For builder nodes, there is no UDP receiver information, since the input rows reflect the DABC socket connections to the BNET input nodes.
+
+Figure \ref{fig:ebmonitorguide:dabc5} contains a kind of tutorial how the user can set up an individual display of monitored values.
+Chosing the webserver address {\tt http://lxhadeb03:8090/?browser=fix} displays the usually hidden hierarchy browser
+of the DABC master node. Here all known parameters, commands, and entities of the BNET are available in a hierarchic tree view.
+Clicking on any icon will display it in the currently active display area (usually the main display area of the default GUI).
+However, it is possible to change the default GUI and arrange the GUI elements in an arbitrary manner.
+Using the ''clear'' link in the DABC object browser will erase the default elements. The user may chose any of the given view partitionings and pad modes
+in the selection menu (e.g. a 4x4 grid of pads, different tabs, or flexible desktop-like subwindows). All newly clicked icons will then draw with respect to the selected style
+(the example in figure \ref{fig:ebmonitorguide:dabc5} shows the ''flex'' option).
+The right mouse button menu of the browser icons offers further draw options, e.g. a gauge instrument instead of trending graphs for rates.
+
+Note that frequent monitoring of the elements must be enabled explicitly in the marked checkbox on the top left.
+
+Once the monitoring entities of interest have been arranged in such way, the user can save it for later usage by means of the globe icon (''Global'')
+on top of the tree view. Right mouse button menu on such icon offers a ''Direct link'' option which will copy the current view to another browser window.
+The URL of this new browser window (or tab depending on browser set up) contains all required information, including the monitoring mode of the plots.
+It can be stored as bookmark (favourite address) in the browser, so it can be recalled later to show the user defined display.
+
+
+% \begin{figure}[htb]
+% \centering
+% \includegraphics[width=1\textwidth]{bnetgui_plain_cut.png}
+% \caption[Default BNET master control gui]{Default BNET master control gui}
+% \label{fig:ebmonitorguide:bnetguiplain}
+% \end{figure}
+%
+\clearpage
+
+
+
+\begin{figure}[!htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=1,angle=90]{bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: overview]{BNET web GUI: overview}
+ \label{fig:ebmonitorguide:dabc1}
+\end{figure}
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=2,angle=90]{bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: file controls]{BNET web GUI: file and run control}
+ \label{fig:ebmonitorguide:dabc2}
+\end{figure}
+
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=3,angle=90]{bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: input monitor]{BNET web GUI: hub and trb3 monitor}
+ \label{fig:ebmonitorguide:dabc3}
+\end{figure}
+
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=4,angle=90]{bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: show triggertype]{BNET web GUI: how to show trigger type statistics}
+ \label{fig:ebmonitorguide:dabc4}
+\end{figure}
+
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=5,angle=90]{bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: eventbuilder terminals for each node]{BNET web GUI: how to show eventbuilder terminals for each node}
+ \label{fig:ebmonitorguide:dabc5}
+\end{figure}
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=.9\textheight,page=6,angle=90]{bnetmonitorguide.pdf}
+ \caption[DABC BNET web GUI: Expert browser and user defined GUI]{BNET web GUI: how to set up a user defined GUI display}
+ \label{fig:ebmonitorguide:dabc6}
+\end{figure}
+
+
+%
+
+
+
+
+\clearpage
+
+\subsection{Old EPICS monitor with MEDM}
+\label{sec:ebmonguide:medm}
+Before 2018,the event builder parameters were exported via shared memory to separate EPICS IOCs for each eventbuilder process.
+These process variables were displayed on the MEDM gui clients. In the following Figures (\ref{fig:ebmonitorguide1},
+\ref{fig:ebmonitorguide2}, \ref{fig:ebmonitorguide3}, \ref{fig:ebmonitorguide2}),the most important windows of such GUI are explained.
+
+{\bf Please note:} This monitoring was still working both for the old EB software (netmem/evtbuild) and for
+parallel DABC eventbuilders, since most parameters were emulated in DABC to match the previous scheme, although the software mechanisms are different.
+It might still display some reasonable values when used in connection with the improved DABC BNET (builder network), see section \ref{sec:ebmonguide:dabc}.
+However, in this case only the currently active final ''builder'' nodes are shown as active (green lamps). The number of input subsystem refers to the
+input nodes of the BNET then, not to the number of DAQ GBe hubs. Information on the BNET inputs from the DAQ hubs is not available anymore in the EPICS GUI.
+
+
+
+\begin{figure}[htb]
\centering
\includegraphics[width=.9\textheight,page=1,angle=90]{ebmonitordescription.pdf}
- \caption[EB Monitor Guide 1]{EB Monitor Guide 1}
+ \caption[EB EPICS general GUI]{EB EPICS general GUI}
\label{fig:ebmonitorguide1}
\end{figure}
-\begin{figure}[hp]
+
+
+\begin{figure}[htb]
\centering
\includegraphics[width=.9\textheight,page=2,angle=90]{ebmonitordescription.pdf}
- \caption[EB Monitor Guide 2]{EB Monitor Guide 2}
+ \caption[EB EPICS data buffer]{EB EPICS data buffer}
\label{fig:ebmonitorguide2}
\end{figure}
-\begin{figure}[hp]
+\begin{figure}[htb]
\centering
\includegraphics[width=.9\textheight,page=3,angle=90]{ebmonitordescription.pdf}
- \caption[EB Monitor Guide 3]{EB Monitor Guide 3}
+ \caption[EB EPICS error statistics]{EB EPICS error statistics}
\label{fig:ebmonitorguide3}
\end{figure}
-\begin{figure}[hp]
+\begin{figure}[htb]
\centering
\includegraphics[width=.9\textheight,page=4,angle=90]{ebmonitordescription.pdf}
- \caption[EB Monitor Guide 4]{EB Monitor Guide 4}
+ \caption[EB EPICS network and trigger statistics]{EB EPICS network and trigger statistics}
\label{fig:ebmonitorguide4}
\end{figure}
+
+
+
+%
+
-\section{Event Building}
+\section{General remarks}
+\label{evtbuild:General}
+Event building means to collect the data that is send from different DAQ subsystems at each trigger signal into a complete event packet, and store it
+on a medium (hard disk or archive tape) for further processing and analysis. Additonally, a part of such event data may be provided to an on-line
+analysis system like Hydra for immediate Quality monitoring (QA).
+
+In HADES DAQ, each detector subsystem concentrates the front-end data that is read out by the trbnet channel protocol into {\em Hub} units.
+The hubs will send their subevents as UDP packets via Gigabit Ethernet (GbE) links to several event builder (EB) servers. These
+consist of PC server computers with Debian/Linux operating system.
+A special event building software on these servers will
+receive the subevent packages of all subsystems, arrange them with respect to the common trigger number in the headers(''build the full event''),
+and store the events into HADES list mode (.hld) formatted raw data files.
+The raw data files are recorded both on the local hard disk on each event builder server, and via a dedicated socket protocol
+to a tape storage archive. Moreover, the archived files will be mirrored automatically to the {\em lustre} file system of the GSI high performance
+compute cluster farm.
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=0.7\textwidth]{EB_nxm.png}
+ \caption[Old eventbuilding topology]{Old eventbuilding topology with NxM barrel shift}
+ \label{fig:evtbuild:nxm}
+\end{figure}
-The four servers lxhadeb01, lxhadeb02, lxhadeb03, lxhadeb04 are the HADES Event Builders plus lxhadeb05 which is a spare server.
-The Event Builder (EB) is aimed to receive subevents from subsystems and to build a complete event out of them. EB consists of two parts: daq\_netmem (the receiving part) and daq\_evtbuild (the building part). A communication between daq\_netmem and daq\_evtbuild is done via a shared memory. Number of opened in a shared memory buffers (queues) is equal to a number of subsystems. A completed event can be written to different mass storage systems.
-The daq\_evtbuild and daq\_netmem can be configured in three ways:
+\section{Event Building since 2018}
+\label{evtbuild:New2018}
+
+\subsection{Server hardware}
+Currently (July 2018) active the event building servers consist of the machines:
\begin{itemize}
- \item take default arguments,
- \item read part of arguments from eb\_p.tcl configuration file (\$DAQ\_SETUP should be set),
- \item read arguments from a command line.
+ \item lxhadeb05
+ \item lxhadeb08
+ \item lxhadeb09
+ \item lxhadeb10
+ \item lxhadeb03 - runs the BNET master process
+ \item lxhadeb02 - spare
+ \item lxhadeb04 - spare
\end{itemize}
-daq\_evtbuild must be executed before daq\_netmem (as daq\_evtbuild is the one who opens buffers in shared memory).
+The newest servers (08,09,10) are equipped with 64 core AMD Epyc 7351 CPU, 256 GB RAM, and RAID6 data disks of 60TB capacity.
+
+
+
+
+
+
+
+\subsection{The DABC framework}
+The software framework DABC (Data Acquisition Backbone Core) provides a flexible environment
+for distributed data acquisition with online monitoring and control capabilities. It is used for event building at
+the HADES experiment since 2014. Initially, DABC had just been configured to replace the previous {\em netmem/evtbuild}
+software (section \ref{evtbuild:Before2018}). In this mode, the run control was still handled by EPICS and all monitoring variables
+of the previous system were emulated in DABC. Additionally, all configuration files and startup scripts were modfied only slightly and would allow
+to switch to the old software immediately.
+
+However, DABC offers more features than the previous software, especially concerning the treatment of trb3 TDC front-ends.
+It contains the functionality to calibrate the TDC fine time values ''on the fly'' in the event builder
+before writing the raw subevents to disk. This has been implemented for different calibration methods
+(statistical approach, temperature calibration function, simple linear
+calibration).
+
+As the HADES detector is being equipped with numerous additional trb3 readout boards in the scope of the
+FAIR-0 beam times, to fully benefit from such an online trb3 TDC calibration, the topology of the event building
+network had to be changed.
+
+
+
+\subsection{The builder network (BNET)}
+
+% In the previous set up, each front-end hub was sending data via Gb Ethernet UDP connections to all receiving event builder
+% nodes in a barrel shift mode (Fig. \ref{fig:evtbuild:nxm}).
+
+In 2018 the HADES event building system was changed in topology. Before, all $n$ DAQ hubs sending their subevents to all $m$ eventbuilders
+each of which combines all subevents to full event packets. By switching the subevent destionation of the hubs in
+a ``barrel-shift'' manner, the eventbuilding load was distributed to all active eventbuilder entities (see Fig. \ref{fig:evtbuild:nxm}). .
+In the new scheme, each DAQ hub is configured to always send to a dedicated input node
+of the DABC builder network (BNET). Each frontend hub will now send to a dedicated entry server only, and the full
+event combination is done by a second TCP/IP ''builder network'' BNET (Fig. \ref{fig:evtbuild:bnet}).
+
+Moreover, run number synchronisation between parallel event builder files is no longer done by the EPICS ''master'' ioc of EB1, as described in section \ref{evtbuild:Before2018:EPICS}.
+Instead, all controls of the BNET is handled by a central {\em Master} process located on the server lxhadeb03. This master processes runs independent of the other BNET processes
+and will re-connect to their command channels whenever the eventbuilders are restarted.
+
+{\bf NOTE: The BNET master process itself can be re-started by button ''Restart BNET Master'' in the expert section of the DAQ control gui, see section \ref{sec:shortguide:daqcontrolwindow}.}
+Usually this is necessary only once before data taking, or after a DABC software upgrade of the master process.
+
+
+
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=1\textwidth]{EB_BNET.png}
+ \caption[New eventbuilding topology]{New eventbuilding topology with BNET. The master node is not shown here.}
+ \label{fig:evtbuild:bnet}
+\end{figure}
+
+The setup with fixed BNET input nodes for each TDC hub allows to configure the TDC calibration modes in DABC
+according to the connected frontends.
+
+
+\clearpage
+\subsection{Control interface}
+The BNET master node offers a monitoring and control GUI on a webserver at the address {\tt http://lxhadeb03:8090/}. It is also available as part of the hmon webserver at \newline
+{\tt http://hades33/eb/}. When this address is opened in any web browser (e.g. firefox, chrome), the default GUI displays the state of the BNET as shown in
+in figure \ref{fig:evtbuild:bnetgui}. All nodes of the BNET show up as separate rows with respective properties (hub inputs, file outputs) monitored. Clicking on
+an element will allow further inspection and trending plots.
+
+\begin{figure}[htb]
+ \centering
+ \includegraphics[width=1\textwidth]{bnetgui_plain_cut.png}
+ \caption[BNET master web control gui]{BNET master web control gui}
+ \label{fig:evtbuild:bnetgui}
+\end{figure}
+
+
+
+Section \ref{sec:ebmonguide:dabc} describes the main elements of such control GUI in more detail.
+
+
+\subsection{Interface to mass data storage LTSM}
+Since the old tape storage tool gstore and the RFIO interface (see section \ref{evtbuild:Before2018:RFIO})
+is not supported anymore at GSI, the new storage interface
+LTSM\footnote{https:/github.com/tstibor/ltsm} has been chosen for HADES.
+A plug-in for DABC has been developed that utilizes the LTSM libraries from the
+event builder process. By this DABC can open new files at a
+dedicated IBM tape storage server and can write directly
+data buffers to such file. This will preserve the previous
+use case that experiment data can be written to the
+mass storage immediately during data taking, without
+requiring an intermediate file on a local disk.
+
+The ltsm libraries and the tool {\em ltsmc} are installed on all active eventbuilder servers.
+
+
+
+
+
+
+\subsection{Configuration}
+The event builder configuration is handled by a number of different files described in the following.
+
+\subsubsection{File eb.conf}
+Located on {\em lxhadesdaq} at {\tt /home/hadaq/trbsoft/hadesdaq/evtbuild/eb.conf}.
+The main configuration file used from eventbuilder startup script.
+
+The most important setup sections:
+
+
+% \begin{description}
+% \item [/home/hadaq/trbsoft/hadesdaq/evtbuild/eb.conf] (on {\em lxhadesdaq}) main configuration file for eventbuilder startup. Most important setup sections:
+\begin{itemize}
+ \item General EB setup:
+ \begin{itemize}
+ \item {\tt EB\_LIST: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0} enable eventbuilder nodes (1-16), also used to enable EPICS ioc monitor)
+ \item {\tt DABC: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0} enable DABC eventbuilder software on single node. If 0, use old netmem/evtbuld software (not possible for BNET)
+ \end{itemize}
+\item For configuration of BNET:
+\begin{itemize}
+ \item {\tt BNETINP 1 1 1 1} - indicate number of BNET input nodes for each of the 4 EB servers. If all are set to 0, BNET is disabled and EB will expect old NxM builder setup
+ \item {\tt BNETBLD 4 4 4 0} - indicate maximum builder nodes for each of the 4 eB servers. The actual number of builders on each node is evaluated from
+this using a command line argument of the startup script
+\end{itemize}
+ \item For configuration of tape archive interface LTSM:
+ \begin{itemize}
+ \item {\tt LTSM\_PATH:} Full path to archive directory. This is also the path to the location of the copys on lustre file sytem.
+ \item {\tt LTSM\_Server:} Hostname of the Tivoli storage management server
+ \item {\tt LTSM\_Node:} Storage node on this server, e.g. {\em hades}
+ \item {\tt LTSM\_Passwd:} Password for the storage node access
+ \item {\tt LTSM\_Filesystem:} Filesystem mount, must be first part of archive path, e.g. {\em /lustre/hebe}
+\end{itemize}
+
+\end{itemize}
+
+ \subsubsection{File data\_sources.db}
+On {\em lxhadesdaq} at {\tt /home/hadaq/trbsoft/hadesdaq/main/data\_sources.db}.
+Switch subevent data sources (hubs) that should be collected by the event builders.
+
+% \item [/home/hadaq/trbsoft/hadesdaq/main/data\_sources.db] (on {\em lxhadesdaq}) Switch subevent data sources (hubs) that should be collected by the event builders
+
+\subsubsection{File register\_configgbe\_ip.db}
+On {\em lxhadesdaq} at {\tt /home/hadaq/trbsoft/hadesdaq/hub/register\_configgbe\_ip.db}.
+
+% \item [/home/hadaq/trbsoft/hadesdaq/hub/register\_configgbe\_ip.db] (on {\em lxhadesdaq})
+Network configuration of GbE hubs and event builders. Also contains flags to
+toggle the BNET trb3 online calibration for single subevent hubs.
+
+For classic NxM builder network (Fig. \ref{fig:evtbuild:nxm}), the destination IP and UDP port numbers are
+specified that each subsystem hub has to send if the CTS specifys an eventbuilder destination type (i.e. an eventbuilder instance) from $0...15$. The active eventbuilder instances are
+selected in bitpattern of the CTS {\em EB\_LUT} register. Each destination instance has a separate table. Additionally, the final table in this file will specify
+the MAC and IP addresses of each destination eventbuilder server network interface (the destination ''type'' range is $100...115$ matching the above $0...15$).
+
+For configuration of BNET (Fig. \ref{fig:evtbuild:bnet}), the destination type 0 is used only, i.e. {\bf the CTS {\em EB\_LUT} must be set to 1 (bit 0)}.
+In this case each hub has a special destination in the table for Type 0. Additionally, the last column will indicate the calibration mode for trb3 TDC fine time
+that DABC should use:
+\begin{itemize}
+ \item 0: NO TDC calibration
+ \item 1: linear calibration
+ \item 2: complex calibration
+ \item 11:linear calibration with time over threshold 15ns
+\end{itemize}
+
+
+
+
+% \begin{verbatim}
+% # Dest Port Src MAC Src IP Src Port # # TDC Calib flag
+% # Hub # Type # C3 # C4 # C6 # C7 # EB IP # 0:NO 1:linear 2:complex 11:linear with ToT 15ns
+% 0x1000 0 0xc660 0xdead1000 0xc0a86490 0xc660 0xc0a86408 0 # mdc - lxhadeb08
+% 0x8a00 0 0xc676 0xdead8a00 0xc0a864A6 0xc676 0xc0a86409 1 # ecal- all to lxhadeb09
+% 0x83c3 0 0xc67f 0xdead83c3 0xc0a864AF 0xc67F 0xc0a8640a 11 # rich - all to lxhadeb10
+% \end{verbatim}
+
+\subsubsection{DABC configuration files}
+The details of the DABC processes are configured in XML files on the eventbuilder servers:
+
+
+\begin{description}
+ \item [/home/hadaq/oper/BnetInputHades.head.xml] (on event builder servers {\em lxhadeb08} etc.) Detail parameters for DABC input nodes of BNET. Is mainly
+steered by environment variables that are set on startup from the eb.conf and register\_configgbe\_ip.db , e.g. to determine port numbers and i/o instances. Specific
+generic values like memory buffer configuration may be set here directly.
+
+\item [/home/hadaq/oper/BnetBuilderHades.head.xml] (on event builder servers {\em lxhadeb08} etc.) Detail parameters for DABC builder nodes of BNET. Is mainly
+steered by environment variables that are set on startup from the eb.conf and register\_configgbe\_ip.db , e.g. to determine run type and i/o instances. Specific
+generic values like memory buffer configuration may be set here directly.
+
+
+\item [/home/hadaq/soft/dabc/head/plugins/hadaq/app/master.xml] (on event builder master server {\em lxhadeb03}) Detail parameters for DABC master controling node
+of BNET. Independent of all other configuration setup.
+
+
+\item [/home/hadaq/oper/EventBuilderHades.head.xml] (on event builder servers {\em lxhadeb08} etc.) Detail parameters for DABC eventbuilder nodes {\bf without BNET}. Is mainly
+steered by environment variables that are set on startup from the eb.conf and register\_configgbe\_ip.db , e.g. to determine port numbers and i/o instances. Specific
+generic values like memory buffer configuration may be set here directly. {\bf this file is deprecated for the BNET setup}.
+
+
+\end{description}
+
+
+
+
+
+
+
+
+% \subsection{Troubleshooting}
+
+
\subsection{Check list: what and how to check before production beam time}
\begin{itemize}
\item Check EB servers:
\begin{itemize}
- \item Check if our EB servers lxhadeb01,2,3,4 are up and running.
+ \item Check if our EB servers lxhadeb05,08,09,10 are up and running. Additionally. lxhadeb03 must be available for the BNET master process.
\item Check if both HADES and GSI VLANs are available from the EB servers.
- \item Clean up all hard disks on all EB servers. If you need to archive some hld files, see subsection~\ref{eb_data_tape}.
+ \item Clean up all hard disks on all EB servers. If you need to archive some hld files, see section~\ref{eb_data_tape}.
\item Check if daq\_disks process runs in the background on each EB server. This process is started by the cronjob at the boot time. If daq\_disks is not running you can start it yourself under 'hadaq' account: \textbf{/home/hadaq/bin/daq\_disks -s 10 > /dev/null 2>\&1 \&}
- \item Check if the following mounts are present. (Unless, execute on lxhadesdaq under 'root': \textbf{mount -a}).
+ \item Check if the following mounts are present on the central server {\em lxhadesdaq}. (Unless, execute on lxhadesdaq under 'root': \textbf{mount -a}).
\begin{itemize}
- \item 192.168.100.11:/home/hadaq/oper -> lxhadesdaq:/home/hadaq/oper/oper\_1
- \item 192.168.100.12:/home/hadaq/oper -> lxhadesdaq:/home/hadaq/oper/oper\_2
- \item 192.168.100.13:/home/hadaq/oper -> lxhadesdaq:/home/hadaq/oper/oper\_3
- \item 192.168.100.14:/home/hadaq/oper -> lxhadesdaq:/home/hadaq/oper/oper\_4
+ \item 192.168.100.15:/home/hadaq/oper -> /home/hadaq/oper/lxhadeb05
+ \item 192.168.100.8:/home/hadaq/oper -> /home/hadaq/oper/lxhadeb08
+ \item 192.168.100.9:/home/hadaq/oper -> /home/hadaq/oper/lxhadeb09
+ \item 192.168.100.10:/home/hadaq/oper -> /home/hadaq/oper/lxhadeb10
\end{itemize}
- \item Check connection to the Data Movers for the data archiving:
+ \end{itemize}
+\item Check connection to LTSM for the data archiving:
\begin{itemize}
- \item Ask Horst Goeringer to create a new archive for the production beam time.
- \item First set a test path in the hadestest archive in lxhadesdaq:\~/trbsoft/daq/evtbuild/eb.conf: \textbf{RFIO\_PATH: rfiodaq:gstore:/hadaqtest/test001}
- \item \textbf{lxhadesdaq:\~/trbsoft/daq/evtbuild/start\_eb\_gbe.pl -e start -n 1-16 --rfio on}
- \item On MEDM GUI you have to see that all 16 EB processes got connected to 8 Data Movers (two connections per Data Mover). In the log file you have to see the message \textbf{rfio\_fopen: opened connection to Data Mover: slxdm17} from each Event Builder. If there are some error messages related to RFIO and Data Movers or there are more than two connections to the same Data Mover, try to contact Horst Goeringer.
- \item Change the archive name and path (RFIO\_PATH:) in lxhadesdaq:\~/trbsoft/daq/evtbuild/eb.conf to the new ones.
+ \item Ask Thomas Stibor/GSI IT to create a new storage node for the production beam time.
+ \item Test availbility of archive by means of tool {\tt ltmsc} (see ''man ltsmc'' on EB servers)
+ \item In configuration file {\tt /home/hadaq/trbsoft/hadesdaq/evtbuild/eb.conf}, on { \em lxhadesdaq} enable LTSM for all builders. Also adjust parameters
+LTSM\_PATH, LTSM\_Node, etc,
+ \item Start eventbuilder processes and observe in control GUI (see \ref{sec:ebmonguide:dabc}) if LTSM files are written.
\end{itemize}
- \end{itemize}
\item Start Perl scripts to insert data to Oracle DB (pay attention to the data base name inside the scripts: \$database):
\begin{itemize}
\item Update Oracle DB with new subevt IDs and new boards (see subsection~\ref{ora_before_running_scripts}).
- \item Start lxhadesdaq:~/trbsoft/daq/oracle/daq2ora\_client.pl (see subsection~\ref{ora_boards_script}).
- \item Start lxhadesdaq:/home/hadaq/oper/runinfo2ora.pl (see subsection~\ref{ora_run_script}).
+ \item Start lxhadesdaq:\~/trbsoft/daq/oracle/daq2ora\_client.pl (see subsection~\ref{ora_boards_script}).
+ \item Start lxhadesdaq:/home/hadaq/oper/runinfo2ora\_start\_parallel.sh (see subsection~\ref{ora_run_script}).
\end{itemize}
\end{itemize}
+
+
+
+
+
+
+
+\section{Event Building before 2018}
+\label{evtbuild:Before2018}
+
+The four servers lxhadeb05, lxhadeb02, lxhadeb03, lxhadeb04 are the HADES Event Builders.
+
+The Event Builder (EB) is aimed to receive subevents from subsystems and to build a complete event out of them. EB consists of two parts: daq\_netmem (the receiving part) and daq\_evtbuild (the building part). A communication between daq\_netmem and daq\_evtbuild is done via a shared memory. Number of opened in a shared memory buffers (queues) is equal to a number of subsystems. A completed event can be written to different mass storage systems.
+
+The daq\_evtbuild and daq\_netmem can be configured in three ways:
+
+\begin{itemize}
+ \item take default arguments,
+ \item read part of arguments from eb\_p.tcl configuration file (\$DAQ\_SETUP should be set),
+ \item read arguments from a command line.
+\end{itemize}
+
+daq\_evtbuild must be executed before daq\_netmem (as daq\_evtbuild is the one who opens buffers in shared memory).
+
+% \subsection{Check list: what and how to check before production beam time}
+%
+% \begin{itemize}
+% \item Check EB servers:
+% \begin{itemize}
+% \item Check if our EB servers lxhadeb01,2,3,4 are up and running.
+% \item Check if both HADES and GSI VLANs are available from the EB servers.
+% \item Clean up all hard disks on all EB servers. If you need to archive some hld files, see subsection~\ref{eb_data_tape}.
+% \item Check if daq\_disks process runs in the background on each EB server. This process is started by the cronjob at the boot time. If daq\_disks is not running you can start it yourself under 'hadaq' account: \textbf{/home/hadaq/bin/daq\_disks -s 10 > /dev/null 2>\&1 \&}
+% \item Check if the following mounts are present. (Unless, execute on lxhadesdaq under 'root': \textbf{mount -a}).
+% \begin{itemize}
+% \item 192.168.100.11:/home/hadaq/oper -> lxhadesdaq:/home/hadaq/oper/oper\_1
+% \item 192.168.100.12:/home/hadaq/oper -> lxhadesdaq:/home/hadaq/oper/oper\_2
+% \item 192.168.100.13:/home/hadaq/oper -> lxhadesdaq:/home/hadaq/oper/oper\_3
+% \item 192.168.100.14:/home/hadaq/oper -> lxhadesdaq:/home/hadaq/oper/oper\_4
+% \end{itemize}
+% \item Check connection to the Data Movers for the data archiving:
+% \begin{itemize}
+% \item Ask Horst Goeringer to create a new archive for the production beam time.
+% \item First set a test path in the hadestest archive in lxhadesdaq:\~/trbsoft/daq/evtbuild/eb.conf: \textbf{RFIO\_PATH: rfiodaq:gstore:/hadaqtest/test001}
+% \item \textbf{lxhadesdaq:\~/trbsoft/daq/evtbuild/start\_eb\_gbe.pl -e start -n 1-16 --rfio on}
+% \item On MEDM GUI you have to see that all 16 EB processes got connected to 8 Data Movers (two connections per Data Mover). In the log file you have to see the message \textbf{rfio\_fopen: opened connection to Data Mover: slxdm17} from each Event Builder. If there are some error messages related to RFIO and Data Movers or there are more than two connections to the same Data Mover, try to contact Horst Goeringer.
+% \item Change the archive name and path (RFIO\_PATH:) in lxhadesdaq:\~/trbsoft/daq/evtbuild/eb.conf to the new ones.
+% \end{itemize}
+% \end{itemize}
+% \item Start Perl scripts to insert data to Oracle DB (pay attention to the data base name inside the scripts: \$database):
+% \begin{itemize}
+% \item Update Oracle DB with new subevt IDs and new boards (see subsection~\ref{ora_before_running_scripts}).
+% \item Start lxhadesdaq:~/trbsoft/daq/oracle/daq2ora\_client.pl (see subsection~\ref{ora_boards_script}).
+% \item Start lxhadesdaq:/home/hadaq/oper/runinfo2ora.pl (see subsection~\ref{ora_run_script}).
+% \end{itemize}
+% \end{itemize}
+
\subsection{daq\_evtbuild options}
daq\_evtbuild can be executed with the following options:
\end{itemize}
\subsection{Event Building control by EPICS IOC}
-
+\label{evtbuild:Before2018:EPICS}
Off-line analysis places the following demands:
\begin{itemize}
\item Synchronization of hld files: all EB processes should open and close hld files at the same time (jitter of a couple of seconds is allowed).
To stop IOCs the '\textbf{start\_eb\_gbe.pl -i stop -n 3-6}' script finds all running IOCs (subroutine findRunningIOC()), creates an expect script /tmp/ioc\_exit.exp and executes this script with two arguments (IP of the EB server where IOC runs and IOC screen name) for each IOC process indicated by \textbf{-n 3-6} argument (from 3 to 6). The /tmp/ioc\_exit.exp then logs on EB server, connects to the corresponding screen session and exits the IOC.
\subsection{Event Building interface to RFIO server}
-
+\label{evtbuild:Before2018:RFIO}
HADES Event Builders will write the full data to Tape via Data Movers. Some fraction of data we want also to write to Lustre. The latter feature can be integrated into the RFIO server on Data Movers. The following three parameters should be passed to gstore:
\begin{itemize}
\end{itemize}
\end{itemize}
+
+
+
+
\section{Storing data to Oracle database}
\begin{itemize}
The script can be executed on lxhadesdaq with all the files from 16 EBs:
/home/hadaq/oper/runinfo2ora.pl -f /home/hadaq/oper/oper\_1/eb\_runinfo2ora\_1.txt \\ -f /home/hadaq/oper/oper\_2/eb\_runinfo2ora\_2.txt -f /home/hadaq/oper/oper\_3/eb\_runinfo2ora\_3.txt -f /home/hadaq/oper/oper\_4/eb\_runinfo2ora\_4.txt -f /home/hadaq/oper/oper\_1/eb\_runinfo2ora\_5.txt -f /home/hadaq/oper/oper\_2/eb\_runinfo2ora\_6.txt -f /home/hadaq/oper/oper\_3/eb\_runinfo2ora\_7.txt -f /home/hadaq/oper/oper\_4/eb\_runinfo2ora\_8.txt -f /home/hadaq/oper/oper\_1/eb\_runinfo2ora\_9.txt -f /home/hadaq/oper/oper\_2/eb\_runinfo2ora\_10.txt -f /home/hadaq/oper/oper\_3/eb\_runinfo2ora\_11.txt -f /home/hadaq/oper/oper\_4/eb\_runinfo2ora\_12.txt -f /home/hadaq/oper/oper\_4/eb\_runinfo2ora\_13.txt -f /home/hadaq/oper/oper\_2/eb\_runinfo2ora\_14.txt -f /home/hadaq/oper/oper\_3/eb\_runinfo2ora\_15.txt -f /home/hadaq/oper/oper\_1/eb\_runinfo2ora\_16.txt
-\subsection{Archiving data from EB disks to tape}
+\section{Archiving data from EB disks to tape}
\label{eb_data_tape}
-
+\subsection{Old scripts based on gstore tool (DEPRECATED!)}
There are two scripts which can be used to archive the data from EB disks to tape.
These scripts can also be checked out from CVS:
\end{itemize}
\end{itemize}
+\subsection{New scripts using ltmsc}
+{\bf This section is under construction!}
+{\em here we need at least description of ltsmc command line tool}
+
+
+
+
+
+
-\section{Eventbuilder software for DABC}
+\section{Generic trbnet event builder software with DABC}
\subsection{Introduction}
The future application for trbnet hardware for FAIR detector test beams (e.g. CBM, PANDA)
The complete DABC framework is available at
\href{https://subversion.gsi.de/dabc/trunk/}{https://subversion.gsi.de/dabc/trunk/}
-This contains the actual hadaq plugin at
+This contains at
\href{https://subversion.gsi.de/dabc/trunk/plugins/hadaq}{https://subversion.gsi.de/dabc/trunk/plugins/hadaq}
+the actual hadaq plugin.
After checking out from repository to source installation directory:
\item [HldOutput]- write hadtu format to file
- \item [UdpDataSocket] - receive hadtu packets from trb frontend (single stream). This provides functionality of old daq\_netmem program.
+ \item [UdpTransport] - receive hadtu packets from trb frontend (single stream). This provides functionality of old daq\_netmem program.
- \item [MbsTransmitterModule] - wrap hadtu format into MBS event container output to deliver it to MBS server or lmd file.
-this format is used in mbs subevent combiner and go4 monitoring unpacker
+% \item [MbsTransmitterModule] - wrap hadtu format into MBS event container output to deliver it to MBS server or lmd file.
+% this format is used in mbs subevent combiner and go4 monitoring unpacker
\item [CombinerModule] - do eventbuilding of hadtu format from several data streams.
Output is full event in hadtu format. Should provide functionality of old daq\_evtbuild in the end.
generated in the top directory of the framework installation at the build.
Some example configurations are delivered with the plugin in subfolder {\em plugins/hadaq/app}:
-\begin{description}
- \item [HldConverter.xml] - configuration to read hld file and write the hadtu events wrapped in mbs events to lmd file
- \item [HldServer.xml] - configuration to read hld file and serve the hadtu events as mbs events at transport server dabc will wait 30 seconds before start processing, so go4 may connect to inspect data
- \item [NetmemServer.xml] - configuration to receive hadtu buffers at udp socket and serve them as mbs events at transport/stream server
+\begin{description}
\item [EventBuilder.xml] -configuration to receive hadtu data at n udp sockets, and do a full eventbuilding. Output events can be served via mbs transmitter at transport/stream server. In parallel, output events can be written to hld file. May enable export of variables via shared mem to hades EPICS-based eventbuilder control system
- \item [EventBuilderDim.xml] -as {\em EventBuilder.xml}, but shows how to activate additionally DIM control system export.
+ \item [EventBuilderStream.xml] -as {\em EventBuilder.xml}, but sets up additional online monitoring analysis with the {\em stream} framework. Uses the {\em first.C} analysis.
+ \item [TdcEventBuilder.xml] - example how HADAQ event building with TDC calibration could be configured in DABC
+ \item [master.xml] - This is example of master node configuration which should provide http access to the connected clients
\end{description}
-\subsubsection{Single trbnet input served to online monitor}
-This will read from specified port and transfer data to mbs stream server. This can be evaluated by Go4 online monitor with matching unpacker
-(see section \ref{go4:for:dabc}).
-
-Start with \verb!dabc_run NetmemServer.xml!
+% \subsubsection{Single trbnet input served to online monitor}
+% This will read from specified port and transfer data to mbs stream server. This can be evaluated by Go4 online monitor with matching unpacker
+% (see section \ref{go4:for:dabc}).
+%
+% Start with \verb!dabc_run NetmemServer.xml!
\subsubsection{Eventbuilder on single node}
\subsubsection{Eventbuilders on multiple nodes}
Since the DABC process emulates the control via the standard EPICS {\em ebctrl} software (see Fig.\ref{fig:dabc:hadeb}),
-it would be possible to run multiple event building servers using the DABC software.
-However, the current startup procedure with scripts {\em startup.pl} etc.
-needs to be adjusted for this concerning the parameters. DABC needs the set up to be delivered as xml files, so the startup script would need
-to deliver this format before invoking the \verb!dabc_run!.
-
-Moreover, because DABC provides own script mechanisms to launch configured processes on different nodes, the HADES eventbuilder start would
-need to use these in case it ran with DABC implementations.
-Currently there were no investigations done in this directions, although one could think to replace the old eventbuilder software
-by a DABC solution in the future. Note that export of run information to Oracle data base is also still not implemented for the
-DABC plug-in.
-
-
-\subsubsection{Combination with other DAQ systems}
-This application is interesting in test beams for FAIR experiments that investigate the usage of HADES trb hardware for future
-detectors. Here different types of frontends should be combined. The data streams of different partially free running frontendes are synchronized by
-a sync counter ``message`` that is received by the CTS. For DABC, all inputs are wrapped in MBS subevent containers. The combination is done
-with existing MBS combiner module in dabc, regarding the position of the sync word in the CTS data payload.
-Figure \ref{fig:dabc:cbmeb} shows a typical set up as used for CBM test beam at CERN/PS in November 2012.
-Using the MBS event as generic containers and a sync counter in all data streams, the DABC Supercombiner builds corresponding data packets and
-stores them in a common file. Furthermore, the MBS containers allow simple online monitoring at ''streamserver`` sockets by tools like Go4.
-
-
-\begin{figure}
- \centering
- \includegraphics[width=0.7\textwidth]{DABC-EB.png}
- \caption[DABC combining hadaq and other DAQ systems]{DABC components configured for combining
-trbnet udp data with other DAQ systems. Such setup was used for CBM test beam at CERN/PS in November 2012.
-} \label{fig:dabc:cbmeb}
-\end{figure}
+it be possible to run multiple event building servers using the DABC software. This has been implemented in 2014 for the HADES experiment, replacing
+the previous event building software.
+In 2018, the HADES eventbuilding topology has bee changed for a DABC builder network (BNET). This is described in section \ref{evtbuild:New2018}.
+
+
+% However, the current startup procedure with scripts {\em startup.pl} etc.
+% needs to be adjusted for this concerning the parameters. DABC needs the set up to be delivered as xml files, so the startup script would need
+% to deliver this format before invoking the \verb!dabc_run!.
+%
+% Moreover, because DABC provides own script mechanisms to launch configured processes on different nodes, the HADES eventbuilder start would
+% need to use these in case it ran with DABC implementations.
+% Currently there were no investigations done in this directions, although one could think to replace the old eventbuilder software
+% by a DABC solution in the future. Note that export of run information to Oracle data base is also still not implemented for the
+% DABC plug-in.
+
+
+% \subsubsection{Combination with other DAQ systems}
+% This application is interesting in test beams for FAIR experiments that investigate the usage of HADES trb hardware for future
+% detectors. Here different types of frontends should be combined. The data streams of different partially free running frontendes are synchronized by
+% a sync counter ``message`` that is received by the CTS. For DABC, all inputs are wrapped in MBS subevent containers. The combination is done
+% with existing MBS combiner module in dabc, regarding the position of the sync word in the CTS data payload.
+% Figure \ref{fig:dabc:cbmeb} shows a typical set up as used for CBM test beam at CERN/PS in November 2012.
+% Using the MBS event as generic containers and a sync counter in all data streams, the DABC Supercombiner builds corresponding data packets and
+% stores them in a common file. Furthermore, the MBS containers allow simple online monitoring at ''streamserver`` sockets by tools like Go4.
+%
+%
+% \begin{figure}
+% \centering
+% \includegraphics[width=0.7\textwidth]{DABC-EB.png}
+% \caption[DABC combining hadaq and other DAQ systems]{DABC components configured for combining
+% trbnet udp data with other DAQ systems. Such setup was used for CBM test beam at CERN/PS in November 2012.
+% } \label{fig:dabc:cbmeb}
+% \end{figure}
-\subsection{Go4 online monitoring for DABC with hadaq plug-in}
+\subsection{Go4 online monitoring for DABC with stream framework}
\label{go4:for:dabc}
Several analysis codes for the analysis framework \href{http:go4.gsi.de}{Go4} exist that contain unpackers for the
-hadaq data format and specialized code for trb3 frontends.
+hadaq data format and specialized code for trb3 frontends. These are implemented by means of the {\em stream} framework as described in
+\href{https://subversion.gsi.de/go4/app/stream/Readme.md}{https://subversion.gsi.de/go4/app/stream/Readme.md}.
-The complete Go4 framework is available at\\
-\href{https://subversion.gsi.de/go4/trunk/}{https://subversion.gsi.de/go4/trunk/}
+The code implemented in this framework may run in different environments:
+\begin{itemize}
+ \item In the go4 framework as available at \href{https://subversion.gsi.de/go4/trunk/}{https://subversion.gsi.de/go4/trunk/}
+ \item in a dabc application
+ \item as part of the hydra analysis
+\end{itemize}
+At
+\href{https://web-docs.gsi.de/~dabc/doc/dabc2/hadaq\_trb3\_package.html}{https://web-docs.gsi.de/\tilde{~}dabc/doc/dabc2/hadaq\_trb3\_package.html}.
+a further description can be found.
+The subversion repository \href{https://subversion.gsi.de/dabc/trb3/}{https://subversion.gsi.de/dabc/trb3/}
+contains all ROOT, Go4 and DABC software packages that are required for running a standalone DAQ system with online monitoring analysis.
-The hadaq skeleton (actually used for PANDA-DIRC) is at \\
-\href{https://subversion.gsi.de/go4/app/hadaq}{https://subversion.gsi.de/go4/app/hadaq}
-An more advanced analysis for CBM test beams can be found at \\
-\href{https://subversion.gsi.de/cbm/ROC/trunk/onlinemonitor/trbmonitor}{https://subversion.gsi.de/cbm/ROC/trunk/onlinemonitor/trbmonitor}.
-Note that the latter currently requires the full trunk to be checked out and the Makefile to be called in the top directory.
+% The hadaq skeleton (actually used for PANDA-DIRC) is at \\
+% \href{https://subversion.gsi.de/go4/app/hadaq}{https://subversion.gsi.de/go4/app/hadaq}
+%
+% An more advanced analysis for CBM test beams can be found at \\
+% \href{https://subversion.gsi.de/cbm/ROC/trunk/onlinemonitor/trbmonitor}{https://subversion.gsi.de/cbm/ROC/trunk/onlinemonitor/trbmonitor}.
+% Note that the latter currently requires the full trunk to be checked out and the Makefile to be called in the top directory.
-\subsection{Further documentation}
- This presentation for the GSI Experiment-Electronics Department meeting gives an overview of the software status in July 2012: \\
-\href{http://hades-wiki.gsi.de/pub/DaqSlowControl/EventBuilderDabc/DABC-TRB-EEmeeting.pdf}{http://hades-wiki.gsi.de/pub/DaqSlowControl/EventBuilderDabc/DABC-TRB-EEmeeting.pdf}
-An article for the GSI annual report 2012 is under preparation.
+% \subsection{Further documentation}
+% This presentation for the GSI Experiment-Electronics Department meeting gives an overview of the software status in July 2012: \\
+% \href{http://hades-wiki.gsi.de/pub/DaqSlowControl/EventBuilderDabc/DABC-TRB-EEmeeting.pdf}{http://hades-wiki.gsi.de/pub/DaqSlowControl/EventBuilderDabc/DABC-TRB-EEmeeting.pdf}
+%
+% An article for the GSI annual report 2012 is under preparation.
jspc29 / daqdocu.git / commitdiff
