-\section{DAQ control window}
\label{section:daqcontrolgui}
Most control tasks can be done using the DAQ Control GUI, only few things need access to other tools.
\label{fig:daqcontrol}
\end{figure}
-\subsection{DAQ-Operator}
+\subsection*{DAQ-Operator}
\begin{description}
\item[Start DAQ] The main script to restart the DAQ. Standard solution if some problems appear (red blinking).
\item[Start Stop Trigger] Interrupt sending triggers to stop data taking for a while
The tools marked with (1) can only be used succesfully, if all boards are responding and are not reported as missing. If there are missing boards, first try a DAQ restart and then the reboot.
-\subsection{Monitor Tools}
+\subsection*{Monitor Tools}
\begin{description}
\item[CTS Mon] The monitor for CTS settings and scalers. One instance should be open for experts, but usually not needed by the operator
\item[Hmon] Opens a browser pointed at \url{https://hades33/mon}
\item[Hmon Snapshot] Creates a copy of all plots and windows in Hmon for documentation
\end{description}
-\subsection{Eventbuilder Tools}
+\subsection*{Eventbuilder Tools}
\begin{description}
\item[BEAM files] Start recording files triggered with accelerator beam. These appear with
prefix ''BE'' in eventbuilder monitor and are accounted in the runtime database (logbook).
\end{description}
- \subsection{Settings}
+\subsection*{Settings}
\begin{description}
\item[BLR Reinit FPGA] Reconfigures the analog trigger collection board from TOF / RPC. Use when
\end{description}
-\subsection{Expert Tools}
+\subsection*{Expert Tools}
{\it This section is visible only when the ''Show Expert Tools'' box on the upper left corner is checked.}
\begin{description}
\end{description}
-\subsection{Power Tools}
+\subsection*{Power Tools}
This section offers buttons to cycle the low voltage supplies of several detector systems. These are convenience
shortcuts to avoid operations on the full detector control system GUI.
A power cycle may be necessary in some situations
--- /dev/null
+Detailed data and tables are available from using Daqtools (\url{http://hades33/daqtools}).
+These pages make direct accesses to the DAQ, so that a few access rules are necessary:
+\begin{itemize}
+ \item Close windows you don't use
+ \item Set the update rate to a reasonable value (Settings don't change every second!)
+ \item Always have ``Use Cache'' activated
+ \item Be careful. All changes to registers have a direct effect on the system
+\end{itemize}
-\section{Eventbuilder}
+\section{Eventbuilder Controls}
\label{sec:eboperatorguide}
-
-
-
-\subsection{DABC Webserver controls}
-\label{sec:ebmonguide:dabc}
-The BNET master server of offers a monitoring and control GUI on a webserver at the address \url{http://lxhadeb07:8099/?browser=fix}.
+The BNET master server of offers a monitoring and control GUI on a webserver at the address
+\url{http://lxhadeb07:8099/?browser=fix}.
It is also available as part of the hmon webserver at
\url{http://hades33/eb/}. When this address is opened in any web browser (e.g. firefox, chrome), the default GUI displays the state of the BNET as shown in figure \ref{fig:ebmonitorguide:dabc1}.
This screenshot explains the main graphical elements and the functionality of the most important buttons. Besides the control buttons,
writing to files is stopped. In this case, just the filename display will become empty and the builder node color turns yellow.
Clicking on any of the rate and file size numbers allows to watch the trending graph of the assigned value in the display area.
-\newpage
-
-
-\clearpage
% \end{figure}
-\clearpage
-%
-\subsection{Eventbuilder monitoring in hmon}
-This main hmon monitoring page offers a summary of most relevant eventbuilder status under the ''EBSummary'' section at the address
-\url{http://hades33/mon/monitor.cgi?2-EBSummary}.
-Figure \ref{fig:evtbuild:ebsummary} shows the main components of it
-
-
-
-
-\subsection{Regular beamtime control}
+
During beamtime shift, data taking can be controlled by the main DAQ Operator GUI (Fig.
\ref{fig:daqcontrol}).
The GUI box {\bf Eventbuilder Tools} offers all relevant commands as shortcut buttons.
-\subsubsection{Restart with different number of builder nodes}
+\subsection*{Restart with different number of builder nodes}
Depending on the data rate delivered from the DAQ, it may be necessary to adjust the number of
builder nodes, i.e. parallel files written.
For cosmics usually only 1 builder is enough, for full beam operation it may be 12 builders.
-\subsubsection{Restart of BNET master control process}
+\subsection*{Restart of BNET master control process}
The master control process handles the run synchronization and overall monitoring of all
nodes of the event buiding network (BNET).
{\bf It is necessary to restart the BNET master whenever the setup of the server processes have
GUI (see section
\ref{section:daqcontrolgui})
-\subsection{TDC calibration}
+\subsection*{TDC calibration}
The calibration of the trb3 TDC finetime counters is currently handled in the eventbuilder input
nodes with the DABC software.
The eventbuilder software has to aquire calibration information for each trb3 channel with special
-\section{Addressing scheme}
-
-\begin{table}[hb]
+\begin{table}[H]
\begin{center}
\begin{tabularx}{\textwidth}{l|l|X}
\textbf{Address(es)} & \textbf{Board(s)} & \textbf{Description} \\
-
-\section{Power Switches and Supply control}
\label{powersupplies}
There are several interfaces for control, depending on the supply.
\begin{itemize}
--- /dev/null
+
+\section{Daily News}
+
+\textbf{2019/03/25 - this page will be updated during the beamtime}
+\begin{itemize*}
+\item If one board is missing, do not panic! Do not risk many seconds of possible data taking for
+one missing board. Up to 2 (MDC endpoints missing or MBO w/o data) and up to 3 RICH
+front-ends is acceptable - don't restart the DAQ more than once per 30 minutes if only few boards
+are missing. But don’t run for too long with too many missing boards. The missing MDC endpoints
+should not be in the same chamber (same first 3 digits)
+ \item MDC 0x2057 is off and can’t be recovered by the operator
+\end{itemize*}
+
+
+\section{General Remarks}
+\begin{itemize*}
+\item The DAQ operator should always be present in the \textbf{Mumble channel} ``Operators Corner''
+- DAQi will provide some audible (sometimes even recognizable) hints if something is wrong.
+
+\item \textbf{If one board is missing, do not panic!} Do not risk many seconds of possible data
+taking for one missing board. As a rough guide 2 MDC and/or 2 RICH boards missing is acceptable. If
+a board goes missing after an hour of data taking, you can restart the DAQ, but don't do it too
+often in case of frequent failures.
+
+\item Don't forget to monitor the \textbf{QA plots} to see whether data quality is ok.
+
+\item If you notice something noteworthy in the Hmon monitoring, use the \textbf{``Hmon Snapshot''}
+button to store the current plots for later analysis.
+
+\item \textbf{Before making changes} to configuration or doing a power cycle, don't forget to stop
+triggering (Start Stop Trigger) to prevent corrupted data from being recorded.
+
+\item Whenever there is a \textbf{beam pause} of more than 2-3 minutes, switch eventbuilders to "No
+Files”. When beam comes back, check in the EB Summary monitor that all building nodes are
+still active after switching back to "beam" files.
+
+\item Please don’t do any power cycle without asking an expert – unless it is explicitly stated
+in the manual or help texts.
+
+\item Don’t hesitate to \textbf{call the expert} if there is a problem you can’t solve or you don't
+fully understand!
+
+\item At least \textbf{once a day}, when there is a break >15 minutes, a full restart of the system
+should be performed. Talk to the DAQ expert to get this done!
+
+\item Almost \textbf{everything can be accessed} from the web page (\url{http://hades33/mon}, open
+in Firefox, preferably using an aptly named profile) and the Control GUI (icon on desktop)
+
+\item Make sure you have access to everything well before your shift (e.g. set up tunnels for
+remote work ->
+\href{https://jspc29.x-matter.uni-frankfurt.de/xmatterpad/p/HADES_GSI_Network_Access}{Manual})
+
+\end{itemize*}
bib_extensions=.bib
bibliographyBackendAutoDetected=
bibliographyBackendUserOverride=
-def_graphic_ext=eps
+def_graphic_ext=png
img_extIsRegExp=false
img_extensions=.eps .jpg .jpeg .png .pdf .ps .fig .gif
kileprversion=3
MakeIndex=
QuickBuild=
+[item:daqcontrol.tex]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:eboperatorguide.tex]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:ebstartup.tex]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:expert_networkaddresses.tex]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:expert_powersupplies.tex]
+archive=true
+encoding=
+highlight=
+mode=
+
[item:hadesoperator.kilepr]
archive=true
encoding=
highlight=
mode=
+
+[item:hmon.tex]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:main.aux]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:main.log]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:main.out]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:main.pdf]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:main.synctex.gz]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:main.tex]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:main.toc]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:powercycles.tex]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:tacticaloverview.tex]
+archive=true
+encoding=
+highlight=
+mode=
+
+[item:tdccalibration.tex]
+archive=true
+encoding=
+highlight=
+mode=
The most important windows to be open at all times are the "Tactical Overview" and the ones listed in the "Operator Monitor" box. They should be arranged on the upper left screens so that they can be seen from everywhere in the counting house.
The description of the plots shown and the features to look out for are listed below.
+
+\begin{figure}[htp]
+ \centering
+ \makebox[\textwidth][c]{\includegraphics[width=0.9\paperwidth]{figures/Hmon.pdf}}
+ \caption{Main Hmon Windows (without EB monitor and Tactical Overview)}
+ \label{fig:hmonwindows}
+\end{figure}
+
+
+
+% remarks: tuning procedure
+% insert pin header, to test mainboard and sensor individually
+\subsection*{Busy times}
+Several plots show the busy time of the system, either as bar graph with the current value, or as
+histogram with the history of the last minutes.
+
+Busy time (dead time) is the time where a detector system is processing/transporting the recorded
+data and cannot accept another data acquisition trigger.
+
+The relative dead time of sub systems should stay roughly constant, and the total amount should be
+within useful limits.
+
+\subsection*{Hit Rate Plots}
+A set of rate plots in heatmap style show the rates of each channel in each subdetector (RICH has a
+separate display not in the main operator windows). Each box corresponds to one channel, order is
+by nework addresses and channel numbers, not by physical location, although there can be a
+correlation.
+
+The color axis should be limited, i.e. too high rates are usually cut. Rates are always
+given in Hz per channel.
+
+For Start, Hodo and iTOF there is a second window with positional maps that can be used to
+determine the position of the beam on these detectors.
+
+\subsection*{Eventrates}
+The rate of recorded events is shown in two plots over different time windows. These can be used to
+judge the rate of data taking, but should not be used to make conclusions about the beam shape.
+
+
+\subsection*{Counts per Spill}
+Two plots show the number of events per spill and number of hits in Start per spill. These are
+updated once per spill and can be used to judge the beam intensity and uniformity of spills.
+
+\subsection*{Trigger accepted over total}
+There is a plot showing the ratio of actual events and the number of events that where triggered.
+This gives a reasonable approximation for the effective dead time of the detector and the
+efficiency of data taking.
+
+\subsection*{EB (Event Builder) Summary Window}
+\begin{figure}
+ \centering
+ \includegraphics[width=0.8\textwidth]{figures/EBSummary.png}
+ \caption[Eventbuilder Summary window in hmon]{Eventbuilder Summary window in hmon}
+ \label{fig:evtbuild:ebsummary}
+\end{figure}
+
+
+\begin{description}
+\item[Description:]
+Overview over Event Builder Servers. The event building servers (computers in the rear counting
+house)
+serve two purposes:\\
+\textbf{1.} Receiving data streams from the read-out boards and (input node)\\
+\textbf{2.} Combining the received data belonging to the same event and writing it to disc (building
+node)\\\\
+The input nodes and the building nodes are software processes running on the same computer(s).
+\item[What to check:]
+The total data rate and the total event rate should be approximately the same in the input node and
+building
+node table (there is a delay of several seconds).
+If this is not the case, the total data/event rate field in the building notes table will turn red.
+Every of the \emph{n} input nodes gets approximately 1/n of the total data rate produced by HADES.
+If the load is not balanced, the respective field in the table turns red.
+The \emph{dropped events} rate should be close to zero.
+\item[Calib info:]
+The top right box in this window shows the timestamp of the last TDC calibration and if it was
+successful or not.
+\end{description}
+
+Next to the EB status is a list of the latest written data files.
+The BNET master node offers a monitoring and control GUI on a webserver at the address
+\url{http://hades33/eb/}.
+This interface is usually not needed for the DAQ operator, but gives additional ways to debug error
+situations. The necessary information about event builder status can be found in the main
+monitoring
+page under "EBSummary".
+% übersicht über Eventbuilder server
+
+% oben Zusammenfassung
+
+
+% Jeder der input-nodes kriegt ein 1/5 von jedem Event.
+%für events pro sekunde ist das Average. Daten-Summe ist wirklich daten summe über die INput nodes.
+
+% die dropped events sollte nahe Null sein.
+
+% Events/s wird rot, wenn rate in einem input node zu sehr abweicht.
+
+% Wenn irgendwas nicht grün ist, sollte der Mouseover-Text erklären was schief läuft.
+
+% building nodes sum: Event rate sum wird rot, wenn die Zahl der empfangenen Events nicht der Anzahl
+%der Gebauten events entpspricht.
+
+% Wir sind zufrieden wenn data rate und event rate von input and output ungefähr gleich ist.
+
+% es gibt für gewöhnlich einen Delay zw. input und build notes von ein paar Sekunden.
+
+% obere drei kästchen ist im wesentlichen Zusammenfassung.
+
+% rechts oben steht noch ob Kalibrierung in Ordnung ist.
+
+% rechte box dateinamen von geschriebenen Dateien.
+
+
+
+
+
+
\documentclass[11pt,a4paper,twoside]{scrartcl} %twoside
%Einstellungen der Seitenr?nder
-\usepackage[left=3.5cm,right=3cm,top=2.5cm,bottom=2.5cm,includeheadfoot]{geometry}
+\usepackage[left=3.5cm,right=2cm,top=2cm,bottom=2cm,includeheadfoot]{geometry}
\input{../trb3/trb3qs_preamble}
\tableofcontents
\cleardoublepage
-\part{General Advice}
-
-
-\begin{itemize*}
-\item \textbf{Please read the additional page with notes for the DAQ Operator.}
-
-\item The DAQ operator should always be present in the Mumble channel ``Operators Corner'' - DAQi
-will provide some audible (sometimes even recognizable) hints if something is wrong.
-
-\item If one board is missing, do not panic! Do not risk many seconds of possible data taking for
-one missing board. As a rough guide 1 MDC and/or 2 RICH boards missing is acceptable. If a board
-goes missing after an hour of data taking, you can restart the DAQ, but don't do it too often in
-case of frequent failures.
-
-\item If the high voltage on some detectors is not at nominal values, it's perfectly normal to get
-some warnings for this subsystem, like boards that don't deliver any data. Always check with the
-detector responsible to see whether this is expected behavior or an error in the DAQ.
-
-\item Don't forget to monitor the QA windows to see whether data quality is ok.
-
-\item If you notice something noteworthy in the Hmon monitoring, use the ``Hmon Snapshot'' button
-to store the current plots for later analysis.
-
-\end{itemize*}
+\part{General Advice} \input{generalremarks}
\clearpage
\part{Control}
-\input{daqcontrol}
+\section{DAQ control window} \input{daqcontrol}
\clearpage
\part{Monitoring}
-\section{Tactical Overview}
-\input{tacticaloverview}
+\section{Tactical Overview} \input{tacticaloverview}
\clearpage
-\section{Hmon Plots}
-\input{hmon}
-\input{monitors}
-\clearpage
-\section{Eventbuilder}
-\input{ebmonitoring}
+\section{Hmon Plots} \input{hmon}
+\section{Daqtools} \input{daqtools}
-\clearpage
-\part{Handling Procedures}
\clearpage
-\section{Power Cycles}
-\input{powercycles}
+\part{Handling Procedures}
+\section{Power Cycles} \input{powercycles}
\clearpage
-\section{Eventbuilder Control}
-\input{ebstartup}
+\section{Eventbuilder Control} \input{ebstartup}
% \clearpage
\clearpage
\part{Expert Section}
+\section{Addressing scheme} \input{expert_networkaddresses}
\clearpage
-\input{expert_networkaddresses}
+\section{Power Supply control} \input{expert_powersupplies}
\clearpage
-\input{expert_powersupplies}
-
+ \input{tdccalibration}
\clearpage
-\input{tdccalibration}
-
-
+ \input{eboperatorguide}
\clearpage
-\input{eboperatorguide}
-
+\section{Trigger Collection} \input{triggerbox}
+++ /dev/null
-% remarks: tuning procedure
-% insert pin header, to test mainboard and sensor individually
-\section*{Busy times (bar plot)}
-Place an example plot!
-
-\begin{description}
-\item[Y axis:]
-Busy time/dead time in \% of the total time
-\item[X axis:]
-HADES detector subsystem (one set of bars per subsystem)
-\item[Description:]
-Busy time (dead time) is the time where a detector system is processing/transporting the recorded data and
-cannot accept another data acquisition trigger.
-Each HADES detector subsystem has one blue bar showing the total busy time of the entire subsystem.
-The busy time is given in percent, i.e. the bar plot shows the "mean business" per unit time.
-Accompanying the main blue bar, there are several grey bars which show the individual busy times of the
-sub-subsystems (less important).
-\end{description}
-
-\section*{Busy times (line graph)}
-\begin{description}
-\item[Y axis:]
-Busy time/dead time in \% of the total time
-\item[X axis:]
-Time in seconds.
-\item[Description:]
-See \emph{Busy times (bar plot)}. Same observables graphed as a function of time.
-\end{description}
-
-\section*{Data rates}
-\begin{description}
-\item[Y axis:]
-Total data rate recorded by entire HADES experiment sent to eventbuilder in MB/s.
-\item[X axis:]
-Time in seconds.
-\end{description}
-
-\section*{Start-X/Start-Y/Veto Rate}
-\begin{description}
-\item[Y axis:]
-Particle hit rate on detector electrode in Hz.
-\item[X axis:]
-\# detector electrode
-\item[Description:]
-The Start detector consists of two layers of silicon strip detectors, located several centimeters upbeam of the target.
-Each layer features 16 strips with a width of $\approx\SI{300}{\micro\meter}$.
-One layer is oriented vertically (providing a beam intensity profile in X direction), while the second layer
-is oriented horizontally (providing a beam intensity profile in Y direction).
-During beam operation, the Start-X and the Start-Y plot should show a gaussian beam profile, which should be
-centered and not move.
-
-Bar plots are either red or green. Green means \emph{used for trigger} (usually inner/center electrodes),
-red means \emph{not used for trigger} (usually outer electrodes).
-
-The veto detector consists of four inner (channels 0-3) and four outer electrodes (channels 4-7).
-It detects (beam) particles that did not interact with the target.
-When the beam is perfectly centered, the only the inner four electrodes are hit equally.
-% start-x/start-y/veto
-
-% start detektor hat 16 streifen in x und 16 streifen in y, circa 300 µm pitch
-% in x und y hast du im Idealfall ein Gauß-Profil, was dir das Beam-Profil angibt. Sollte sich im Idealfall nicht bewegen.
-% von den 16 streifen können wir nur auf 8 streifen gleichzeitig triggern. (Limitierung von CTS)
-% zwei farben: grün bedeutet: benutzt für trigger, rot bedeutet nicht ausgewählt für Trigger. Möglichst viel Grün.
-% in y-Plot genauso.
-
-% Veto detector: innere vier Elektroden vs äußere vier. innere vier sind grün, äußere vier sind rot. Wenn die ersten vier elektroden feuern, dann liegt der Strahl gerade.
-\end{description}
-
-
-\section*{Start X counts per spill (line graph)}
-\begin{description}
-\item[Y axis:]
-Hits on start detector in Mega-Counts/spill
-\item[X axis:]
-Spill number. The last approx. 100 spills ($\sim$ time)
-\item[Description:]
-In normal beam operation this observable should stay mostly constant. If it fluctuates, the beam intensity/spill structure
-is not stable.
-\end{description}
-% Start X counts. wieviele counts auf dem Startdetektor habe ich per spill. y-Achse start counts (in Mega-counts). x-Achse : die letzten 100 Spills. Das sollte konstant bleiben. Wenn stark fluktuiert, dann nix gut.
-
-
-\section*{Recorded events per spill (line graph)}
-\begin{description}
-\item[Y axis:]
-Events recorded by DAQ per spill (in kilo-Counts/spill)
-\item[X axis:]
-Spill number. The last approx. 100 spills ($\sim$ time)
-\item[Description:]
-Events which were actually recorded by the DAQ.
-In normal beam operation this observable should stay mostly constant
-and be proportional to \emph{Start X counts per spill}.
-If the proportion changes, the micro spill structure has changed.
-\end{description}
-% recorded events per spill: Wieviel events sind wirklich aufgezeichnet worden? sollte mit dem vorigen zusammenhängen. wenn sich das Verhältnis zwischen beiden ändert, dann heißt das, etwas hat sich an der micro-spill-struktur geändert.
-
-\section*{Event rates (2x line graphs)}
-\begin{description}
-\item[Y axis:]
-Event rate - Events accepted by DAQ per time unit (in kHz)
-\item[X axis:]
-Time in seconds.
-\item[Description:]
-There are two versions of this plot. One shows the event rate over 10-30 sec, the other shows the event rate
-over 10 min.
-The short-time plot shows the time structure of a single spill.
-The long time plot shows the history over the last dozens of spills.
-The ideal spill should have steep slopes, a constant plateau, no peaks/spikes and a short spill pause.
-\end{description}
-% event rate: akzeptierente rate pro sekunde, einmal über 10 min, einmal über 10-30 sekunden (damit man Struktur von einem Spill sieht (spill shape) ) Der ideale spill ist ein Plateau mit steilen Kanten, kurzer Totzeit. Keine Peaks.
-
-% Jeder plot hat eine Uhrzeit, um zu checken ob sich das System aktualisiert.
-
-
-\section*{EB (Event Builder) Summary Window}
-\begin{figure}[H]
-\centering
-\includegraphics[width=0.5\textwidth]{figures/eb_summary.png}
-\caption{}
-\label{fig:eb_summary}
-%figure~\ref{fig:eb_summary}
-\end{figure}
-
-
-\begin{description}
-\item[Description:]
-Overview over Event Builder Servers. The event building servers (computers in the rear counting house)
-serve two purposes:\\
-\textbf{1.} Receiving data streams from the read-out boards and (input node)\\
-\textbf{2.} Combining the received data belonging to the same event and writing it to disc (building node)\\\\
-The input nodes and the building nodes are software processes running on the same computer(s).
-\item[What to check:]
-The total data rate and the total event rate should be approximately the same in the input node and building
-node table (there is a delay of several seconds).
-If this is not the case, the total data/event rate field in the building notes table will turn red.
-Every of the \emph{n} input nodes gets approximately 1/n of the total data rate produced by HADES.
-If the load is not balanced, the respective field in the table turns red.
-The \emph{dropped events} rate should be close to zero.
-\item[Calib info:]
-The top right box in this window shows the timestamp of the last TDC calibration and if it was successful or not.
-\end{description}
-
-Next to the EB status is a list of the latest written data files.
-
-% übersicht über Eventbuilder server
-
-% oben Zusammenfassung
-
-
-% Jeder der input-nodes kriegt ein 1/5 von jedem Event.
-%für events pro sekunde ist das Average. Daten-Summe ist wirklich daten summe über die INput nodes.
-
-% die dropped events sollte nahe Null sein.
-
-% Events/s wird rot, wenn rate in einem input node zu sehr abweicht.
-
-% Wenn irgendwas nicht grün ist, sollte der Mouseover-Text erklären was schief läuft.
-
-% building nodes sum: Event rate sum wird rot, wenn die Zahl der empfangenen Events nicht der Anzahl der Gebauten events entpspricht.
-
-% Wir sind zufrieden wenn data rate und event rate von input and output ungefähr gleich ist.
-
-% es gibt für gewöhnlich einen Delay zw. input und build notes von ein paar Sekunden.
-
-% obere drei kästchen ist im wesentlichen Zusammenfassung.
-
-% rechts oben steht noch ob Kalibrierung in Ordnung ist.
-
-% rechte box dateinamen von geschriebenen Dateien.
-
-
-
-
-
-
-\subsection{Summary window in hmon}
-The most important eventbuilder status values are collected in a EBSummary window
-(Fig. \ref{fig:evtbuild:ebsummary})
-that is available from the hmon ''Tactical Overview'' at
-\newline
-\url{http://hades33/mon/monitor.cgi?2-EBSummary}.
-This information is directly fetched from the
-controlling BNET master webserver and summarizes several details as provided by the full web control BNET\_GUI
-(Fig. \ref{fig:evtbuild:bnetgui})
-
-
-
\subsection{BNET web GUI}
The BNET master node offers a monitoring and control GUI on a webserver at the address
\url{http://hades33/eb/}.
an element will allow further inspection and trending plots.
Please see section \ref{sec:ebmonguide:dabc} for more details.
-\begin{figure}[htb]
- \centering
- \includegraphics[width=0.8\textwidth]{figures/EBSummary.png}
- \caption[Eventbuilder Summary window in hmon]{Eventbuilder Summary window in hmon}
- \label{fig:evtbuild:ebsummary}
-\end{figure}
-
\begin{figure}[htb]
\centering
\includegraphics[width=0.8\textwidth]{figures/BNET_GUI.png}
% \caption{Controls for main Power Supplies}
% \label{fig:powersupplycontrol}
% \end{figure}
-\subsection{MDC Chamber}
+\subsection*{MDC Chamber}
If a few MDC boards fail (with a "FEE error" or "MBO w/o data") and a DAQ restart doesn't help (or
is needed frequently), a power cycle of one MDC chamber can be performed: Read the address of the
failing board (e.g. 2045) from the Tactical Overview. Use the Icon "MDC LV Powercycle" and select
fails, first run a DAQ start-up until the script complains ``OEP design too old''. Close the window
and try ``Reboot OEP'' again.
-\subsection{Full MDC}
+\subsection*{Full MDC}
If it's unclear which part of MDC causes an error, or if there is a larger number of boards
failing, you can do a complete MDC power cycle using "EPICS Power" and "MDC Group" instead of using
the "MDC LV Powercycle".
-\subsection{RICH}
+\subsection*{RICH}
All boards of the RICH subsystem (addresses 7xxx and 82xx) are powered by the supplies in the "RICH
Group". Use the button "RICH Powercycle" to switch them off and on again. When all power supplies
are active again (see the list in ``EPICS Power'' -> ``RICH LV''), do a DAQ restart.
A power cycle will trigger the temperature interlock for RICH power supplies.
The 3.3V supply might be blocked by the temperature interlock - restart DAQ, then try to reset it.
-\subsection{Central Controls}
+\subsection*{Central Controls}
If a power cycle of the ``Central Controls'' was done, don't forget to ``Reprogram FPGA'' and
``BLR Reinit FPGA'' before the DAQ restart.
-\subsection{Other Subdetectors}
+\subsection*{Other Subdetectors}
Check section \ref{powersupplies} to see which power supply to switch.
-\subsection{Full Powercycle}
+\subsection*{Full Powercycle}
A full powercycle should only be done after contact with an expert. Follow the guide for the
``Daily Powercycle''.
+
+
+\section{Full Power Cycle / Daily DAQ reboot}
+This should be performed at least once a day, when there is a break of ~ 15 minutes and in presence
+of the DAQ expert: {\bf To be updated for 2022!}
+\begin{enumerate}
+\item MDC power supply group off (EPICS)
+\item RICH power supply group power cycle (EPICS) \\
+ Main power supply group power cycle (EPICS)
+\item ECAL Padiwa Powercycle \\
+ Start Powercycle \\
+ CentralControls Powercycle \\
+ iTof Powercycle \\
+ Hodo Powercycle
+\item Reprogram FPGA \\
+ BLR Reload Settings
+\item Start DAQ (x2, because of MDC Hubs)
+\item MDC power supply group on (EPICS)
+ MDC LV Turn On All Relais
+\item Reboot OEP
+\item Start DAQ
+
+\end{enumerate}
+
+
+\begin{figure}[htp]
+ \centering
+
+\includegraphics[width=0.6\textwidth]{figures/MonitorTacticalOverview.png}
+ \caption{Tactical Overview}
+ \label{fig:tacticaloverview}
+\end{figure}
+
+The Tactical Overview gives a comprehensive view of the status of the DAQ system.
+During normal operation all fields should be green indicating a normal status. Colors change as
+errors get more severe (green - yellow - orange - red - red blinking).
+
+Hovering the mouse over a field shows more details, and a click opens a window with some further
+information and troubleshooting advice.
+
+Please note:
+\begin{itemize}
+ \item Under certain conditions (e.g. one subsystem not working perfectly, no beam or cosmics data
+taking) not everything will be green. Try to memorize the pattern to find out if anything changes
+during the run.
+
+ \item Not all colors are working perfectly. Some vary between error and good states - this can be
+either acceptable or not. Check the documentation of each field to see which applies.
+
+ \item Note the special colors: black - unsued field. white - failure of script. olive -
+acknowledged non-perfect condition.
+
+ \item Often several errors appear at once - you'll have to find out what is the cause and what is
+just a result. Some help texts tell you to ``look elsewhere''. Eventbuilder problems can't be fixed
+while there is a DAQ problem.
+
+\end{itemize}
-\section{TDC Calibration procedure}
-The calibration needs to be done by an expert, because the output of the Eventbuilders needs to be
-checked. This might take a while and therefore should only be done during planned breaks, e.g.
-periods without beam.
+\section{TDC Calibration procedure} The calibration needs to be done by an expert, because the
+output of the Eventbuilders needs to be checked. This might take a while and therefore should only
+be done during planned breaks, e.g. periods without beam.
The calibration procedure is as follows:
-\begin{enumerate}
- \item Be sure that the beam is off and no other data source may interfere with the calibration pulser!
-
- \item Click the Button {\bf Start TDC Cal} in the DAQ operator GUI. This will disable all inputs of the CTS and put it into the
- special calibration mode (trigger type D, 500~Hz pulser). The eventbuilders will start writing files with prefix ''TC''.
-
-
- \item Observe the eventbuilder BNET GUI (Fig. \ref{fig:evtbuild:bnetgui}). The ''Input nodes'' box shows for each eventbuilder server
- the subsystems (''HUBs'') which send data to them from the cave. These are labeled with their trbnet hub address, e.g. ''0x83c1''.
- During calibration, the trb3 systems will turn to a blue color. The ''Run control'' caption of the webpage will show the state ''Calibrating''.
-
- \item When all of the input hubs in the BNET GUI have turned to green color, the calibration statistics is sufficient.
- In this state the ''Run control'' state will show ''Ready''. It can take several minutes until this is reached, depending
- on the configured calibration precision in the setup.
-
-
- \item If BNET GUI (or corresponding hmon display) shows that calibration is OK, stop the calibration procedure by
- pressing the {\bf Stop TDC Cal} button in the DAQ control GUI ({\bf do not stop the run with the ''NO FILE'' button,
- and do not start immediately recording any other files types by the corresponding control GUI buttons}).
- The eventbuilders will close the calibration run files and will archive the most recent calibration information.
- The trigger settings of the CTS will be restored to the values just before the calibration procedure.
-
- \item If BNET GUI should show that calibration is not successful after 5 minutes, find out which subsystems have
-not turned to green and stay blue instead. Please check thresholds and detector setup of these systems, it might be that there are noise
-hits in addition to the calibration pulser hits. The calibration procedure can be stopped
-at any time anyway and will use the calibration infos although not complete. Repeat the calibration procedure after the reason
-for the missing statistics has been fixed as soon as possible.
-
- \item Be sure that the beam or the desired trigger source is switched on before starting beam or cosmics files again!
+\begin{enumerate} \item Be sure that the beam is off and no other data source may interfere with the
+calibration pulser!
+
+ \item Remove old calibration files, as they will corrupt the new calibration is some larger changes
+happened. There is ~/bin/deletecalfiles.sh to help.
+
+ \item Click the Button {\bf Start TDC Cal} in the DAQ operator GUI. This will disable all inputs of
+the CTS and put it into the special calibration mode (trigger type D, 500~Hz pulser). The
+eventbuilders will start writing files with prefix ''TC''.
+
+ \item Check the Tactical Overview for an increasing calibration status. For details, check the
+eventbuilder BNET GUI (Fig. \ref{fig:evtbuild:bnetgui}). The ''Input nodes'' box shows for each
+eventbuilder server the subsystems (''HUBs'') which send data to them from the cave. These are
+labeled with their trbnet hub address, e.g. ''0x83c1''. During calibration, the trb3 systems will
+turn to a blue color. The ''Run control'' caption of the webpage will show the state
+''Calibrating''.
+
+ \item When all of the input hubs in the BNET GUI have turned to green color, the calibration
+statistics is sufficient. In this state the ''Run control'' state will show ''Ready''. It can take
+several minutes until this is reached, depending on the configured calibration precision in the
+setup.
+
+
+ \item If BNET GUI (or corresponding hmon display) shows that calibration is OK, stop the
+calibration procedure by pressing the {\bf Stop TDC Cal} button in the DAQ control GUI ({\bf do not
+stop the run with the ''NO FILE'' button, and do not start immediately recording any other files
+types by the corresponding control GUI buttons}). The eventbuilders will close the calibration run
+files and will archive the most recent calibration information. The trigger settings of the CTS will
+be restored to the values just before the calibration procedure.
+
+ \item If BNET GUI should show that calibration is not successful after 5 minutes, find out which
+subsystems have not turned to green and stay blue instead. Please check thresholds and detector
+setup of these systems, it might be that there are noise hits in addition to the calibration pulser
+hits. The calibration procedure can be stopped at any time anyway and will use the calibration infos
+although not complete. Repeat the calibration procedure after the reason for the missing statistics
+has been fixed as soon as possible.
+
+ \item Be sure that the beam or the desired trigger source is switched on before starting beam or
+cosmics files again!
\end{enumerate}
--- /dev/null
+The documentation of the trigger box can be found here:
+\url{https://hessenbox-a10.rz.uni-frankfurt.de/getlink/fiUjpMeXrUCkLQeTXHtuw8un/TriggerBoxLogic.docx
+}
+
+More information on the trigger collection scheme:
+\url{https://hessenbox-a10.rz.uni-frankfurt.de/getlink/fi3okFdMs98kqxjyqKvziPid/Trigger2021.pptx}
jspc29 / daqdocu.git / commitdiff
