--- /dev/null
+<h3>Number of busy boards</h3>
+<p>
+This button gives you the number of busy front-end boards, that have stopped working and are blocking the trigger.
+During normal operation, the full message gives the busy times for all sub-systems.
+
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+<ul>
+<li>First, try to reSync missing OEPs
+<li>Check the list of board addresses, which board is busy.
+<li>If there is a FEE error at the same time and the busy board belongs to MDC, use "resync missing OEP", do a power-cycle of the corresponding chamber if this fails.
+<li>A DAQ restart should solve the problem.
+</ul>
+<br>Note that in some cases this is not the real source of the error - e.g. if the data transport to EB failed, the board with the smallest buffer and the most data gets busy first.
+
+
+
+
+
+
--- /dev/null
+<h3>DAQ Timeouts</h3>
+<p>
+This button gives information about not responding front-ends.
+</p>
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+<ul>
+<li>This time-out detection is only available for MDC. Try to ReSync missing OEPs (takes 15 seconds).
+<li>If the same board shows errors repeatingly, try a power cycle of this chamber.
+</ul>
--- /dev/null
+
+<h3>TrbNet Basic Status</h3>
+The basic status of the DAQ network is checked by a simple access to one network hub to see, if a board in the network has failed
+to answer, or the network is completely out of order.
+
+
+<h4>Error Handling</h4>
+This error typically pops up in combination with many other boxes showing errors. Follow the normal DAQ restart guidelines.
+
+<dl><dt>Error: RPC connection failed<dd>Restart the interface to the DAQ network, i.e. trbnetd. Then restart DAQ
+(Note that "RPC" refers to "Remote Procedure Call", not our detector!)
+<dt>Error: Timeout<dd>The network failed to answer within the given amount of time. Try to restart the DAQ.
+<dt>Warning: One endpoint didn't react<dd>Restart DAQ
+<dt>Other Errors<dd> A DAQ restart should cure it.
+<dt>OK<dd>Don't touch a running system.
+</dl>
\ No newline at end of file
--- /dev/null
+<h3>Data Rate</h3>
+<p>
+The data rate gives the amount of data per second transfered by all Eventbuilders. <br><br>
+<b>Also displayed are:</b> <br>
+the amount of data transfered per event and <br>
+the number of
+events per second per Eventbuilder.
+</p>
+
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+The number of MDC calibration events and status events are too low. Check at the CTS Monitor if they are switched on.
+Check the long error message to see if the number of MDC calibration and status triggers increases by 1 per second - if so, everything is ok and the error can be ignored.
+<br>
+If not: Switch them on.
+<br>
+In case nothing helps: DAQ restart (For restart instructions, see <a href="/mon/doc/restartdaqguideline.htm">restart-DAQ guidelines</a>)
+
+
+
+
+
+
+
--- /dev/null
+<h3>Error bits</h3>
+<p>
+This buttons shows the number of accepted events (written to disk) with errors reported by the front-ends.
+<br>Try to do a DAQ restart
+<br>In some cases a power-cycle of the corresponding system / chamber is necessary.
+
+
+
+
+
+
--- /dev/null
+<h3>Number of discarded events</h3>
+<p>
+This button gives you the number of discarded events in the Eventbuilders.
+<br>The reason can be a overloaded CPU in the EB or any other error. Try to restart the Eventbuilders.
+
--- /dev/null
+<h3>Comparison of CTS Rate and Eventbuilder Rate</h3>
+<p>
+This number gives the difference between the Eventbuilder and the CTS Rate. This difference should be around zero.
+As long as the button is green everything is O.K. - even if the box shows some large deviations for a short time.
+</p>
+<h4>Error Handling if the button is not <font color="gree">Green</font> for more than 10 seconds: </h4>
+
+<dl>
+<dt>In case of a negative deviation</dt>
+<dd>the Eventbuilder rate is below the
+ CTS rate, first check the
+ <a href="doc.cgi?eb-run">#EB running</a>
+ and
+ <a href="doc.cgi?eb-lostevt">#Evt Discarded</a>
+ buttons as
+ well. Usually they should give further information on what the
+ problem is.</dd>
+<dt>In case of a positive deviation</dt>
+<dd>the Eventbuilder rate is above the
+ CTS rate, there must be a seldom but serious error. Both the state of
+ all Eventbuilders and the CTS must be investigated in detail.</dd>
+</dl>
+
+
+
+
+
--- /dev/null
+<h3>Number of running Eventbuilders</h3>
+<p>
+The right number gives the number of Eventbuilders where the DAQ sends data to (stays always the same).<br>
+The left number gives the number of Eventbuilders writing data to disc.<br>
+This number should be equal and in this case the button would be green. (Cosmics: 1 EB, Beamtime: 8 EB)
+
+</p>
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+<dl>
+<dt><font color="red">Restart Eventbuilders</font></dt>
+<dd>See <a href="http://hadesdaq02/mon/doc/restartEBguideline.htm">restart-instructions</a> for
+Eventbuilders</a></dd>
+</dl>
--- /dev/null
+<h3>MDC Endpoints</h3>
+
+This button shows if all MDC front-end modules are active. The button is red if at least one module is missing.<br>
+
+
+<h4>Error Handling</h4>
+<ul><li>If there is a Timeout reported at the same time, do a "Resync missing OEP"
+<li>If 1-3 boards show an error, do a power-cycle of the corresponding sector
+<li>If 7-9 boards show an error, do a DAQ restart
+<li>If 14/18 boards show an error after a power-cycle, click "MDC LV Turn On ALL Relais"
+</ul>
+<br>
+
--- /dev/null
+<h3>Other Endpoints</h3>
+
+This button shows if all front-end modules of Shower, Forward Wall, Start-/Veto-detector and CTS are active.
+The button is red if at least one module is missing.
+
+<h4>Error Handling</h4>
+<p> Restart DAQ. If this doesn't help, do a power-cycle of the corresponding system.
+
--- /dev/null
+<h3>RICH Endpoints</h3>
+
+
+This button shows if all RICH front-end modules are active. The button is red if at least one module is missing.
+
+<h4>Error Handling</h4>
+<p> Restart DAQ. If this doesn't help, do a power-cycle of the corresponding system.
+
--- /dev/null
+<h3>RPC Endpoints</h3>
+
+This button shows if all RPC front-end modules are active. The button is red if at least one module is missing.
+
+<h4>Error Handling</h4>
+<p> Restart DAQ. If this doesn't help, do a power-cycle of the corresponding system.
+
--- /dev/null
+<h3>TOF Endpoints</h3>
+
+
+This button shows if all TOF front-end modules are active. The button is red if at least one module is missing.
+
+<h4>Error Handling</h4>
+<p> Restart DAQ. If this doesn't help, do a power-cycle of the corresponding system.
+
+
--- /dev/null
+<h3>FEE Buffer</h3>
--- /dev/null
+<h3>Front-End Electronic Errors</h3>
+<p>
+This button shows the number of boards that have not been initialised in the last few seconds.
+This might appear from time to time because a front-end had an error bus recovered automatically.
+
+<h4>Error Handling in case it is not <font color="gree">Green</font> for more than 30 seconds.</h4>
+<ul><li>If there is a busy at the same time:
+<ul><li>If the board belongs to MDC: Do a LV power-cycle of the chamber
+<li>Else: Do a DAQ restart.
+</ul>
+<li>Else: Do a DAQ restart.
+</ul>
+
+
+
+
--- /dev/null
+<p>
+RICH APV Satus Report, it tells the status of all RICH APV Frontends. As
+long as the button is green everything is O.K.
+</p>
+<h4>Error Handling in case it is not <font color="gree">Green</font></h4>
+In most cases a DAQ restart helps. In rare cases a power-cycle has to be performed.
+
+<h4>Check the short Error Messgage Indicator dislayed on the button</h4>
+<ul>
+<li> <code>'S__'</code> : APV Sync Error: resync the APVs by calling the script (not_yet_implemented)</li>
+<li> <code>'_T_'</code> : APV Trigger Counter Error: (not_yet_implemented)</li>
+<li> <code>'__I'</code> : APV IPU Readout Counter Error: (not_yet_implemented)</li>
+</ul>
+
--- /dev/null
+<h3>TRB TDC</h3>
+<p>
+This button shows if the TDC on the TRB are synchronized to the DAQ system.
+</p>
+<h4>Error Handling in case it is not <font color="gree">Green</font></h4>
+
+If they are not synchronized there is the danger of mixing different events. <br>
+You should restart DAQ several times.<br>
+
+
+
+
+
+
+
--- /dev/null
+<h3>Trigger Inputs</h3>
+<p>
+This button shows the trigger input errors. This could indicate a problem with a trigger / reference time / common stop signal.
+</p>
+<h4>Error Handling</h4>
+Restarting DAQ should help, but could also be a mechanical issue.
+
+
+
+
+
--- /dev/null
+<h3>Trigger Quality</h3>
+<p>
+This button shows the trigger errors per second and the total sum of errors since the last DAQ restart. This rate should be low to avoid events with errors.
+</p>
+<h4>Error Handling in case it is not <font color="gree">Green</font></h4>
+Try a DAQ restart.
\ No newline at end of file
--- /dev/null
+<h1>Wall Clock</h1>
+
+
+<p>This is the system time. If it fails, run away.
\ No newline at end of file
--- /dev/null
+<h3>Online QA</h3>
+<p>
+Checks if the online QA server is running.
+</p>
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+Check the QA machine if everything is ok.
--- /dev/null
+
+<h3>CTS Current Rate</h3>
+The current rate at which triggers are sent by the CTS. The rate should always be in the expected region!!! If it is not, first take a look at the CTS monitor for the TOF and RPC rates to find out where the problem is located.
+
+
+<h4>Error Handling</h4>
+<dl><dt>Shown trigger rate is low, but DAQ is running faster<dd>The counter has a bug if the trigger rate is above 65 kHz (only reachable with pulser) due to a counter overflow. Ignore the warning message in this case.
+<dt>Rate is 2-5 Hz<dd>Typically this happens when only calibration triggers are sent and no additional triggers are generated. Make sure this is desired and trigger settings are right.
+<dt>The rate is 0<dd><ul><li>Check the "<a href="doc.cgi?daq-busy">Busy Boards</a>" monitor to see if a front-end has stopped working and is blocking the trigger. Follow the usual <a href="doc/restartdaqguideline.htm">restart-DAQ guidelines</a> to get the DAQ running again.
+<li>Check the CTS settings if the right trigger source is selected.
--- /dev/null
+<h3>Wall Clock</h3>
+<p>This is the system time. Will always be "ok", because it is always a good time to work.
+
+
+<h4>Error Handling</h4>
+
+<ul><li>If it fails, run away.
+<li>If you don't know what it means, ask your parents.
+<li>If time seems to run slowly, pay more attention to the status reports of DAQ and QA.
+</ul>
\ No newline at end of file
--- /dev/null
+<h3>MDC Blocked</h3>
+<p>
+This button gives the number of MDC-MBO that do not deliver data (updated only if rate per motherboard is above 1000, and trigger is running at more than 1000 Hz).
+
+</p>
+<h4>Error Handling</h4>
+Do a power-cycle of the corresponding MDC chamber.
+
+
--- /dev/null
+<h3>MDC Link Errors</h3>
+<p>
+This button shows the number of link errors per second. If this number is too high, this could indicate that a front-end board
+is damaged. No action on the DAQ is required.
+</p>
+<p>Problems can be related to unstable HV in some chambers.
+
+
+
+
--- /dev/null
+<h3>MDC OEPS</h3>
+<p>
+This button shows the number of "interesting" OEPS (without giving the adress). For error handling see the button
+<a href="http://hadesdaq02/mon/doc.cgi?endp-mdc">MDC system</a>.
+</p>
+
+
+
+
+
--- /dev/null
+<h3>MDC Temperature</h3>
+<p>
+This button shows the temperatures of the four MDC planes. For more detailed temperature distributions in the single MDC sectors, watch
+<a href="/mon/monitor.cgi?10-MDCTemperature">here</a>.
+</p>
+<h4>Error Handling</h4>
+If there is an over-temperature error, check the power consumption of boards (6.8V should not be above 190A). Restart DAQ to see if power-consumption goes down. A power shutdown of MDC may be required.
+
+
+
+
+
--- /dev/null
+<h3>MDC Token</h3>
+<p>
+This button shows the number of automatically corrected motherboard errors. The number shouldn't be too high since any re-initialisation needs time. For very high rates, consider a power cycle of the corresponding chamber.
+</p>
+<br>
+
+
+
+
+
--- /dev/null
+<h3>MDC Voltages</h3>
+<p>Voltages for all main components are measured on the MDC OEP front-ends continuosly.
+
+
+<h4>Error Handling</h4>
+<p>Any error can only be solved by direct access to the MDC chamber were the problem exists. Thus: No immediate error handling is required. If you notice something strange, a logbook entry describing the problem is sufficient.
+Voltages which are far off (should not happen after the system is commissioned for the beam-time) may cause problems with the DAQ, but boards are quite tolerant here. Data quality might be affected earlier.
\ No newline at end of file
--- /dev/null
+<h3>Magnet</h3>
+<p>
+This button shows the status of the Magnet systems.
+</p>
+<h4>Error Handling in case it is not <font color="gree">Green</font> for > 2 min.</h4>
+
+Call <b>Torsten Heinz</b> (mobile: 0175 388 4066 or home: 06162 982292 or work: 2781<br>
+and <b>Wolfgang Koenig</b> (mobile: 0172 877 50 49 or home: 06071 35998 or work: 2720)!
+
+
+<p>If no ssh connection to the magnet PC is possible (hadesp28), check if the machine is still running (upstairs, next to cryo)
+
+
+
+
+
--- /dev/null
+<h3>SHower status</h3>
+<p>
+This button shows if there is nothing wrong with general data rates generated by Shower sectors.
+</p>
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+<dl>
+<dt>
+YELLOW - Message about currents:<br> No need to restart DAQ or stop data taking but immediately contact Shower expert. The voltage on the detector has to be changed.<br> DAQ restart not required.<br>
+RED - Message about configuration:<br> Try to restart DAQ and if this doesn't help contact Shower expert.<br>
+YELLOW:<br> Usually one front end board on sector 4 starts to generate much noise for no reason, but will also stop by itself.
+If it's sector 4 then nothing can be done, if it's a different one you can ask the nearest Shower expert if it should be like this.
+Restarting DAQ, which will reload pedestals can help</dt>
+<dd></dd>
+</dl>
+
+
--- /dev/null
+<h3>Speech Output</h3>
+<p>
+This button shows the status of the speech deamon which gives you accustic informations in case of critical problems
+with any subsystem.
+
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+It can only be switched off. If that is the case, turn it on.
+
+
+
--- /dev/null
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+<title>Tactical Overview Documentation</title>
+<meta http-equiv="content-type" content="text/html;charset=UTF-8"/>
+
+<link href="../files/indexstyles.css" rel="stylesheet" type="text/css"/>
+</head>
+<body class=index>
+<div class="button" style="right:-8px;" onclick="history.back();"> back </div>
+<h1>Restart EB Guideline</h1>
+<h2>In case EB has to be restarted</h2>
+</body>
+</html>
\ No newline at end of file
--- /dev/null
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+<title>Tactical Overview Documentation</title>
+<meta http-equiv="content-type" content="text/html;charset=UTF-8"/>
+
+<link href="../files/indexstyles.css" rel="stylesheet" type="text/css"/>
+</head>
+<body class=index>
+<div class="button" style="right:-8px;" onclick="history.back();"> back </div>
+<h1>Restart DAQ Guideline</h1>
+<h2>In case DAQ has to be restarted</h2>
+<p>First, check the QA monitor and user guide to see if there is a more appropriate solution to the problem you experience.
+
+<p>The DAQ-Control provides the necessary operations for a DAQ restart.:
+
+<ol><li>Press "Start DAQ" to restart DAQ. An orange window appears and
+should disappear after 30s-40s if the restart was successful.
+An unsuccessful attempt is indicated by an error message in
+the orange window. Try a second DAQ restart, if the first
+one failed.
+
+<li>In the case of an unsuccessful DAQ restart a MDC Power Cycle
+has to be performed. The steps of a MDC Power Cycle are described
+in the Operator Guide Chapter 34.2. After this operation the DAQ
+has to be restarted. Follow the instructions given in step "1. )".
+
+<li>In the Case "2. )" failed, the "RebootMDCHub" icon should be used.
+After pressing "RebootMDCHub" and waiting 30 seconds, the DAQ has to be
+restarted which is described in "1. )".
+
+<li>In the case step "3. )" was not successful, a "Full Power Cycle" has to be
+performed. This procedure is described in the "Operator Guide Chapter 33.1"
+Restart DAQ following the steps in "1.)" .
+
+<li>The DAQ expert has to be called if the DAQ is still not working.
+</ol>
+
+
+<p>All DAQ-Control Tools will open a orange window with their output which disappears
+after they finished. Make sure that no two orange windows are open at the same time.
+</body>
+</html>
\ No newline at end of file
--- /dev/null
+<h3>Server CPU</h3>
+<p>
+This number gives you the server cpu with the highest load, which should not be exceeding ##%. A high server cpu leads to discarded events. <br>
+See a list of the CPU loads <a href="http://hadesdaq02/mon/monitor.cgi?10-window-EBCPU">here</a>.
+</p>
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+<dl>
+<dt>You should have a look if there is some error in the status of the Eventbuilder-buttons.</dt> If not, ignore the error.
+<dd>If there is a problem, restart Eventbuilders (see <a href="http://hadesdaq02/mon/doc/restartEBguideline.htm">restart-instructions</a> for
+Eventbuilders)</a></dd>
+</dl>
+</dl>
+
+
+
+
+
--- /dev/null
+<h3>Status of the TRB</h3>
+
+This field shows if the TRB is up and checks its status.
+
+
+<h4>Error Handling</h4>
+
+<p>
+In case of an error (red field), please take a look to
+the Etrax <a href="http://icingaadmin@cerberus.x-matter.uni-frankfurt.de:8888/icinga/" target="_blank">tactical overview</a>.
+
+<p>An overview of the different servers and services monitored is given in the <a href="http://hades-wiki.gsi.de/cgi-bin/view/DaqSlowControl/IcingaMonitor" target="_blank">HADES Wiki</a>
+
--- /dev/null
+<h3>Disk fill level </h3>
+<p>
+This button gives the disk fill level.
+</p>
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+<dl>
+A detailled view of the fill level of every single disk can be seen <a href="http://hadesdaq02/mon/monitor.cgi?30-EBDisks">here</a>. The DAQ expert (!) should delete
+files, that have already been copied to lustre or on tape.
+</dl>
+
+
+
+
+
+
--- /dev/null
+<h3>Status of the Icinga Server</h3>
+
+This field shows if the Icinga Server is up and checks its status.
+
+
+<h4>Error Handling</h4>
+
+<p>
+In case of an error (red field), please take a look to
+the Icinga <a href="http://icingaadmin@cerberus.x-matter.uni-frankfurt.de:8888/icinga/" target="_blank">tactical overview</a>. An eMail is automatically sent to a DAQ-expert,
+who should know how to proceed.
+
+<p>An overview of the different servers and services monitored is given in the <a href="http://hades-wiki.gsi.de/cgi-bin/view/DaqSlowControl/IcingaMonitor" target="_blank">HADES Wiki</a>.
+
--- /dev/null
+<h3>Status of the Power Supply</h3>
+
+This field shows if the Power Supply is okay and checks its status.
+
+
+<h4>Error Handling</h4>
+
+<p>
+In case of an error (red field), please take a look to
+the Icinga <a href="http://hadesdaq02/icinga/" target="_blank">tactical overview</a>.
+<p>
+
+If nothing helps, call a DAQ expert.
+
+<p>
--- /dev/null
+<h3>Trigger accepted</h3>
+<p>
+This button shows in percentage of available triggers accepted by the CTS to read out the front-end boards.
+Typical value should be around 50% in total.
+The first value shows the percentage of accepted triggers in the last second, the second value is the integral number for the last spill.
+</p>
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+<dl>
+The beam quality could be bad. The shift leader has to decide whether to call the control room.
+</dl>
+
+
+
+
+
--- /dev/null
+<h3>PT3 Rate</h3>
+<p>
+This button shows the triggered high multiplicity events per second compared to the countrate in the Start-detector.
+The second number should be around 0.7% during beam time since the interaction rate is around 1%. The first value is the rate
+of PT3 triggers provided to the CTS.
+<br>
+If something goes wrong, ask your shift leader.
+
+
--- /dev/null
+<h3>Trigger sources</h3>
+<p>
+This button displays the used Trigger source. It is <font color="yellow">yellow</font> if there is a trigger input error to CTS.
+</p>
+If the button is <font color="#AC0">green-yellow</font>, the pulser is switched on.
+
+
+<h4>How to deal with problems:</h4>
+<ul>
+<li>If the port that is displayed above is not a selected trigger source or not corresponding to Start/Veto, the error
+can be ignored.</li>
+<li>Restart DAQ to make sure all settings are loaded correctly
+</ul>
+
+
+
+
--- /dev/null
+<h3>CTS Spill Rate</h3>
+The number of accepted triggers per spill is calculated based on the event counter, and spill start/stop information provided by the CTS. Spill start is a signal we get from the accelerator, spill stop is generated by the CTS after the spill length given in the CTS settings (see CTS monitor / settings overview). Additionally, the 10-spill and 50-spill average are calculated.
+
+
+<h4>Error Handling</h4>
--- /dev/null
+<h3>Trigger Start Counter</h3>
+<p>
+This button gives you the number of counts per second in the start-detector of the last spill (second value). The first one is the trigger count for the last second.
+<br>
+In case of problems, ask your shift leader.
+
+
+
jspc29 / hadesdaq.git / commitdiff
