<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
<ul>
-<li>First, try to reSync missing OEPs
<li>Check the list of board addresses, which board is busy.
-<li>If there is a FEE error at the same time and the busy board belongs to MDC, use "resync missing OEP", do a power-cycle of the corresponding chamber if this fails.
-<li>A DAQ restart should solve the problem.
+<li>If the address starts with '2' (from MDC), try to resync missing OEPs
+<li>If not, a DAQ restart should solve the problem.
+<li>If the misbehaving board is from MDC and the error happens several times for the same board,
+do a MDC power-cycle of the corresponding chamber.
+<li>If the board is from MDC, and there is a "FEE Error" at the same time, the MDC power-cycle is needed.
</ul>
-<br>Note that in some cases this is not the real source of the error - e.g. if the data transport to EB failed, the board with the smallest buffer and the most data gets busy first.
+<br>Note that in some cases this is not the real source of the error -
+e.g. if the data transport to EB failed, the board with the smallest
+buffer and the most data gets busy first. Check also the next box "Read-out" for further information.
</p>
<h4>Error Handling in case it is not <font color="gree">Green</font></h4>
-If they are not synchronized there is the danger of mixing different events. <br>
-You should restart DAQ several times.<br>
+Some of the TRB2-based boards saw problems on the reference time input. There is the risk of event mixing. Restart DAQ.
--- /dev/null
+<h3>Magnet</h3>
+<p>
+This button shows the status of the Magnet systems.
+</p>
+<h4>Error Handling in case it is not <font color="gree">Green</font> for > 2 min.</h4>
+
+<p>The script might show red precisely at midnight due to change of logfiles. Don't worry in this case.
+
+Call <b>Torsten Heinz</b> (mobile: 0175 388 4066 or home: 06162 982292 or work: 2781<br>
+and <b>Wolfgang Koenig</b> (mobile: 0172 877 50 49 or home: 06071 35998 or work: 2720)!
+
+
+<p>If no ssh connection to the magnet PC is possible (hadesp28), check if the machine is still running (upstairs, next to cryo)
+
+
+
+
+
--- /dev/null
+<h3>RICH High Voltage</h3>
+<p>
+This button gives you the minimum and maximum Voltage of the HV of the RICH detector.
+If this is off, the MAPMTs are not in measuring condition!
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+<ul>
+<li>HV is not in right condition. Are all channels on?
+<li>Check <a href="monitor.cgi?2-RichHV">HV status</a>. Are all values in the target region?
+<li>Call an expert immediately!
+</ul>
+
+
+
+
+
--- /dev/null
+<h3>EPICS HV Sequencer</h3>
+
+<p>Some high voltage control is not working any more. Please inform the MDC operator before taking any action.
+<h4>Error Handling</h4>
+<p>Click the 'Restart Sequencer' button in the control window or run on lxhadeb06:
+<br><pre>ssh scs@lxhadeb06 'echo -en "\x18" | netcat -w 1 localhost 4813</pre>
+<br>Restarting might take a minute until the button is green again.
<h3>CTS Current Rate</h3>
-The current rate at which triggers are sent by the CTS. The rate should always be in the expected region!!! If it is not, first take a look at the CTS monitor for the TOF and RPC rates to find out where the problem is located.
+The current rate at which triggers are sent by the CTS. The rate should always be in the expected region! If it is not, first take a look at the CTS monitor for the TOF and RPC rates to find out where the problem is located.
<h4>Error Handling</h4>
-<dl><dt>Shown trigger rate is low, but DAQ is running faster<dd>The counter has a bug if the trigger rate is above 65 kHz (only reachable with pulser) due to a counter overflow. Ignore the warning message in this case.
+<dl><dt><strike>Shown trigger rate is low, but DAQ is running faster<dd>The counter has a bug if the trigger rate is above 130kHz (only reachable with pulser) due to a counter overflow. Ignore the warning message in this case.</strike>
<dt>Rate is 2-5 Hz<dd>Typically this happens when only calibration triggers are sent and no additional triggers are generated. Make sure this is desired and trigger settings are right.
<dt>The rate is 0<dd><ul><li>Check the "<a href="doc.cgi?daq-busy">Busy Boards</a>" monitor to see if a front-end has stopped working and is blocking the trigger. Follow the usual <a href="doc/restartdaqguideline.htm">restart-DAQ guidelines</a> to get the DAQ running again.
<li>Check the CTS settings if the right trigger source is selected.
<h3>MDC Link Errors</h3>
<p>
-This button shows the number of link errors per second. If this number is too high, this could indicate that a front-end board
-is damaged. No action on the DAQ is required.
-</p>
-<p>Problems can be related to unstable HV in some chambers.
+This button shows the number of link errors per second. If this number is too high, this could indicate that a front-end board is damaged. Usually, this doesn't need any error handling. Problems can be related to unstable HV in some chambers.
+<p>If this number is huge and at the same time "trigger errors" also shows problems, there might be a problem with a front-end. Restart the DAQ - usually it will tell you that the compile time of one OEP is wrong, so you have to do a "Reload MDC".
--- /dev/null
+<h3>TDC Calibration status</h3>
+<p>
+This button gives you the status of the last TDC calibration.
+
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+Please run a new TDC calibration. Possible reasons are:
+<ul>
+<li>The temperatur has shifted to far from last calibration.
+<li>There are front-ends in the system whose calibration file has not been updated
+<li>The time since the last calibration is too long
+<li>Eventbuilders return a bad quality of the last calibration run
+</ul>
+
+
+
+
+
--- /dev/null
+<h3>DiRICH Temperature on Backplane</h3>
+<p>
+This button gives you the minimum and maximum temperature between the Backplane and the MAPMTs (as well as at the inner spokes) in the RICH detector. <br/>
+This temperatur is critical for the RICH! The backplane temperature is connected to a interlock that shuts down the RICH after a certain temperature is reached (See Tactical Overview).
+
+<h4>Error Handling if the button is not <font color="gree">Green</font> or <font color="orange">Orange</font></h4>
+
+<ul>
+<li>Temperature of DiRICH is too high.
+<li>Check temperature of the <a href="rich_drawing_bkpl.htm#richTemperatureBackplane">Backplane</a> and the <a href="monitor.cgi?10-RichInnerTemp">Gas </a>. Is update time up to date?
+<li>If temperature is realy to high and it stays to high (ca. 2 min), call the operator.
+<li>If Interlock has fired and RICH is down, call the operator.
+</ul>
+
+
+
+
+
--- /dev/null
+<h3>DiRICH Current</h3>
+<p>
+This button gives you the minimum and maximum current of the FEE of the RICH detector.
+<!--h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+<ul>
+<li>Check if a backplane has significant different currents values in XXX and XXX.
+<li>If Backplanes are white, possibly a LV powersupply failed. An expert is needed!
+</ul-->
+
+
+
+
+
--- /dev/null
+<h3>Isobutan Gaspipe (RICH radiator)</h3>
+<p>
+This value gives you the pressure of the inlet line of the Isobutan pipe near the Gassystems in front of the HADES Cave.
+
+
+
+
--- /dev/null
+<h3>DiRICH Temperature</h3>
+<p>
+This button gives you the minimum and maximum temperature of the DiRICH boards of the RICH detector.
+This temperature is not directly criticly for the RICH operation, but it influences the inner temperature.
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+<ul>
+<li>Temperature of DiRICH is too high.
+<li>Check temperature of <a href="rich_drawing_2.htm#richTemperature">all DiRICHs</a>. Is update time up to date.
+<li>If temperature is realy to high and it stays to high (ca. 2 min), call the operator.
+</ul>
+
+
+
+
+
--- /dev/null
+<h3>DiRICH Voltage</h3>
+<p>
+This button gives you the minimum and maximum voltage of the FEE of the RICH detector.
+<h4>Error Handling if the button is not <font color="gree">Green</font></h4>
+
+<ul>
+<li>Check if a backplane has significant different voltage values in <a href="rich_drawing_2.htm#rich1V">DiRICH 1.1V</a> and <a href="rich_drawing_2.htm#rich2V5">DiRICH 2.5V</a>. Higher values on 4x4 backplanes are
+normal!
+<li>If a single backplane is on low power and makes problems, the LV connector has to be pressed in. An expert is needed!
+<li>If Backplanes are white, possibly a LV Powersupply failed. An expert is needed!
+</ul>
+
+
+
+
+
--- /dev/null
+<h3>PT3 Rate</h3>
+<p>
+This button shows the triggered high multiplicity events per second compared to the countrate in the Start-detector.
+The second number should be around 0.7% during beam time since the interaction rate is around 1%. The first value is the rate
+of PT3 triggers provided to the CTS.
+<br>
+If something goes wrong, ask your shift leader.
+
+
--- /dev/null
+#!/usr/bin/perl -w
+
+#Gets json files from EB, changes them to Hashes instead of arrays of names, and changes times to timestamps
+
+use strict;
+use warnings;
+use LWP::Simple qw(get);
+use JSON qw( decode_json encode_json );
+use Data::Dumper;
+use Try::Tiny;
+use DateTime;
+use DateTime::Format::ISO8601;
+
+sub hashify {
+ my $d = $_[0];
+ if ($d->{'_childs'}) {
+ my $i = 0;
+ for my $c (0 .. scalar @{$d->{'_childs'}}) {
+ foreach my $k (keys %{$d->{'_childs'}[$i]}) {
+ $d->{$d->{'_childs'}[$i]{'_name'}}{$k} = $d->{'_childs'}[$i]{$k};
+ if($k eq 'time') {
+ my $dt = DateTime::Format::ISO8601->parse_datetime($d->{$d->{'_childs'}[$i]{'_name'}}{$k});
+ $d->{$d->{'_childs'}[$i]{'_name'}}{$k} = $dt->epoch();
+ }
+ if($k eq '_childs') {
+ hashify($d->{$d->{'_childs'}[$i]{'_name'}});
+ }
+ }
+ $i++;
+ }
+ delete ($d->{'_childs'});
+ }
+
+ }
+
+sub getjsonhash {
+ my ($src, $name) = @_;
+ my $baseurl;
+ if($src eq 'rawmon') {$baseurl = 'http://lxhadeb12:8090/';}
+ if($src eq 'ebmon') {$baseurl = 'http://lxhadeb12:8090/';}
+ if($src eq 'eb') {$baseurl = 'http://lxhadeb07:8099/';}
+
+ my $rawdata = get ($baseurl.$name);
+
+ my $data;
+ if (defined $rawdata) {
+ try{
+ $data = decode_json($rawdata);
+ }
+ catch {sleep 5; return [];}
+ }
+
+
+ hashify($data);
+ return $data;
+ }
+
+# my $o = getjsonhash('eb', 'Master/BNET/get.json');
+
+
+1;
--- /dev/null
+#!/usr/bin/perl -w
+
+use warnings;
+use strict;
+use Time::HiRes qw( gettimeofday usleep time );
+use Data::Dumper;
+use Hmon;
+use QA;
+use LWP::Simple;
+use JSON qw( decode_json );
+use POSIX qw/floor ceil strftime/;
+use lib '.';
+use getebjson;
+use Storable qw(lock_store lock_retrieve);
+
+
+my $fqa = QA::OpenQAFile();
+
+
+
+while(1){
+ my $data = getjsonhash('eb', 'Master/BNET/get.json');
+ unless($data->{Inputs}{value}) {sleep 5; next;}
+
+
+ my $timesincecalib = time() - ($data->{LastCalibr}{'time'});
+ my $txt = strftime("%d.%m. %H:%M",localtime($data->{LastCalibr}{'time'}));
+ my $longtext = "Last Calibration was done at ".$txt;
+ my $status = QA::GetQAState('below',$timesincecalib,@QA::TdcCalibrationInterval);;
+
+
+ QA::WriteQALog($fqa,"misc","calib",1200,$status,'Last TDC Calib',$txt,$longtext);
+
+
+ my $store;
+
+ if($data->{LastCalibr}{'time'} != $store->{LastCalibr) {
+
+ }
+
+ sleep 60;
+ }
-#!/bin/bash
+#!/usr/bin/perl
+
+use warnings;
+use strict;
+
#ssh -X hadaq@hadesdaq01 " kdialog --title '$1' --passivepopup '$3: $2' 12 " 2>/dev/null 1>/dev/null&
#ssh -X hadaq@hadesdaq01 " kdialog --title '$1' --passivepopup '$3: $2' 12 " &
#ssh -X hadaq@hadesdaqp02 "DISPLAY=:0 kdialog --title '$1' --passivepopup '$3: $2' 12 " 2>/dev/null 1>/dev/null&
-ssh -X hadaq@hadesdaqp03 "DISPLAY=:0 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
-ssh -X hadaq@hadesdaqp04 "DISPLAY=:0 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
+my $a = $ARGV[0];
+my $b = $ARGV[1];
+my $c = $ARGV[2];
+
+$a=~s/<//g;
+$b=~s/<//g;
+$c=~s/<//g;
+
+$a=~s/>//g;
+$b=~s/>//g;
+$c=~s/>//g;
+
+my @list = qw(hadaq@hadesdaqp03:0 hadaq@hadesdaqp04:0 hadaq@hadesp30:1 hadaq@hadesp50:0);
-ssh -X hadaq@hadesp30 "DISPLAY=:0 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
-ssh -X hadaq@hadesp30 "DISPLAY=:1 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
-ssh -X hadaq@hadesp50 "DISPLAY=:0 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
+foreach my $cur (@list) {
+ my ($server, $display) = split /:/, $cur;
+ my $command = qq|ssh -X $server "DISPLAY=:$display notify-send --expire-time=12000 '$a' '$c: $b'" 2>/dev/null 1>/dev/null&|;
+ #print $command . "\n";
+ qx($command);
+}
+
+
+my $command = qq|logger -p local1.info -t "$ARGV[0]" "$ARGV[2]: $ARGV[1]"|;
+qx($command);
+
+exit;
+
+#ssh -X hadaq@hadesdaqp03 "DISPLAY=:0 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
+#ssh -X hadaq@hadesdaqp04 "DISPLAY=:0 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
+#ssh -X hadaq@hadesp30 "DISPLAY=:0 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
+#ssh -X hadaq@hadesp30 "DISPLAY=:1 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
+#ssh -X hadaq@hadesp50 "DISPLAY=:0 notify-send --expire-time=12000 '$1' '$3: $2'" 2>/dev/null 1>/dev/null&
#ssh -X hadaq@hadesp30 "DISPLAY=:0 kdialog --title '$1' --passivepopup '$3: $2' 12 " 2>/dev/null 1>/dev/null&
#ssh -X hadaq@hadesp50 "DISPLAY=:0 kdialog --title '$1' --passivepopup '$3: $2' 12 " 2>/dev/null 1>/dev/null&
#ssh -X hadaq@lxhadesdaqp " kdialog --title '$1' --passivepopup '$3: $2' 12 " 2>/dev/null 1>/dev/null &
#ssh -X hadaq@lxhadesdaq " logger -p local1.info -t DAQ '$3 $2'" &
-logger -p local1.info -t DAQ "$3 $2"
+
jspc29 / hadesdaq.git / commitdiff