From 6cbe7aff621e5013c1af203e87e9b85f5e348dda Mon Sep 17 00:00:00 2001
From: Jan Michel <mail@janmichel.eu>
Date: Thu, 10 Feb 2022 20:46:45 +0100
Subject: [PATCH] update operator manual

---
 hadesoperator/powercycles.tex | 72 +++++++++++++++++++++++++++++------
 1 file changed, 61 insertions(+), 11 deletions(-)

diff --git a/hadesoperator/powercycles.tex b/hadesoperator/powercycles.tex
index f062ef4..df67335 100644
--- a/hadesoperator/powercycles.tex
+++ b/hadesoperator/powercycles.tex
@@ -32,17 +32,67 @@ are active again (see the list in ``EPICS Power'' -> ``RICH LV''), do a DAQ rest
 In the case that the power cycle triggered the temperature interlock for RICH power supplies.
 The 3.3V supply might be blocked by the temperature interlock - restart DAQ, then try to reset it.
 
-\subsection*{CTS}
-If a power cycle of the CTS was done, don't forget to ``Reprogram FPGA'' and 
-``BLR Reinit FPGA'' before the DAQ restart.
-
-\subsection*{STS FEE}
-After a powercycle of the STS (Fee or Trb, thresholds need to be loaded manually, using ``Threshold 
-STS''.
 
 \subsection*{Other Subdetectors}
 Check section \ref{powersupplies} to see which power supply to switch.
 
+
+\subsection{Jörg's helpful tips}
+\subsubsection*{ECAL / RPC}
+\begin{itemize}
+\item After an ECAL powercycle most often a ECAL Padiwa powercycle is needed. You can check this 
+looking at the ECAL rate plot seeing weird structures (e.g. 8 green dots in a row) or completely 
+missing parts.
+
+\item After the powercycle and DAQ restart often the RPC is still not fully operational. Often you 
+need to repeat the powercycle until RPC and ECAL (connected to same LV) are both operational again.
+
+\item After the first DAQ restart you should press the Fix missing IP preemptively as otherwise the 
+EB might fail.
+\end{itemize}
+
+\subsubsection*{STS}
+\begin{itemize}
+\item Whenever you do an STS powercycle, you need to do "Thresholds STS"  if it did not work (you 
+see e.g. blue spots in the STS Hit rate plot) do a "Thresholds STS reset" (this takes a bit longer 
+as it is comprised of four different scripts)
+\end{itemize}
+
+\subsubsection*{TOF}
+\begin{itemize}
+\item After the TOF powercycle check if the rates in all sectors are equal - if not a "Thresholds 
+TOF" is needed.
+\item If settings thresholds doesn't work in the first try, repeat it a couple more times. 
+\end{itemize}
+
+\subsubsection*{iTOF}
+\begin{itemize}
+\item After the iTOF powercycle check if the rates in all sectors are equal - if not a "Thresholds 
+iTOF" is needed. If this doesn't help, do a "Powercycle iTOF Padiwa" and try again.
+\end{itemize}
+
+\subsubsection*{Procedure if TRBnet fails}
+If suddenly TRBnet fails (normally indicated by every button on the tactical display blinking red) 
+do a powercycle of the common group. Thereafter you need to switch on the RICH LV again after 
+waiting about 10 seconds.
+If you powercycled the common group, check all the thresholds and rates as explained in the ECAL, 
+RPC and TOF sections.
+
+\subsubsection*{General remark}
+\begin{itemize}
+\item If the EBs do not come up after a (or several) power cycles, try a "Fix missing IP".
+\item Sometimes some MDC boards are missing for a longer time such that the tactical overview 
+informs you to take action.
+Very often a DAQ restart is sufficient to get these boards working again.
+If however after some minutes the exact same MDC boards go missing again, you should consider a LV 
+power cycle of the respective sectors when you restart the DAQ next time.
+
+\item If boards frequently fail to reboot after a power cycle you might consider switching off the 
+beam for the power cycling and subsequent startup procedure. You can do that via the PC in the 
+server room.
+
+\end{itemize}
+
 \subsection*{Full Powercycle}
 A full powercycle should only be done after contact with an expert. Follow the guide for the 
 ``Daily DAQ reboot''.
@@ -55,13 +105,13 @@ of the DAQ expert: {\bf To be updated for 2022!}
 \item Switch off beam on control panel\\
       Switch EB to No File
 \item MDC power supply group off (EPICS)\\
-      Common Group power supply group power cycle (EPICS)  
+      Common Group power supply power cycle (EPICS)  
 \item RICH power supply group power cycle (EPICS) 
 \item ECAL Padiwa Powercycle \\
       Hodo Powercycle\\
       Start Powercycle \\
       STS Fee Powercycle\\
-      iTof Powercycle \\
+      iTof Padiwa Powercycle \\
       Triggerbox Beam Powercycle
 
 
@@ -70,11 +120,11 @@ of the DAQ expert: {\bf To be updated for 2022!}
       BLR Reinit FPGA
 \item Start DAQ (x2, because of MDC Hubs)   
 \item Run Fix Missing IP
-\item MDC power supply group on (EPICS)
+\item MDC power supply group on (EPICS) \\
       MDC LV Turn On All Relais
 \item Reboot OEP
 \item Start DAQ
-\item Thresholds STS Reset\
+\item Thresholds STS Reset\\
       Check Eventbuilders getting data / all boards have IPs
 
 \item Switch on beam
-- 
2.43.0