\item The hub FPGA is in reset while no fiber is connected or the master port it is connected
to has its \texttt{TX{\_}DIS} line set.
\item \textbf{CTS MP:} the \texttt{TX{\_}DIS} is released once the SerDes TX channel is ready,
- and a stream of IDLE kommas is sent to initiate the link.
- \item \textbf{HUB SP:} A stream of IDLE kommas is received, so the RX reset sequence is completed
+ and a stream of IDLE0 kommas is sent to initiate the link.
+ \item \textbf{HUB SP:} A stream of IDLE0 kommas is received, so the RX reset sequence is completed
and a lock to incoming kommas by the FPGA internal state machine is made.
- \item \textbf{HUB SP:} Once WAP=0 is achieved, the RX reset control will declare the RX recovered clock stable,
- and initiate the TX reset sequence. \texttt{LINK{\_}RX{\_}READY} is set, while toggling IDLEs are received,
- the uplink is stable now.
- \item \textbf{HUB SP:} The TX reset sequence waits for all QUADs in use to lock on the TX reference clock (i.e.
- the recovered RX clock from slave port), and sync all TX Serializers to bit 0. \texttt{LINK{\_}TX{\_}READY} is set.
- \item \textbf{HUB SP:} The TX control sends a togglinge idle sequence after the downlink is stable.
- \item \textbf{HUB SP:} The RX LSM will check for a sequence of toggling IDLEs, and after detection set the
+ \item \textbf{HUB SP:} Once WAP=0 is achieved, the RX reset control will declare the RX recovered
+ clock stable, and initiate the TX reset sequence. \texttt{LINK{\_}RX{\_}READY} is set, while
+ IDLE0 are received, the uplink is stable now.
+ \item \textbf{HUB SP:} The TX reset sequence waits for all QUADs in use to lock on the TX reference
+ clock (i.e. the recovered RX clock from slave port), and sync all TX Serializers to bit 0.
+ \texttt{LINK{\_}TX{\_}READY} is set.
+ \item \textbf{HUB SP:} The TX control sends IDLE0 after the downlink is stable.
+ \item \textbf{HUB SP:} The RX LSM will check for IDLE0, and after detection set the
\texttt{LINK{\_}HALF{\_}DONE} signal.
- \item \textbf{CTS MP:} the toggling IDLEs are received, a RX reset sequence is performed and a lock
- to WAP=0 made. This will set \texttt{LINK{\_}RX{\_}READY} in the master port, and change the toggling IDLEs in the uplink
- to steady IDLEs (IDLE1).
- \item \textbf{HUB SP:} The RX LSM detects a series of steady IDLEs, and sets \texttt{LINK{\_}FULL{\_}DONE} internally.
- The TX control is instructed to send steady IDLEs now. Both UP and DL are considered stable, and TrbNet
- payloads (as well as special kommas) can be sent and received now.
- \item \textbf{CTS MP:} The RX LSM detects a series of steady IDLEs, and sets to \texttt{LINK{\_}FULL{\_}DONE} internally.
- Both UP and DL are considered stable, and TrbNet payloads as well as special kommas are enabled now.
+ \item \textbf{CTS MP:} the IDLE0 are received, a RX reset sequence is performed and a lock
+ to WAP=0 made. This will set \texttt{LINK{\_}RX{\_}READY} in the master port, and change the IDLE0
+ in the uplink to IDLE1.
+ \item \textbf{HUB SP:} The RX LSM detects a series of IDLE1, and sets \texttt{LINK{\_}FULL{\_}DONE}
+ internally. The TX control is instructed to send IDLE1 now. Both UP and DL are considered stable,
+ and TrbNet payloads (as well as special kommas) can be sent and received now.
+ \item \textbf{CTS MP:} The RX LSM detects a series of IDLE1, and sets to \texttt{LINK{\_}FULL{\_}DONE}
+ internally. Both UP and DL are considered stable, and TrbNet payloads as well as special kommas are
+ enabled now.
\end{itemize*}
To faciliate setups where electrical connections are used for the link (like on the TRB3sc crate backplane)
all SerDes TX channels send out K{\_}NULL kommas while no stable clock is available. This will inhibit komma
locking inside the WordAlignment block.
+In case CTC is to be used (which turns synchronous TRBnet basically useless), make sure that IDLE1 is used as
+replacement mask. Using CTC during the link establishment is not possible due to different IDLE0 being used,
+but taking the short duration of link establishment into account, this is not a real problem (in fact, if you
+need CTC during a 10s phase, you have a different problem).
\subsection{New resets}
+\subsubsection{Distributed resets}
+
Resets will be handled differently.
\begin{itemize*}
As noted before, a master reset will be included in the new TrbNet.
-Any master port in the system will keep its SFP transmitter in "turned off" mode while
+Any MP in the system will keep its SFP transmitter in "turned off" mode while
being in reset, and can also control it for sending a hard reset to all boards connected.
The current reset sequence (handled by a timed broadcast of RST kommas (0xfe) will be
replaced by a much simpler scheme: a reset komma will consist of one RST komma character,
-followed by one data byte. RST kommas are unconditionally forwarded from any slave port
-to all master ports available, turning them into a broadcast komma.
+followed by one data byte. RST kommas are unconditionally forwarded from any SP
+to all MPs available, turning them into a broadcast komma.
The RX control state machine will simply set a register with reception of a RST komma, and
provides up to 8 independent reset lines, which should be assigned to dedicated groups of
\caption{Reset Usage}
\end{center}
\end{table}
+
+\subsubsection{Reset philosophy}
+
+Handling resets on the FPGA in a synchronous network is tricky. To accomodate for the clock
+distriubtion, some changes had to be made. A general overview on how resets are handled is
+given here.
+
+The main source of reset is the PLL handling internal / external clock for the fabric.
+On TRB3sc the external clock is probed, and if available, selected as main clock source.
+While this process is ongoing, \texttt{GSR{\_}N} is set to generate the top level entity
+\texttt{RESET} signal.
+
+This \texttt{RESET} signal must not be used to reset any of the new entities responsible for
+the link establishment. These entities are running free, and will contribute to the general reset
+if needed, so deadlocks may occur if they are connected to \texttt{RESET}.
+
+The SFP link contributes by the \texttt{SFP{\_}LOS} signal: as long as no signal is sensed by
+the SFP, the \texttt{SFP{\_}LOS} signal will set the \texttt{GSR{\_}N} signal, keeping
+fabric logic in reset. A loss of signal on the SFP will always reset the FPGA in case a SP
+is present.
+
+In any FPGA containing a SP all QUADs need to be reset with loss of signal of the uplink.
+As the recovered RX clock will not be available, a safe restart of the TX PLLs is needed.
+
+Link re-establishment will be initiated on a loss of signal, and, in addition, by reseting the
+TX PLL LOL signal. A dedicated reset input is available on the SerDes (\texttt{tx{\_}serdes{\_}rst{\_}c}),
+the generic \texttt{rst{\_}qd{\_}c} signal must not be used, as this will destroy RX channel sync.
+
jspc29 / daqdocu.git / commitdiff