From 9aa50f8f6eb7be7ad01afc662ceda7c7e54af8d8 Mon Sep 17 00:00:00 2001 From: Michael Boehmer Date: Tue, 21 Dec 2021 08:52:16 +0100 Subject: [PATCH] updated Link Establishment, added reset stuff --- trb3/DS_LinkEstablishment.tex | 40 ++++++++++++++++++++--------------- trb3/DS_NewResets.tex | 36 ++++++++++++++++++++++++++++--- 2 files changed, 56 insertions(+), 20 deletions(-) diff --git a/trb3/DS_LinkEstablishment.tex b/trb3/DS_LinkEstablishment.tex index a2730c6..3cf4c31 100644 --- a/trb3/DS_LinkEstablishment.tex +++ b/trb3/DS_LinkEstablishment.tex @@ -26,28 +26,34 @@ The link establishment process is explained on a hub FPGA slave port as example: \item The hub FPGA is in reset while no fiber is connected or the master port it is connected to has its \texttt{TX{\_}DIS} line set. \item \textbf{CTS MP:} the \texttt{TX{\_}DIS} is released once the SerDes TX channel is ready, - and a stream of IDLE kommas is sent to initiate the link. - \item \textbf{HUB SP:} A stream of IDLE kommas is received, so the RX reset sequence is completed + and a stream of IDLE0 kommas is sent to initiate the link. + \item \textbf{HUB SP:} A stream of IDLE0 kommas is received, so the RX reset sequence is completed and a lock to incoming kommas by the FPGA internal state machine is made. - \item \textbf{HUB SP:} Once WAP=0 is achieved, the RX reset control will declare the RX recovered clock stable, - and initiate the TX reset sequence. \texttt{LINK{\_}RX{\_}READY} is set, while toggling IDLEs are received, - the uplink is stable now. - \item \textbf{HUB SP:} The TX reset sequence waits for all QUADs in use to lock on the TX reference clock (i.e. - the recovered RX clock from slave port), and sync all TX Serializers to bit 0. \texttt{LINK{\_}TX{\_}READY} is set. - \item \textbf{HUB SP:} The TX control sends a togglinge idle sequence after the downlink is stable. - \item \textbf{HUB SP:} The RX LSM will check for a sequence of toggling IDLEs, and after detection set the + \item \textbf{HUB SP:} Once WAP=0 is achieved, the RX reset control will declare the RX recovered + clock stable, and initiate the TX reset sequence. \texttt{LINK{\_}RX{\_}READY} is set, while + IDLE0 are received, the uplink is stable now. + \item \textbf{HUB SP:} The TX reset sequence waits for all QUADs in use to lock on the TX reference + clock (i.e. the recovered RX clock from slave port), and sync all TX Serializers to bit 0. + \texttt{LINK{\_}TX{\_}READY} is set. + \item \textbf{HUB SP:} The TX control sends IDLE0 after the downlink is stable. + \item \textbf{HUB SP:} The RX LSM will check for IDLE0, and after detection set the \texttt{LINK{\_}HALF{\_}DONE} signal. - \item \textbf{CTS MP:} the toggling IDLEs are received, a RX reset sequence is performed and a lock - to WAP=0 made. This will set \texttt{LINK{\_}RX{\_}READY} in the master port, and change the toggling IDLEs in the uplink - to steady IDLEs (IDLE1). - \item \textbf{HUB SP:} The RX LSM detects a series of steady IDLEs, and sets \texttt{LINK{\_}FULL{\_}DONE} internally. - The TX control is instructed to send steady IDLEs now. Both UP and DL are considered stable, and TrbNet - payloads (as well as special kommas) can be sent and received now. - \item \textbf{CTS MP:} The RX LSM detects a series of steady IDLEs, and sets to \texttt{LINK{\_}FULL{\_}DONE} internally. - Both UP and DL are considered stable, and TrbNet payloads as well as special kommas are enabled now. + \item \textbf{CTS MP:} the IDLE0 are received, a RX reset sequence is performed and a lock + to WAP=0 made. This will set \texttt{LINK{\_}RX{\_}READY} in the master port, and change the IDLE0 + in the uplink to IDLE1. + \item \textbf{HUB SP:} The RX LSM detects a series of IDLE1, and sets \texttt{LINK{\_}FULL{\_}DONE} + internally. The TX control is instructed to send IDLE1 now. Both UP and DL are considered stable, + and TrbNet payloads (as well as special kommas) can be sent and received now. + \item \textbf{CTS MP:} The RX LSM detects a series of IDLE1, and sets to \texttt{LINK{\_}FULL{\_}DONE} + internally. Both UP and DL are considered stable, and TrbNet payloads as well as special kommas are + enabled now. \end{itemize*} To faciliate setups where electrical connections are used for the link (like on the TRB3sc crate backplane) all SerDes TX channels send out K{\_}NULL kommas while no stable clock is available. This will inhibit komma locking inside the WordAlignment block. +In case CTC is to be used (which turns synchronous TRBnet basically useless), make sure that IDLE1 is used as +replacement mask. Using CTC during the link establishment is not possible due to different IDLE0 being used, +but taking the short duration of link establishment into account, this is not a real problem (in fact, if you +need CTC during a 10s phase, you have a different problem). diff --git a/trb3/DS_NewResets.tex b/trb3/DS_NewResets.tex index 5f78c59..2dc0d52 100644 --- a/trb3/DS_NewResets.tex +++ b/trb3/DS_NewResets.tex @@ -1,5 +1,7 @@ \subsection{New resets} +\subsubsection{Distributed resets} + Resets will be handled differently. \begin{itemize*} @@ -11,13 +13,13 @@ Resets will be handled differently. As noted before, a master reset will be included in the new TrbNet. -Any master port in the system will keep its SFP transmitter in "turned off" mode while +Any MP in the system will keep its SFP transmitter in "turned off" mode while being in reset, and can also control it for sending a hard reset to all boards connected. The current reset sequence (handled by a timed broadcast of RST kommas (0xfe) will be replaced by a much simpler scheme: a reset komma will consist of one RST komma character, -followed by one data byte. RST kommas are unconditionally forwarded from any slave port -to all master ports available, turning them into a broadcast komma. +followed by one data byte. RST kommas are unconditionally forwarded from any SP +to all MPs available, turning them into a broadcast komma. The RX control state machine will simply set a register with reception of a RST komma, and provides up to 8 independent reset lines, which should be assigned to dedicated groups of @@ -44,3 +46,31 @@ Proposed reset bit usage: \caption{Reset Usage} \end{center} \end{table} + +\subsubsection{Reset philosophy} + +Handling resets on the FPGA in a synchronous network is tricky. To accomodate for the clock +distriubtion, some changes had to be made. A general overview on how resets are handled is +given here. + +The main source of reset is the PLL handling internal / external clock for the fabric. +On TRB3sc the external clock is probed, and if available, selected as main clock source. +While this process is ongoing, \texttt{GSR{\_}N} is set to generate the top level entity +\texttt{RESET} signal. + +This \texttt{RESET} signal must not be used to reset any of the new entities responsible for +the link establishment. These entities are running free, and will contribute to the general reset +if needed, so deadlocks may occur if they are connected to \texttt{RESET}. + +The SFP link contributes by the \texttt{SFP{\_}LOS} signal: as long as no signal is sensed by +the SFP, the \texttt{SFP{\_}LOS} signal will set the \texttt{GSR{\_}N} signal, keeping +fabric logic in reset. A loss of signal on the SFP will always reset the FPGA in case a SP +is present. + +In any FPGA containing a SP all QUADs need to be reset with loss of signal of the uplink. +As the recovered RX clock will not be available, a safe restart of the TX PLLs is needed. + +Link re-establishment will be initiated on a loss of signal, and, in addition, by reseting the +TX PLL LOL signal. A dedicated reset input is available on the SerDes (\texttt{tx{\_}serdes{\_}rst{\_}c}), +the generic \texttt{rst{\_}qd{\_}c} signal must not be used, as this will destroy RX channel sync. + -- 2.43.0