--- /dev/null
+The bridge offers two high-level features:
+it enables TRB3 installations to tunnel their complete ``TrbNet data stream'' via CBMNet and send it to the FLES..
+Data of all CBMNet and TrbNet frontends is therefore available from a common source and can be processed for online monitoring and analysis.
+The bridge additionally provides the means to synchronise both networks, i.e. to achieve a fixed delay between two networks that remains during operation and after a restart of the DAQ.
+
+\subsection{Synthesising the Bridge}
+The bridge is currently available for the CTS design only. In order to build it, you have to obtain a copy of the CBMNet source:
+\url{https://subversion.gsi.de/cbmsoft/firmware/trunk}
+
+Place the \texttt{cbmnet} folder in the same directory the \texttt{trb3} and \texttt{trbnet} repositories reside.
+
+To active the CBMNet module, make sure the CTS's \texttt{config.vhd} contains to following settings:
+
+\begin{lstlisting}[label=lst:cbmnet_config,language=VHDL,caption={Configuration required for building CBMNet bridge with CTS}]
+constant INCLUDE_CTS : integer range c_NO to c_YES := c_YES;
+constant INCLUDE_CBMNET : integer range c_NO to c_YES := c_YES;
+constant USE_4_SFP : integer range c_NO to c_YES := c_NO;
+constant INCLUDE_ETM : integer range c_NO to c_YES := c_NO;
+
+-- optional if, TDC messurements are required
+constant INCLUDE_TDC : integer range c_NO to c_YES := c_YES;
+constant TDC_CHANNEL_NUMBER : integer := 5;
+\end{lstlisting}
+
+\subsection{Read-Out via CBMNet}
+By default the bridge listens on the Streaming API Bus between hub and Gbe and ignores all data.
+To active the uplink, set bit Listen of the ReadoutCtrl register (see XML-DB).
+The bridge will now try to uplink TrbNet events via CBMNet.
+If the FLES invokes backpressure, the events are discarded and no pressure is put onto TrbNet.
+
+By deselecting the EnableGBE bit in the ReadoutCtrl register, GBE does not receive data anymore and the possible CBMNet uplink rate will increase.
+Even in this configuration, the bridge refrains from forwarding backpressure into TrbNet.
+Thus, events still can get lost (around 1 to 10 ppm during for high data rates).
+
+An unpacker for the CBMNet bridge is included into CbmRoot; this section is hence only included for completeness.
+The CBMNet bridge currently supports sub-events with up-to 64~KB of size and hence has to split the packet into multiple CBMNet frames.
+The transmission of a single sub-event is called a transaction.
+A transaction can only be started once a pending transaction is completed and cannot be gracefully stopped.
+Due to CBMNet's strong data integrity guarantees, no redundancy or recovery scheme was implemented; if a single error is detected, the whole transaction should be discarded.
+CBMNet v3 discards corrupt frames without retransmission.
+While this should happen very rarely (less than $\ll 1$ error in $10^7$ frames), current tests show, much higher error rates.
+
+A transaction consists of one or more CBMNet frames containing a single 16~bit header followed by up to 62~byte payload.
+No routing extensions are used (as inherently supported by TrbNet).
+
+\begin{figure}[b]
+ \includegraphics[width=1\textwidth]{figures/cbmnet_frames.pdf}
+ \caption{Two frame header types employed by the CBMNet Bridge. On top: A start- or stop-frame header. On bottom: An inner frame's header}
+ \label{fig:cbmnet_frame_header}
+\end{figure}
+
+
+As shown in Figure \ref{fig:cbmnet_frame_header}, there are two types of frame headers \texttt{uint16\_t hdr}.
+In both the MSB \texttt{(hdr>>15)\&1} indicates the last frame of a transaction, while \texttt{(hdr>>14)\&1} marks the start frame.
+Note, that both bits are set, iff the transaction consists of a single frame.
+
+\begin{itemize}
+ \item
+ \texttt{(hdr >> 14) \& 3 != 0}: Start- or stop frame.\\
+ The lower 12 bit store a sequential transaction number \texttt{hdrTransNo = hdr \& 0xfff} intended to detect data loss:
+ The bridge increments the number with each new transaction; if you detect a mismatch, it happened somewhere along the way.
+
+ \item
+ \texttt{(hdr >> 14) \& 3 == 0}: Inner rame.\\
+ \texttt{frameTransNo = (hdr >> 7) \& 0x1f} stores the lower 5~bits of the transaction number and \texttt{frameNo = hdr \& 0x7f} holds a 7~bits sequential frame number.
+ The transaction number does not change within a pending transaction, i.e. \\ \texttt{assert(frameTransNo == hdrTransNo \& 0x1f)}.
+ The start frame has the (omitted) frame number \texttt{frameNo = 0}, i.e. the first inner frame has \texttt{frameNo = 1}.
+\end{itemize}
+
+The bridge transmits all frames in order, so no sorting is necessary on the receiving side.
+Frame numbers only provide the means to detect transmission errors.
+
+For consistency reasons, the TrbNet's 32~bit words\footnote{In this document, a word is 4 bytes long. In this notation, CBMNet has a half-word granularity.} are sent MSB first in two half-words via CBMNet, i.e. big-endian.
+The FLIB then represents the half-words in little-endian, leading to a middle-endian representation of the word that cannot be dealt with by the standard HLD unpackers.
+It is therefore advisable, to switch the half-words forming a TrbNet word and thus to obtain a complete little-endian representation.
+
+All frames but the last one contain exactly 31 half-words payload; only the stop frame may be shorter.
+Note, that the last frame can contain as little as 1 half-word as payload (incl. header: 4 bytes).
+In order to satisfy CBMNet's minimal frame length of 8 byte, up to 2 half-words of padding \texttt{0xaaaa} may be appended by the bridge.
+Padding can be detected by considering the length information (in bytes, incl. all headers) stored in the first word of the sub-event header, i.e. the first word of payload in the transaction.
+
+\subsection{\change{Synchronisation with CBMNet}}
+In order to express the TrbNet reference time in CBMNet time using a TDC the following approach is suitable:
+For every physics trigger the CTS provides a reference time signal, that is distributed in the whole network using a dedicated physical line.
+The arrival time of this signal gives the means to synchronise the system and gives a $T_0$ for each event .
+
+The reference time is also sampled in the 125~MHz CBMNet clock domain.
+Its leading edge triggers the storage of the CBMNet timestamp (1 word, incremented with each clock cycle), yielding a granularity of 8~ns.
+To increase the accuracy, the moment of timestamping is measured using TDC channel 3.
+Hence, the accurate time is given by
+\[
+ T^\text{event}_\text{cbm} = 8\text{~ns} \cdot T^\text{timestamp}_\text{cbm} - (T^\text{ref}_\text{cbm} - T^\text{ref}_\text{trb}),
+\]
+where $T^\text{timestamp}_\text{cbm}$ is given in the sync-module (2nd word), $T^\text{ref}_\text{cbm}$ is provided by TDC-channel 3 and $T^\text{ref}_\text{trb}$ by TDC-channel 0.
+The difference between the two TDC measurements should be in the order of $16$~ns to $32$~ns.
+In lab tests, the RMS jitter $j$ of the $T^\text{event}_\text{cbm}$ measurement was found the be $40\text{~ps} < j < 60\text{~ps}$.
+
+The Sync-Module can be configured to receive any DLM (but DLM0 which is reserved for CBMNet purposes).
+After enabling the $i$-th bit in the DLMSenseMask (see XML-DB), the sync-module begins listening to the DLMs of type $i$ and store the arrival with with a CBMNet timestamp (see read-out format).
+Additionally a TDC hit is registers for any active DLM; this, however, should not be required when using the aforementioned approach based on the TrbNet reference time.
+
+\subsubsection{Read-Out Format}
+The CTS-sub-event contains data from three sub-systems in the following order: CTS, CBM Synchronisation and TDC.
+Each component has its own data structure and must be dealt with individually in order to find its length.
+
+The CBM Synchronisation Module measures the relationship between the CBMNet- and TRBNet clock domains only with synchronous logic (in contrast to the TDCs).
+The first word of the data section contains the header \texttt{hdr} that indicates which data follows:
+\begin{itemize}
+ \item \texttt{(hdr >> 26) \& 0x3 == 0} -- \textsc{NoData}: section's length (incl. header): 4 bytes\\
+ The CBMNet link is not active, so no data follows.
+ \item \texttt{(hdr >> 26) \& 0x3 == 3} -- \textsc{ExtendedData}: section's length (incl. header): 44 bytes\\
+ This type is transmitted for all timining-less triggers (i.e. status triggers). The header is followed by 10 words
+ as shown in Table \ref{tab:cbmnet_sync_mod_rdo}.
+ \item \texttt{(hdr >> 26) \& 0x3 == 1} -- \textsc{ShortData}: section's length (incl. header): 20 bytes\\
+ All physics trigger include only a shortened version, i.e. the first four data words of the \textsc{ExtendedData} format (Timestamps \& Pulser Timestamp).
+\end{itemize}
+
+The TDC channels are mapped as follows:
+
+\begin{enumerate}
+ \item[channel 0:] TrbNet Timing Trigger
+ \item[channel 1:] CBM Pulser generate by sync. module. (simple pulser operating at CBMNet clock with a configurable period.)
+ \item[channel 2:] CBM DLM received
+ \item[channel 3:] TrbNet Timing Trigger sampled in the CBMNet clock domain
+ \item[channel 4:] DLM timing reference provided by FLIB for debugging (n.c. in beam-time!)
+\end{enumerate}
+
+\begin{figure}
+ \includegraphics{figures/cbmnet_sync_chain.pdf}
+ \caption{The TrbNet timing reference is sampled in the CBMNet clock domain using an ordinary register chain to avoid meta-stability.
+ The arrival time is expressed in terms of the CBMNet-clock yielding an accuracy of $8~$ns.
+ To increase the accuracy, the original- and the sampled strobe signals are measured in the TrbNet domain using two TDC channels. }
+ \label{fig:cbmnet_sync_chain}
+\end{figure}
+
+\begin{table}
+ \small
+ \begin{tabular}{ l c p{12.5cm} }
+ \toprule
+ \textbf{Addr} & \textbf{Bits} & \textbf{Description} \\
+ \midrule
+ \midrule
+ 0 & & \textbf{Header} \\
+ & 1--0 & Epoch ID defined by: \\
+ & & \hspace{1em} 0x0: Incremented epoch \\
+ & & \hspace{1em} 0x1: Preset via TrbNet slow control \\
+ & & \hspace{1em} 0x2: Preset via CBMNet \gls{dcm} \\
+ & 3 & Low, if slow control update was chosen and no update between last two \gls{dlm} \\
+ & 7--4 & Barrel shifter position of CBMNet \gls{phy} (always 0 in default config) \\
+ & 23--8& Lower half-word of CBMNet pulser threshold (in clock cycles) \\
+ & 26 & CBMNet link active \\
+ & 27 & Extended Data \\
+ &31--28& Header version (= 0x1) \\
+ \rowcolor{light-gray}
+ 1 &31--0& Timestamp (TrbNet clock) \\
+ 2 &31--0& Timestamp (CBMNet clock) \\
+ \rowcolor{light-gray}
+ 3 &31--0& Timestamp of last pulse (TrbNet clock) \\
+ 4 &31--0& Timestamp of last pulse (CBMNet clock) \\
+ \rowcolor{light-gray}
+ 5 &31--0& Current epoch \\
+ 6 &31--0& Timestamp of last \gls{dlm} (TrbNet clock) \\
+ \rowcolor{light-gray}
+ 7 &31--0& Timestamp of last \gls{dlm} (CBMNet clock) \\
+ 8 &31--0& Number of \gls{dlm} received \\
+ \rowcolor{light-gray}
+ 9 &31--0& Number of Pulser pulses \\
+ 10 & & \textbf{Reset counter}\\
+ &15--0& Number of CBMNet resets since \gls{fpga} start-up \\
+ &31--16& Number of TrbNet resets since \gls{fpga} start-up \\
+ \bottomrule
+ \end{tabular}
+ \caption{CBMNet synchronisation module read-out format.}
+ \label{tab:cbmnet_sync_mod_rdo}
+\end{table}
\ No newline at end of file
jspc29 / daqdocu.git / commitdiff