From: Manuel Penschuck Date: Wed, 7 Jan 2015 14:12:05 +0000 (+0100) Subject: Added CBMNet bridge docu X-Git-Url: https://jspc29.x-matter.uni-frankfurt.de/git/?a=commitdiff_plain;h=97cabb780c47bb9400a6609ba5b1d19cd3a06fc4;p=daqdocu.git Added CBMNet bridge docu --- diff --git a/trb3/CBMNetBridge.tex b/trb3/CBMNetBridge.tex new file mode 100644 index 0000000..79bd015 --- /dev/null +++ b/trb3/CBMNetBridge.tex @@ -0,0 +1,175 @@ +The bridge offers two high-level features: +it enables TRB3 installations to tunnel their complete ``TrbNet data stream'' via CBMNet and send it to the FLES.. +Data of all CBMNet and TrbNet frontends is therefore available from a common source and can be processed for online monitoring and analysis. +The bridge additionally provides the means to synchronise both networks, i.e. to achieve a fixed delay between two networks that remains during operation and after a restart of the DAQ. + +\subsection{Synthesising the Bridge} +The bridge is currently available for the CTS design only. In order to build it, you have to obtain a copy of the CBMNet source: +\url{https://subversion.gsi.de/cbmsoft/firmware/trunk} + +Place the \texttt{cbmnet} folder in the same directory the \texttt{trb3} and \texttt{trbnet} repositories reside. + +To active the CBMNet module, make sure the CTS's \texttt{config.vhd} contains to following settings: + +\begin{lstlisting}[label=lst:cbmnet_config,language=VHDL,caption={Configuration required for building CBMNet bridge with CTS}] +constant INCLUDE_CTS : integer range c_NO to c_YES := c_YES; +constant INCLUDE_CBMNET : integer range c_NO to c_YES := c_YES; +constant USE_4_SFP : integer range c_NO to c_YES := c_NO; +constant INCLUDE_ETM : integer range c_NO to c_YES := c_NO; + +-- optional if, TDC messurements are required +constant INCLUDE_TDC : integer range c_NO to c_YES := c_YES; +constant TDC_CHANNEL_NUMBER : integer := 5; +\end{lstlisting} + +\subsection{Read-Out via CBMNet} +By default the bridge listens on the Streaming API Bus between hub and Gbe and ignores all data. +To active the uplink, set bit Listen of the ReadoutCtrl register (see XML-DB). +The bridge will now try to uplink TrbNet events via CBMNet. +If the FLES invokes backpressure, the events are discarded and no pressure is put onto TrbNet. + +By deselecting the EnableGBE bit in the ReadoutCtrl register, GBE does not receive data anymore and the possible CBMNet uplink rate will increase. +Even in this configuration, the bridge refrains from forwarding backpressure into TrbNet. +Thus, events still can get lost (around 1 to 10 ppm during for high data rates). + +An unpacker for the CBMNet bridge is included into CbmRoot; this section is hence only included for completeness. +The CBMNet bridge currently supports sub-events with up-to 64~KB of size and hence has to split the packet into multiple CBMNet frames. +The transmission of a single sub-event is called a transaction. +A transaction can only be started once a pending transaction is completed and cannot be gracefully stopped. +Due to CBMNet's strong data integrity guarantees, no redundancy or recovery scheme was implemented; if a single error is detected, the whole transaction should be discarded. +CBMNet v3 discards corrupt frames without retransmission. +While this should happen very rarely (less than $\ll 1$ error in $10^7$ frames), current tests show, much higher error rates. + +A transaction consists of one or more CBMNet frames containing a single 16~bit header followed by up to 62~byte payload. +No routing extensions are used (as inherently supported by TrbNet). + +\begin{figure}[b] + \includegraphics[width=1\textwidth]{figures/cbmnet_frames.pdf} + \caption{Two frame header types employed by the CBMNet Bridge. On top: A start- or stop-frame header. On bottom: An inner frame's header} + \label{fig:cbmnet_frame_header} +\end{figure} + + +As shown in Figure \ref{fig:cbmnet_frame_header}, there are two types of frame headers \texttt{uint16\_t hdr}. +In both the MSB \texttt{(hdr>>15)\&1} indicates the last frame of a transaction, while \texttt{(hdr>>14)\&1} marks the start frame. +Note, that both bits are set, iff the transaction consists of a single frame. + +\begin{itemize} + \item + \texttt{(hdr >> 14) \& 3 != 0}: Start- or stop frame.\\ + The lower 12 bit store a sequential transaction number \texttt{hdrTransNo = hdr \& 0xfff} intended to detect data loss: + The bridge increments the number with each new transaction; if you detect a mismatch, it happened somewhere along the way. + + \item + \texttt{(hdr >> 14) \& 3 == 0}: Inner rame.\\ + \texttt{frameTransNo = (hdr >> 7) \& 0x1f} stores the lower 5~bits of the transaction number and \texttt{frameNo = hdr \& 0x7f} holds a 7~bits sequential frame number. + The transaction number does not change within a pending transaction, i.e. \\ \texttt{assert(frameTransNo == hdrTransNo \& 0x1f)}. + The start frame has the (omitted) frame number \texttt{frameNo = 0}, i.e. the first inner frame has \texttt{frameNo = 1}. +\end{itemize} + +The bridge transmits all frames in order, so no sorting is necessary on the receiving side. +Frame numbers only provide the means to detect transmission errors. + +For consistency reasons, the TrbNet's 32~bit words\footnote{In this document, a word is 4 bytes long. In this notation, CBMNet has a half-word granularity.} are sent MSB first in two half-words via CBMNet, i.e. big-endian. +The FLIB then represents the half-words in little-endian, leading to a middle-endian representation of the word that cannot be dealt with by the standard HLD unpackers. +It is therefore advisable, to switch the half-words forming a TrbNet word and thus to obtain a complete little-endian representation. + +All frames but the last one contain exactly 31 half-words payload; only the stop frame may be shorter. +Note, that the last frame can contain as little as 1 half-word as payload (incl. header: 4 bytes). +In order to satisfy CBMNet's minimal frame length of 8 byte, up to 2 half-words of padding \texttt{0xaaaa} may be appended by the bridge. +Padding can be detected by considering the length information (in bytes, incl. all headers) stored in the first word of the sub-event header, i.e. the first word of payload in the transaction. + +\subsection{\change{Synchronisation with CBMNet}} +In order to express the TrbNet reference time in CBMNet time using a TDC the following approach is suitable: +For every physics trigger the CTS provides a reference time signal, that is distributed in the whole network using a dedicated physical line. +The arrival time of this signal gives the means to synchronise the system and gives a $T_0$ for each event . + +The reference time is also sampled in the 125~MHz CBMNet clock domain. +Its leading edge triggers the storage of the CBMNet timestamp (1 word, incremented with each clock cycle), yielding a granularity of 8~ns. +To increase the accuracy, the moment of timestamping is measured using TDC channel 3. +Hence, the accurate time is given by +\[ + T^\text{event}_\text{cbm} = 8\text{~ns} \cdot T^\text{timestamp}_\text{cbm} - (T^\text{ref}_\text{cbm} - T^\text{ref}_\text{trb}), +\] +where $T^\text{timestamp}_\text{cbm}$ is given in the sync-module (2nd word), $T^\text{ref}_\text{cbm}$ is provided by TDC-channel 3 and $T^\text{ref}_\text{trb}$ by TDC-channel 0. +The difference between the two TDC measurements should be in the order of $16$~ns to $32$~ns. +In lab tests, the RMS jitter $j$ of the $T^\text{event}_\text{cbm}$ measurement was found the be $40\text{~ps} < j < 60\text{~ps}$. + +The Sync-Module can be configured to receive any DLM (but DLM0 which is reserved for CBMNet purposes). +After enabling the $i$-th bit in the DLMSenseMask (see XML-DB), the sync-module begins listening to the DLMs of type $i$ and store the arrival with with a CBMNet timestamp (see read-out format). +Additionally a TDC hit is registers for any active DLM; this, however, should not be required when using the aforementioned approach based on the TrbNet reference time. + +\subsubsection{Read-Out Format} +The CTS-sub-event contains data from three sub-systems in the following order: CTS, CBM Synchronisation and TDC. +Each component has its own data structure and must be dealt with individually in order to find its length. + +The CBM Synchronisation Module measures the relationship between the CBMNet- and TRBNet clock domains only with synchronous logic (in contrast to the TDCs). +The first word of the data section contains the header \texttt{hdr} that indicates which data follows: +\begin{itemize} + \item \texttt{(hdr >> 26) \& 0x3 == 0} -- \textsc{NoData}: section's length (incl. header): 4 bytes\\ + The CBMNet link is not active, so no data follows. + \item \texttt{(hdr >> 26) \& 0x3 == 3} -- \textsc{ExtendedData}: section's length (incl. header): 44 bytes\\ + This type is transmitted for all timining-less triggers (i.e. status triggers). The header is followed by 10 words + as shown in Table \ref{tab:cbmnet_sync_mod_rdo}. + \item \texttt{(hdr >> 26) \& 0x3 == 1} -- \textsc{ShortData}: section's length (incl. header): 20 bytes\\ + All physics trigger include only a shortened version, i.e. the first four data words of the \textsc{ExtendedData} format (Timestamps \& Pulser Timestamp). +\end{itemize} + +The TDC channels are mapped as follows: + +\begin{enumerate} + \item[channel 0:] TrbNet Timing Trigger + \item[channel 1:] CBM Pulser generate by sync. module. (simple pulser operating at CBMNet clock with a configurable period.) + \item[channel 2:] CBM DLM received + \item[channel 3:] TrbNet Timing Trigger sampled in the CBMNet clock domain + \item[channel 4:] DLM timing reference provided by FLIB for debugging (n.c. in beam-time!) +\end{enumerate} + +\begin{figure} + \includegraphics{figures/cbmnet_sync_chain.pdf} + \caption{The TrbNet timing reference is sampled in the CBMNet clock domain using an ordinary register chain to avoid meta-stability. + The arrival time is expressed in terms of the CBMNet-clock yielding an accuracy of $8~$ns. + To increase the accuracy, the original- and the sampled strobe signals are measured in the TrbNet domain using two TDC channels. } + \label{fig:cbmnet_sync_chain} +\end{figure} + +\begin{table} + \small + \begin{tabular}{ l c p{12.5cm} } + \toprule + \textbf{Addr} & \textbf{Bits} & \textbf{Description} \\ + \midrule + \midrule + 0 & & \textbf{Header} \\ + & 1--0 & Epoch ID defined by: \\ + & & \hspace{1em} 0x0: Incremented epoch \\ + & & \hspace{1em} 0x1: Preset via TrbNet slow control \\ + & & \hspace{1em} 0x2: Preset via CBMNet \gls{dcm} \\ + & 3 & Low, if slow control update was chosen and no update between last two \gls{dlm} \\ + & 7--4 & Barrel shifter position of CBMNet \gls{phy} (always 0 in default config) \\ + & 23--8& Lower half-word of CBMNet pulser threshold (in clock cycles) \\ + & 26 & CBMNet link active \\ + & 27 & Extended Data \\ + &31--28& Header version (= 0x1) \\ + \rowcolor{light-gray} + 1 &31--0& Timestamp (TrbNet clock) \\ + 2 &31--0& Timestamp (CBMNet clock) \\ + \rowcolor{light-gray} + 3 &31--0& Timestamp of last pulse (TrbNet clock) \\ + 4 &31--0& Timestamp of last pulse (CBMNet clock) \\ + \rowcolor{light-gray} + 5 &31--0& Current epoch \\ + 6 &31--0& Timestamp of last \gls{dlm} (TrbNet clock) \\ + \rowcolor{light-gray} + 7 &31--0& Timestamp of last \gls{dlm} (CBMNet clock) \\ + 8 &31--0& Number of \gls{dlm} received \\ + \rowcolor{light-gray} + 9 &31--0& Number of Pulser pulses \\ + 10 & & \textbf{Reset counter}\\ + &15--0& Number of CBMNet resets since \gls{fpga} start-up \\ + &31--16& Number of TrbNet resets since \gls{fpga} start-up \\ + \bottomrule + \end{tabular} + \caption{CBMNet synchronisation module read-out format.} + \label{tab:cbmnet_sync_mod_rdo} +\end{table} \ No newline at end of file diff --git a/trb3/figures/cbmnet_frames.pdf b/trb3/figures/cbmnet_frames.pdf new file mode 100644 index 0000000..16646f4 Binary files /dev/null and b/trb3/figures/cbmnet_frames.pdf differ diff --git a/trb3/figures/cbmnet_sync_chain.pdf b/trb3/figures/cbmnet_sync_chain.pdf new file mode 100644 index 0000000..39fddf3 Binary files /dev/null and b/trb3/figures/cbmnet_sync_chain.pdf differ diff --git a/trb3/main.tex b/trb3/main.tex index 175169f..3fb287b 100644 --- a/trb3/main.tex +++ b/trb3/main.tex @@ -239,10 +239,14 @@ \clearpage \section{Billboard} \input{Billboard.tex} + \clearpage \section{CBM-MBS Receiver} \input{CBMMBSRecv.tex} - + + \clearpage{CBMNet Bridge} + \section{CBMNet Bridge} + \input{CBMNetBridge.tex} \cleardoublepage