From aadddc4c26ace55094563d3a8fa7eac7b439014c Mon Sep 17 00:00:00 2001 From: Michael Traxler Date: Thu, 10 Sep 2020 23:19:02 +0200 Subject: [PATCH] added description for kernel and eb tuning paramenters, mt --- trb3/trb3qs_performance.tex | 45 +++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 trb3/trb3qs_performance.tex diff --git a/trb3/trb3qs_performance.tex b/trb3/trb3qs_performance.tex new file mode 100644 index 0000000..0093354 --- /dev/null +++ b/trb3/trb3qs_performance.tex @@ -0,0 +1,45 @@ +\section{Performance and Kernel and Eventbuilder Tweaks} + +When a lot of data is sent to the eventbuilder it can happen that data is lost +in the kernel-network-buffers and in the eventbuilder. + +To avoid that, there are some simple kernel and eventbuilder tweaks. + +First, the kernel should use larger IP receive queues (e.g. 10 times larger +than the default).. +Second, when many sources are used and multieventqueues are used to transport the data +from the FEE to the eventbuilder, the memory which is used to reassemble the +UDP packets from the IP-fragments has to be increased (e.g. 10 times more than +default, which is 212992), also 100 times larger is sometimes used. +Third, when (depending on the machine) the data rates are > 400 MBytes/s the +kernel has to be instructed that UDP packets have to reassembled, even if the +distance of the arrival of the packets is quite large (e.g. 1024). + +This can be enabled via to following commands: + +\begin{lstlisting}[label=,caption=Larger Receive Queues] +sysctl -w net.core.rmem_max=2129920 +sysctl -w net.core.rmem_default=2129920 + +sysctl -w net.ipv4.ipfrag_high_thresh=41943040 +sysctl -w net.ipv4.ipfrag_max_dist=1024 + +\end{lstlisting} + +When dealing with extreme data rates differences from different data sources +to the eventbuilder, also the memory for the input buffers of the eventbuilder +has to be increased to avoid loss of packets. +Also the flush time of the buffers mustn't be to large, 0.2s is a often used value +(the combination of the pool-size and the flush time needs to fit to factor of +the data rate discrepancy of the various inputs). +If still data is discarded go up with the pool size by some factors. + +So, the following has to be increased in the .xml file for the eventbuilder + +\begin{lstlisting}[label=,caption=Event Builder Input Buffers] + + + + + +\end{lstlisting} -- 2.43.0