From b66ae4272398fcf5c2fd08cee2a25fd203895b58 Mon Sep 17 00:00:00 2001 From: andsel Date: Tue, 25 Jul 2023 17:19:12 +0200 Subject: [PATCH 1/3] First draft to describe how to size the native memory used by Beats input to decode a batch of events. --- docs/index.asciidoc | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/index.asciidoc b/docs/index.asciidoc index d1721ed5..ddb3415b 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -100,6 +100,28 @@ Setting direct memory too low decreases the performance of ingestion. NOTE: Be sure that heap and direct memory combined does not exceed the total memory available on the server to avoid an OutOfDirectMemoryError +[id="plugins-{type}s-{plugin}-memory-sizing"] +===== How to size the direct memory used + +To correctly size the direct memory to sustain the flow of incoming Beats connections, the medium size of the transmitted +log lines has to be known and also the batch size used by Beats (default to 2048). Overall the connections, only a +subset of them are actively processed in parallel by Netty, corresponding to the number of workers which equals the +number of CPU cores available. For each under processing channel a batch of events is read and due to the way +the decompressing and decoding part works, it keeps two copies of the batch in memory. +The expression used to calculate the maximum direct memory usage is: +["source","text"] +----- +event size * batch size * 2 * netty workers +----- + +Supposing a 1Kb event size, there a small overhead of ~500 bytes of metadata transferred, considering 12 core CPU the memory +consumption could be estimated as: +["source","text"] +----- +1,5 KB * 2048 * 2 * 12 +----- +This totalling to about 140MB. So if you have some data about the medium size of the events to process you can size +the memory accordingly without risking to go in Out-Of-Memory error on the direct memory space in production environment. //Content for Beats ifeval::["{plugin}"=="beats"] From ecaf36a19ed1b8bc875599a977497ddacc2b496e Mon Sep 17 00:00:00 2001 From: andsel Date: Thu, 27 Jul 2023 08:30:25 +0200 Subject: [PATCH 2/3] Minor, fixed number of Netty event loop threads count --- docs/index.asciidoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/index.asciidoc b/docs/index.asciidoc index ddb3415b..1026b6c4 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -106,7 +106,7 @@ NOTE: Be sure that heap and direct memory combined does not exceed the total mem To correctly size the direct memory to sustain the flow of incoming Beats connections, the medium size of the transmitted log lines has to be known and also the batch size used by Beats (default to 2048). Overall the connections, only a subset of them are actively processed in parallel by Netty, corresponding to the number of workers which equals the -number of CPU cores available. For each under processing channel a batch of events is read and due to the way +number of CPU cores available multiplied by 2. For each under processing channel a batch of events is read and due to the way the decompressing and decoding part works, it keeps two copies of the batch in memory. The expression used to calculate the maximum direct memory usage is: ["source","text"] @@ -118,9 +118,9 @@ Supposing a 1Kb event size, there a small overhead of ~500 bytes of metadata tra consumption could be estimated as: ["source","text"] ----- -1,5 KB * 2048 * 2 * 12 +1,5 KB * 2048 * 2 * 24 ----- -This totalling to about 140MB. So if you have some data about the medium size of the events to process you can size +This totalling to about 280MB. So if you have some data about the medium size of the events to process you can size the memory accordingly without risking to go in Out-Of-Memory error on the direct memory space in production environment. //Content for Beats From 015d2e00694228d9b1b302d546939c55a81a5361 Mon Sep 17 00:00:00 2001 From: andsel Date: Mon, 25 Sep 2023 12:29:46 +0200 Subject: [PATCH 3/3] Fixed description, is not the number of Netty event loops that determine how many batch in flight we have, but the number of connected channels. --- docs/index.asciidoc | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 1026b6c4..6a8f6313 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -104,25 +104,28 @@ NOTE: Be sure that heap and direct memory combined does not exceed the total mem ===== How to size the direct memory used To correctly size the direct memory to sustain the flow of incoming Beats connections, the medium size of the transmitted -log lines has to be known and also the batch size used by Beats (default to 2048). Overall the connections, only a -subset of them are actively processed in parallel by Netty, corresponding to the number of workers which equals the -number of CPU cores available multiplied by 2. For each under processing channel a batch of events is read and due to the way -the decompressing and decoding part works, it keeps two copies of the batch in memory. -The expression used to calculate the maximum direct memory usage is: +log lines and the batch size used by Beats (default to 2048), has to be known. For each connected client, a batch of events +is read and due to the way the decompressing and decoding part works, it keeps two copies of the batch in memory. +The expression used to calculate the maximum direct memory is: ["source","text"] ----- -event size * batch size * 2 * netty workers +event size * batch size * 2 * beat clients ----- -Supposing a 1Kb event size, there a small overhead of ~500 bytes of metadata transferred, considering 12 core CPU the memory -consumption could be estimated as: +Supposing a 1Kb event size, there is a small overhead of ~500 bytes of metadata transferred, considering 1000 connected clients, +the maximum memory needed could be estimated as: ["source","text"] ----- -1,5 KB * 2048 * 2 * 24 +1,5 KB * 2048 * 2 * 1000 ----- -This totalling to about 280MB. So if you have some data about the medium size of the events to process you can size +This totalling to about ~6GB. So if you have some data about the medium size of the events to process you can size the memory accordingly without risking to go in Out-Of-Memory error on the direct memory space in production environment. +NOTE: This calculation is the worst case scenario, where all Beats clients have sent almost full batches, but neither has +yet completed. In normal circumstances this is unlikely to happen, because different Beats send data with different rates, +when one client is sending, some other is idle. However, this situation could happen after a Logstash crash, on restart +all clients will bomb the Logstash process. + //Content for Beats ifeval::["{plugin}"=="beats"] [id="plugins-{type}s-{plugin}-multiline"]