Support checkpointing on interval only, not batch completion

[mbrancato]

Please add support for the checkpoint interval to be the only influence on writing a checkpoint to blob storage. Currently, checkpointing occurs when any batch completes as well. The problem is that some outputs are constrained by the batch size, and that might lead to smaller batches, causing significant write operations. I've seen this be very expensive even for smaller environments. Even then, most batches at the default size of 50 are going to cause large reads/writes for checkpointing.

That said, our use case would be fine with a purely time-based checkpoint, not batch based. By reducing checkpoint intervals to 30 seconds or so, a significant cost savings could be realized.

[choovick]

Also interested in any feature that can improve costs of storage accounts. I'm surprised there is no way to use local filesystem or centralized cache/document storage like redis or mongo to save checkpoints...

[choovick]

@mbrancato Thinking about it again. Can't we achieve this by setting very large max_batch_size and set checkpoint_interval to desired delay? We will have to watch memory usage thou...

[mbrancato]

I had other limitations on batch sizes. But batch sizes don’t let Logstash wait for the batch queue to fill up. If you are using Azure storage, be sure to use V1 storage accounts since the transaction costs are 90% less than V2.

[choovick]

@mbrancato IC, Thanks! it does not look that MS planing to End Of Life V1, definitely gonna try it out.

[SpencerLN]

+1 for any feature that can reduce the storage costs associated with check-pointing.

[shauryagarg2006]

Also I think the checkpointing interval at the moment is not serving any purpose. Every event is checkpointed.

I made the following change in my fork to fix it.

shauryagarg2006@60292a1

[ghost]

Hello,
I had the same issue with the cost associated with the Azure Storage. I decided to switch to the Kafka input as Azure Event Hub supports it. The cost are quite low now !
The Microsoft documentation states that Azure Event Hub uses an Azure Storage internally when using the Kafka interface, I hope they won't change their mind about providing this storage without additional costs.
Some resources you might find useful :

• https://docs.microsoft.com/fr-fr/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
• https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/logstash
• https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/tutorials/filebeat
• https://github.com/Azure/azure-event-hubs-for-kafka#more-faq

[lucianaparaschivei]

hello,
we are facing the same issue with high costs on the azure storage. The plugin is making way too many storage transactions. Looks like 1 checkpoint per 3-4 messages which is a lot. Can you please address this?

[alaeddineG]

Hi,
We are having a similar cost issue with storage account on a system that feed logstash from event hub (for logging purposes)
Logstash works perfect for as and having to plug kafka to reduce the cost like @ghost mentioned will trigger another unnecessary infrastructure element to maintain and pay for as well.
Im wondering if using another checkpoint manager will make sense like Redis (https://learn.microsoft.com/en-us/java/api/overview/azure/messaging-eventhubs-checkpointstore-jedis-readme?view=azure-java-preview https://github.com/Azure/azure-sdk-for-java/tree/azure-messaging-eventhubs-checkpointstore-jedis_1.0.0-beta.3/sdk/eventhubs/azure-messaging-eventhubs-checkpointstore-jedis)
Will you be open for such a contribution ?