[feature] Support configurations so it behaves like JDBC input

[tsouza]

Currently, the ES input plugin has no way to pick up from where it left off last time it executed, just as the JDBC input does with the sql_last_value value.

I propose a set of configurations that makes ES input plugin query from a past known @timestamp (by default)

[marceliq]

+1

[strawgate]

We think this will solve a couple problems for us. Curious if anyone has a PR or modifications they have made that implement this that they can share.

[lucasbemol]

I have a same problem.

I'm having some problems with ElasticSearch. Keeping it short, I need to look into my logs and normalize them in various indexes.
My logs are stored in both an index of ElasticSearch and in a relational database. These other indexes I want to feed are also in ElasticSearch.
At first, I thought of using the Logstash to do that. I would keep feeding my ElasticSearch event store as I do today (e.g., getting data from the DB) and would have a Logstash pipeline to look the data coming at the event store in ElasticSearch and outputting the normlized data in my other index(es). Problem is, the Logstash plugin to input data from ElasticSearch seems to a built-in way to track data, as JDBC provides provides with tracking_column .
Is there a way to simulate the tracking_column behaviour in the ElasticSearch plugin, or should I go another solution? If so, do you guys have a suggestion?

[gaurav-ml1]

I have same problem.

[Jack47]

Any updates on this? We really needs this

[pussinboots1992]

+1

[JordiNeil]

+1

[ljia2023]

+1. Anyone found a reasonable solution for this issue?

[strawgate]

When I ran into this issue in 2019 I ended up creating a pipeline that read from elasticsearch and then updated the documents read in elasticsearch to have a "processed" field set to true (in addition to the other processing I needed).

The elasticsearch input query would exclude documents with processed set to true. Giving me the ability to only process data once at the expense of writing each document twice.