Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

duplicate ID infinite retry (and blocks pipeline) #86

Open
jpuskar opened this issue May 5, 2023 · 0 comments
Open

duplicate ID infinite retry (and blocks pipeline) #86

jpuskar opened this issue May 5, 2023 · 0 comments
Labels

Comments

@jpuskar
Copy link

jpuskar commented May 5, 2023

My plugin version is 3.1.7.
Logstash version is 8.7.1.
MongoDB version is 6.0.4.

It looks like duplicate ID's are supposed to be skipped, per this code:

Copy/pasta for convenience:

    rescue => e
      if e.message =~ /^E11000/
        # On a duplicate key error, skip the insert.
        # We could check if the duplicate key err is the _id key
        # and generate a new primary key.
        # If the duplicate key error is on another field, we have no way
        # to fix the issue.
        @logger.warn("Skipping insert because of a duplicate key error", :event => event, :exception => e)
      else
        @logger.warn("Failed to send event to MongoDB, retrying in #{@retry_delay.to_s} seconds", :event => event, :exception => e)
        sleep(@retry_delay)
        retry
      end

However, sending messages with a duplicate ID gives me this error and blocks the pipeline.

2023-05-05T10:13:56-04:00 [2023-05-05T14:13:56,041][WARN ][logstash.outputs.mongodb ][mongodb-output][9c30f49a59926f75431e33f9dd35df2bd89174638cf70dcc7119c17b22898d83] Failed to send event to MongoDB, retrying in 3 seconds {:event=>#<LogStash::Event:0x1243a1dd>, :exception=>#<Mongo::Error::OperationFailure: [11000]: E11000 duplicate key error collection: selfservice.transactions index: _id_ dup key: { _id: "1d8b1e49-33a5-4eec-b768-9ebd9be81f42" } (on mongodb-0.mongodb-svc.selfservice-shared.svc.cluster.local:27017, modern retry, attempt 1)>}

This error indicates that the conditional e.message =~ /^E11000/ is never truthy.
Perhaps the mongodb library changed the structure of error messages?

@jpuskar jpuskar added the bug label May 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant