Dead Letter Queue is never cleared after processing.

[kreiger]

• Steps to Reproduce:

1. Logstash fails to write events to ElasticSearch for some reason. In our case, a mapping error.
2. Logstash writes the events to DLQ
3. Second pipeline reads DLQ using logstash-input-dead_letter_queue and writes them somewhere else, in our case Kafka.

Expected behavior:

DLQ is cleared after processing.

Actual behavior:

DLQ is not cleared after processing.
DLQ becomes full.
Logs are spammed with warnings about DLQ being full.
Log events are lost once DLQ is full.

[pdscopes]

I have the same issue using the DLQ input. I think that there should be a configuration option to enable/disable the clearing of the DLQ after processing which defaults to false.

[naimmalek]

Having same issue. Agree with @pdscopes there should be an option in configuration to clear event from DLQ.

[Cr3idhne]

Will this be available soon in an future release? Facing the same issues, would be great to have the functionality to preform this as mentioned by @pdscopes . Seen multiple elastic.discussion topics on this also.

[willemveerman]

I've got around this in kubernetes by configuring an output plugin in logstash which kills the container if the DLQ exceeds 80% of its pre-configured size.

The DLQ itself is at a path within the container, so when the pod is recycled it is cleared completely - as is logstash's internal DLQ count.

The sincedb files which track the log file reading position are in a dir bind-mounted from the host, so they survive the pod recycle

    input {
      dead_letter_queue {
        id => "kubernetes_dlq"
        path => "${PATH_DEAD_LETTER_QUEUE}"
        sincedb_path => {{ .Values.logstash.dead_letter_queue_sincedb_path | quote }}
        commit_offsets => true
        tags => ["dlq"]
      }
    }

    # If the DLQ is > 80% full kill the pod
    output {
      if "dlq" in [tags] {
        exec {
          command => "if (( $(du -sb {{ .Values.logstash.path_dead_letter_queue }}/main/ | cut -f1) > $(( {{ .Values.logstash.dead_letter_queue_max_bytes }}*80/100 )) )); then kill 1; fi &"
        }
      }
    }

Also worth pointing out:

1. The DLQ checking feature works by sending a kill to the logstash PID - this command sends a SIGTERM by default: https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html
2. A SIGTERM causes logstash to gracefully stop, completing in-process events before shutting down: https://discuss.elastic.co/t/gracefully-shutdown-logstash/40132