Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support compressed fluent input #33

Open
thomas-boehm-tractive opened this issue May 28, 2024 · 0 comments
Open

Support compressed fluent input #33

thomas-boehm-tractive opened this issue May 28, 2024 · 0 comments

Comments

@thomas-boehm-tractive
Copy link

Support input of compressed (gzip) fluent input

According to fluentbit and fluentd, the forward plugin, which we use to write date to logstash, supports compression by setting compress: gzip.
However, it appears that (in my example) the tcp plugin, with the fluent codec, doesn't support this yet.

I think it would make sense, especially for bigger documents, to allow the transfer of compressed data.

This would be a very simple configuration to verify the issue:

Logstash pipeline

input {
  tcp {
    port => "40005"
    codec => fluent {
      nanosecond_precision => true
    }
  }
}

output {
  stdout { 
    codec => json
  }
}

fluentbit-config:

[INPUT]
    name              tail
    path              /tmp/input/*.log
[OUTPUT]
    Name stdout
    Match *

[OUTPUT]
    Name forward
    Match *
    Host 172.17.0.3
    Port 40005
    compress gzip

This is the resulting error message, important part (Error) PackedForward with compression is not supported:

{"level":"ERROR","loggerName":"logstash.codecs.fluent","timeMillis":1716882646539,"thread":"nioEventLoopGroup-2-1","logEvent":{"message":"Fluent parse error, original data now in message field","error":{"stackTrace":[{"class":"RUBY","method":"decode_event","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-fluent-3.4.2-java/lib/logstash/codecs/fluent.rb","line":116},{"class":"RUBY","method":"decode","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-fluent-3.4.2-java/lib/logstash/codecs/fluent.rb","line":61},{"class":"org.msgpack.jruby.Unpacker","method":"each","file":"org/msgpack/jruby/Unpacker.java","line":222},{"class":"org.msgpack.jruby.Unpacker","method":"feed_each","file":"org/msgpack/jruby/Unpacker.java","line":209},{"class":"RUBY","method":"decode","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-codec-fluent-3.4.2-java/lib/logstash/codecs/fluent.rb","line":60},{"class":"RUBY","method":"decode_buffer","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-tcp-6.4.1-java/lib/logstash/inputs/tcp.rb","line":219},{"class":"RUBY","method":"decode","file":"/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-tcp-6.4.1-java/lib/logstash/inputs/tcp/decoder_impl.rb","line":23}],"exception":{"metaClass":{"metaClass":{"metaClass":{"error":"org.jruby.exceptions.Exception: (Error) PackedForward with compression is not supported","data":.....}]}}}}}}}

Also, I think, the error message is not super helpful, as I don't know what PackedForward is, why it is used, or if I can use some other type with compression?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant