enhancement(codecs): Add semantic meaning to gelf decoded data.

[apollo13]

Summary

Set semantic meaning for gelf decoded messages.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

How did you test this PR?

Inserted data using the following config and verified that the fields where set as expected.

sources:
  my_source_id:
    type: demo_logs
    format: json
    log_namespace: true

transforms:
  test:
    type: remap
    inputs:
      - my_source_id
    source: |

      log(.)
      log(%)

sinks:
  my_sink_id:
    type: elasticsearch
    endpoints:
      - https://localhost:9200
    inputs:
        - test
    api_version: v8
    tls:
      verify_certificate: false
    mode: data_stream
    auth:
      strategy: basic
      user: admin
      password: xxx
    data_stream:
      auto_routing: true
      sync_fields: false

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

• Please read our Vector contributor resources.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

[jszwedko]

Thanks @apollo13 !

[jszwedko]

@apollo13 it looks like there is a failing gelf test here when you get some time.

[apollo13]

@jszwedko Yes, as I have written on discord the failing tests are the new tests that I added. While manual testing did work unless I tested something completely wrong it looks as if the log.get_by_meaning returns nothing. I am not sure what I would need to fix here though or if this is expected at this stage of parsing :/ Do you have any pointers?

[jszwedko]

@jszwedko Yes, as I have written on discord the failing tests are the new tests that I added. While manual testing did work unless I tested something completely wrong it looks as if the log.get_by_meaning returns nothing. I am not sure what I would need to fix here though or if this is expected at this stage of parsing :/ Do you have any pointers?

Ah, I see, yes, I think the schema isn't applied to the log event at the level this test is at 🤔 Using dbg(&log) seems to show no schema attached. I think that is associated one level up. I see the syslog codec lacks tests for meaning, even though it associates some, so I'd be ok with dropping the tests you have here.

[jszwedko]

@apollo13 actually, I was able to modify the tests to set the schema definition (see 45ccc73). It is still failing for message though if you want to take a look. host seems to be working correctly.

[apollo13]

Thanks @jszwedko, I am a bit baffled about what is happening though. During the test failure the log event and metadata is shown. The meaning is setup as follows:

                meaning: {
                    "host": Valid(
                        .host,
                    ),
                    "message": Valid(
                        .short_message,
                    ),
                    "timestamp": Valid(
                        .timestamp,
                    ),
                },

Which is what we have defined (ie message should be taken from short_message). But if we look at the log event itself we see:

                KeyString(
                    "message",
                ): Bytes(
                    b"A short message that helps you identify what is going on",
                ),

instead of short_message. Is that a result of parsing into the legacy namespace which rewrites short_message to message or similar?

[apollo13]

Ah this always parses it into message:

vector/lib/codecs/src/decoding/format/gelf.rs

Line 125 in 45ccc73

let mut log = LogEvent::from_str_legacy(parsed.short_message.to_string());

[apollo13]

I see the syslog codec lacks tests for meaning, even though it associates some, so I'd be ok with dropping the tests you have here.

Looking at the syslog codec it handles the log namespace in schema_definition, I think this might be needed for the gelf decoder as well, what do you think?

[jszwedko]

I see the syslog codec lacks tests for meaning, even though it associates some, so I'd be ok with dropping the tests you have here.

Looking at the syslog codec it handles the log namespace in schema_definition, I think this might be needed for the gelf decoder as well, what do you think?

Ah, yeah, I think you may be right.