Add (arbitrary) new fields from inside a grok pattern

[nick-george]

Hi Jordan and co,

I'd love to be able to create arbitrary logstash fields (name and value) from within a logstash pattern. This requirement has come up numerous times for me while working on patterns. The feature would allow the pattern author to add some additional context to the LS event from inside the pattern. I've posted about this before (a couple of years ago, see below for links), but didn't get much joy. I feel that this feature would take patterns from being almost "complete" to being a one-stop-shop for all grokking needs. The Oniguruma syntax is helpful, but doesn't quite provide all the features I need.

Here's a quick example of what I'm talking about:
Message contents:

Invalid password for group 'blah' from 'nick'

Using the Oniguruma syntax, I can capture the fact that there was an invalid password. However, I'd like to be able to fill in some additional fields (from the ECS) directly via the pattern. E.g

• event.category = 'unix-accounts'
• event.action = 'switch group'
• event.outcome = 'failure'
• error message = 'Invalid password for group'

Here are my previous posts about this.

• https://discuss.elastic.co/t/can-i-add-arbitrary-new-fields-from-inside-a-grok-pattern/76497
• Support adding fields from grok pattern logstash-plugins/logstash-filter-grok#109

I can see that quite a few other people (~60) have been interested in this feature, as that's the number who have followed the link Magnus provided in my discuss topic.

Many thanks for your time,
Nick

[nickmz]

+1 - I imagine this would also greatly reduce the reliance on conditional logic in the Logstash configurations, potentially simplifying them and improving efficiency substantially.