Change parsing for JSON logs

[markdboyd]

Changes Proposed

Currently, when Logstash receives an input log that is JSON, it parses the JSON properties into a field called app.

While this behavior is nice in that it allows logs to be searched by the custom properties from the JSON log (based on the detected field type for each JSON property), it has the problem that the first log received for an index will determine the type of any fields parsed from the JSON log.

In our case where all customer logs are ingested into a shared index per day, this is problematic, since subsequent JSON logs may have different field types for keys that existed on JSON logs that have already been ingested. When this occurs, Elasticsearch will reject the incoming JSON log due to the mismatch in field types.

The proposed fix here is to use a flattened type for handling dynamic fields from JSON logs on a new custom field. When you use a flattened field type, the entire object for a custom JSON log is indexed and still searchable, but only using basic search functionality.

So for a log like:

{"custom":{"foo":"bar"}}

You could search for the document like so:

custom.foo: "bar"

The only downside of flattened fields is that they can only be queried using basic search functionality, but that should be sufficient for letting users query their documents in Kibana. The enormous upside is that a flattened field can contain a complex data-structure for searching, but only takes up 1 field in your index mappings, not 1 field per custom property in the JSON logs. Also, flattened fields will not be subject to the field type mismatch issues that are causing some customer logs to fail ingestion.

So this PR:

• Adds a default mapping for the custom field to be a flattened type
• Updates Logstash to parse incoming JSON into the custom field

Security Considerations

There are no changes to security configuration for Elasticsearch here, just changing the destination field type for custom JSON logs.

[JasonTheMain]

This might work, lets give it a shot