Regex performance question

[ptr1120]

Hello,

I'm working with Vector and have a query regarding its regex capabilities in the remap language, specifically about compilation and caching. In my setup, I use various regex patterns within a remap transform, as shown below:

logFormat = .kubernetes.pod_annotations."my-company.io/log-format"
if logFormat == "format1" {
    pattern = r'...'
    . |= parse_regex!(.message, pattern)
} else if logFormat == "format2" {
    pattern = r'...'
    . |= parse_regex!(.message, pattern)
}

I plan to introduce more patterns and am concerned about potential performance impacts. My questions are:

1. Are regexes compiled and/or cached in Vector to enhance performance?
2. If regexes are not inherently cached, is there a way to precompile them or store them in a compiled format using an enrichment_table or similar feature?

Your insights would be greatly appreciated.

Best regards,
Peter

[jszwedko]

• Are regexes compiled and/or cached in Vector to enhance performance?

Regexes are, generally, compiled once at start-up. The only exception is the to_regex function which does it at run time.

• If regexes are not inherently cached, is there a way to precompile them or store them in a compiled format using an enrichment_table or similar feature?

There is no caching, no, but again, static regexes are compiled upfront. #137 is tracking an enhancement to VRL to allow for caching within functions. If it existed, we'd take advantage of it with to_regex.

[ptr1120]

thank you very much for the detailed information!