Log 2 Metric filter

[agup006]

From the Fluent Community meeting we were discussing Log 2 Metric filter, and now starting this to how we can achieve log 2 metric

[seveas]

For our use case: we are maintaining counters about log messages, e.g. per SYSTEMD_UNIT or kubernetes['namespace_name'], so ideally we could say: count messages and bytes received with distinct values for these attributes. Our current lua implementation that sends data to statsd:

function kube_stats(tag, timestamp, record)
    if record.kubernetes then
        stat("fluentbit.kube.messages", 1, COUNTER, { "kube_namespace:"..(record.kubernetes.namespace_name or "unknown"), "kube_container:"..(record.kubernetes.container_name or "unknown") })
    end
    return UNCHANGED, nil, nil
end

function stat(name, value, statstype, tags)
    local line = string.format("%s:%s|%s|#%s", name, value, statstype, table.concat(tags or {}, ","))
    assert(statsd:send(line))
end

And it would be nice if we could replace that with something like

[FILTER]
   type = counters
   attributes = kubernetes[namespace_name],SYSTEMD_UNIT

[seveas]

One thing we'd also like to move to fluent-bit, which is a different way of doing metrics: we have logs containing metric data in logfmt format, e.g delay=0.1 size=1.5 . Something like this would be nice for those:

[FILTER]
  type = metrics
  metric delay = myapp.queue.delay gauge "Description goes here"
  metric size = myapp.response.size gauge "Description goes here"

Which would then map these logfmt keys to metrics. I am currently planning to implement this with another lua script, but something native would be terriffic.

[patrick-stephens]

Be good to make it work for the internal format then we can just make sure logs are parsed with whatever parser is required and get the output.

Various examples of metrics in logs in different formats are here: https://github.com/couchbase/couchbase-fluent-bit/tree/main/test/logs

The Erlang ones are particularly verbose: https://github.com/couchbase/couchbase-fluent-bit/blob/main/test/logs/babysitter.log
There are relatively straightforward ones too: https://github.com/couchbase/couchbase-fluent-bit/blob/main/test/logs/projector.log#L11

[patrick-stephens]

See #6674