Performance with one vs multiple output configurations

[adiforluls]

I have the following use case:
I'm running a kubernetes environment where I have pods running in multiple namespaces (1000s). I need to write logs coming out of a few hundred of these namespace to opensearch using output(s), let's call these namespaces ns1, ns2, ns3, ns4..., ns100. This is the input config:

[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    docker
    Tag    kube.*

I have two ways to configure output(s)

1. Approach $1

Configure a single output plugin for that writes log to the destination coming out of any of these 4 namespaces

[Output]
    Name    opensearch
    Match_Regex    ^(.*)\.(.*)_(?:ns1|ns2|ns3|ns4...|ns100)_.*$
    Host    foo.bar
    Port    9200
    Index    foo-$kubernetes['namespace_name']
    Suppress_Type_Name    true

As I see, the Match_Regex will be huge for 100 namespaces.

2. Approach $2

Configure an output plugin per namespace that writes logs to the same output in parallel

[Output]
    Name    opensearch
    Match    *.*_ns1_*.*
    Host    foo.bar
    Port    9200
    Index    foo-ns1
    Suppress_Type_Name    true
[Output]
    Name    opensearch
    Match    *.*_ns2_*.*
    Host    foo.bar
    Port    9200
    Index    foo-ns2
    Suppress_Type_Name    true

and so on. For 100 namespaces I'll have 100 outputs now.

My questions are:

1. Are there any benefits of approach $2 over $1? My understanding is that since we have a per namespace output writers/plugins, these will write logs in parallel, so the logs will be written to opensearch faster, leading to higher performance? Is this correct.
2. If $2 doesn't come with the benefits I'm assuming it would, will approach $2 create problems as I increase my number of target namespaces? Will a single output be able to handle a good amount of traffic generated from 100s of namespaces?
3. What's the best way to configure outputs if I want to ship logs to a destination from some namespaces but not all?

cc: @patrick-stephens if you could help answer these questions or know anyone who can answer these.

[patrick-stephens]

Why not just use a grep to drop the namespaces you do not care about then just a single output matching anything left?

input --> kubernetes filter (to get metadata) --> grep kubernetes.namespace key --> output

[adiforluls]

It's not so simple sadly, I see that I failed to mention that I can have multiple destinations (sorry about that), by default it will just be opensearch. I need to control namespaces that write logs to opensearch, the other namespace's logs can go to any arbitrary destination if configured. I cannot drop other namespace's logs or "selectively" tail a few namespaces.

To give an example, I can have destinations X, Y and Z. Some namespace's logs go to X, some go to Y, all of them go to Z. I came up with two solutions as described above, I don't see a third solution to control what gets sent to an output.

[patrick-stephens]

Rewrite tag could also do it, e.g. use a opensearch.x tag then and just match that.

[patrick-stephens]

You could also modify the tail input to only tail the namespaces you care about too. The namespace is part of the filename I believe.

You could either do this in one tail input or maybe better to have multiple tail inputs which then stops a noisy namespace set of logs starving the other namespaces, i.e. a noisy neighbours problem.

[patrick-stephens]

All these questions are pretty subjective based on your actual data and infrastructure configuration I would say. Theoretical answers can probably be provided but honestly it would be easy to verify performance directly on your cluster to confirm actual results with your specific log files and data rates plus pod configuration.

[adiforluls]

Agreed, it makes more sense to verify performance in a live cluster. Although if there's a theoretical answer then it will be easier to validate that with some actual testing, it's fine if there's no straightforward way to answer this theoretically.

[patrick-stephens]

I mean, the answer will be "it depends" I think :)
Things like which specific version of Fluent Bit as well may lead into it too.

For any performance question though, always validate yourself rather than trust a random internet person.

[adiforluls]

The questions originated since I could find various benchmarking for FluentBit based on size of data or rate of incoming data, but not based on the number of configs (Can multiple outputs cause a problem? Can multiple filters slow down FluentBit? etc). But I see that it can be hardware dependent and there are definitely many variables.

Thanks for putting up with my questions.

[patrick-stephens]

No worries, and I'll try to answer without a generic "it depends" (but it does!) :)

Multiple outputs should not cause a problem in the simple, happy day scenario of all outputs being reachable, i.e. no backpressure.
If there is backpressure then you need to decide what to do - should we block input to allow stuff to catch up (what if it never does?), how many times should we retry, how much buffering do you want and do you want it persistent or in-memory, etc.
And obviously still depends on data rates but this is no different to a single pipeline having to do lots of data too, i.e. can you actually process the data rate required with the CPU you have available? Chunking it up into multiple outputs should not change this problem, overall can you achieve the load?

Outputs have supported multiple threads for a while now via the workers parameter - this can allow you to process output chunks in parallel but obviously you need the CPU resources allocated to your pod/process to allow it. Also, again, what about the destination for that output - can it handle the load in parallel, out of order, etc.?

Filters are processing so more of it requires more effort to do. Typically these were all run on the main thread so starvation or contention can be an issue.

Inputs can now be threaded as well to prevent main thread usage plus you can attach processors to an input, i.e. run filters only for that input in its dedicated thread.

TLDR; the number of configured plugins does not inherently cause an issue, it's what you're doing with them that does.

[adiforluls]

Super Thanks Patrick! This helps to clear the picture.