40x & 50x Emissary Traces not categorised as ERRORS in Datadog

[pkalantri-ebsco]

Describe the bug
We have 2 emissary ingresses running in our EKS cluster.
They are successfully integrated with a Datadog Agent pushing traces to our DataDog Backend.

We are now receiving the traces/spans in DataDog but their statuses are not being classified correctly - everything is classified as "OK".
According to Datadog support, this is an issue with how the Ambassador team sends the attributes with their traces.
This is the response we have gotten from DataDog Support :

...
It looks like the root issue is that Ambassador has made a mistake when implementing our tracing library that is causing error spans to not be properly marked. 
To help you if you're interested in filing a bug report with Ambassador, my reasoning is below:
 
Currently, spans submitted by the Ambassador are being marked as errors, but are not following our conventions for how to mark the errors. 
Spans from Ambassador are setting the span attribute @error to "true", where we expect the span tag status to be set to error and the error span tag to be set to 1 (you can see the difference between span tags and attributes [here](https://docs.datadoghq.com/tracing/trace_collection/tracing_naming_convention/#span-tags), to clarify further). 
To fix this on their end, they can use the set_error method to automatically follow our conventions for flagging errors which you can see in our documentation on this here.
...

Links sent by DataDog:
https://docs.datadoghq.com/tracing/trace_collection/tracing_naming_convention/#span-tags
https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/ruby/dd-api/#setting-errors-on-a-span

To Reproduce
Steps to reproduce the behavior:

1. Install Emissary - version below
2. Install Datadog Agent - version below
3. Connect traces to Datadog Backend via DD Agent
4. Check Traces

Expected behavior
We are supposed to see traces with HTTP error code 50x & 40x as errors. Instead they are being ingested as "OK" similar to 200 status codes.

Versions (please complete the following information):

• Ambassador Emissary: v8.9.1
• Datadog Agent: v3.70.1
• Kubernetes environment - EKS v1.30

Please help. Thanks!