datadog_metrics sink's traffic flows to our VPC endpoint but not to Datadog

[nzxwang]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I would like to configure vector to route traffic to Datadog’s PrivateLink offering in us-east-1 from other regions using VPC Peering. I've configured datadog_metrics with endpoint = "https://metrics.agent.datadoghq.com/api/v1/series" and verified my test metrics are flowing through my pipeline in the trace logs (as per previous advice #21325 (comment)) as well as vector taping into my pipeline. Moreover, my VPC endpoint is correctly reporting traffic for my test metrics:

However, the test metrics are either being dropped by Datadog's VPC Endpoint Service or by Datadog itself.

Configuration

[sinks.datadog_metrics_sink_4]
type = "datadog_metrics"
inputs = ["remap_fanout_del_4"]
default_api_key = "${DD_API_KEY}"
batch.timeout_secs = 5
batch.max_events = 2000
buffer.max_events = 200000
request.concurrency = "adaptive"
endpoint = "https://metrics.agent.datadoghq.com/api/v1/series"

Version

vector 0.39.0 (x86_64-unknown-linux-musl 73da9bb 2024-06-17 16:00:23.791735272)

Debug Output

2024-11-21T22:40:30.948719Z DEBUG sink{component_kind="sink" component_id=datadog_metrics_sink_4 component_type=datadog_metrics}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://metrics.agent.datadoghq.com/api/v1/series/api/v1/series method=POST version=HTTP/1.1 headers={"dd-api-key": "REDACTED", "dd-agent-payload": "4.87.0", "content-type": "application/json", "content-encoding": "deflate", "user-agent": "Vector/0.39.0 (x86_64-unknown-linux-musl 73da9bb 2024-06-17 16:00:23.791735272)", "accept-encoding": "identity"} body=[245 bytes]

Example Data

No response

Additional Context

No response

References

No response

[nzxwang]

I also cut a ticket to Datadog support (id: 1937927) as I'm unsure if this issue lies in vector or in Datadog. Apologies in advance if vector is working correctly.

[pront]

Hi @nzxwang, thank you for creating this issue.

Did you by any chance inspect other sink metrics e.g. https://vector.dev/docs/reference/configuration/sinks/datadog_metrics/#component_sent_events_total? I would also like to confirm that the response codes on the Vector service are OK.

[nzxwang]

Hi @pront thank you for the quick response! I should have thought to check the other sink metrics like component_sent_events_total earlier but I just assumed that it worked since there was traffic over our endpoint. I'm currently observing that our datadog_metrics sinks are receiving events but not sending them, despite there being traffic over our vpc endpoint (please don't mind the multiple sinks; #21373 (reply in thread)):

Do you have any insight into how this is possible?