NH-91603: custom otlp metrics through otlp protocol

[xuan-cao-swi]

Description

1. add custom metrics with otlp protocol
2. add metrics-sdk as one of the dependencies

Current limitation (need more clarification):

1. user have to install metrics exporter
2. user have to decide when the metrics exporter will happen (either through periodic reader or manual)

Test (if applicable)

[cheempz]

Thanks for noting the limitation/question @xuan-cao-swi! I'd like our custom distro to do both: take a dependency on the exporter so that it is installed by default (guess it's not yet part of opentelemetry-sdk and eventually will be?), and to set up a periodic reader.

Any thoughts on possible downside / risk if we went this way?

[xuan-cao-swi]

I'd like our custom distro to do both: take a dependency on the exporter so that it is installed by default (guess it's not yet part of opentelemetry-sdk and eventually will be?)

Also, otlp trace exporter is not part of opentelemetry-ruby sdk. I don't think they will include any exporter in sdk; but metrics-sdk will be part of sdk eventually.

We install opentelemetry-exporter-otlp and opentelemetry-exporter-otlp-metrics by default in lambda through layer build, but not in our agent gemspec.

To have periodic reader by default, then we may discourage user to manually pull/export the metrics. If user use aggregation_temporality as :delta, then user may experience strange metrics overtime if they use periodic reader and manual export together.

[xuan-cao-swi]

Actually the PeriodicMetricReader is enabled by default: https://github.com/open-telemetry/opentelemetry-ruby/blob/opentelemetry-metrics-sdk/v0.4.0/metrics_sdk/lib/opentelemetry/sdk/metrics/configuration_patch.rb#L44

User have choice to decide the duration of exporting

[cheempz]

We install opentelemetry-exporter-otlp and opentelemetry-exporter-otlp-metrics by default in lambda through layer build, but not in our agent gemspec.

Gotcha, let's make the same change in our custom distro, which also means no need to special case this in the lambda layer build :)