Connexion v3, Flask v3 integration with dd-trace fails: datadog context not present in ASGI request scope, trace middleware may be missing

[whyisdifficult]

Description

Running the latest version of the Connexion framework with a Flask application I get the following warning while trying to log messages to Datadog:

datadog context not present in ASGI request scope, trace middleware may be missing

{
  "asctime": "2024-06-24 16:41:31,417",
  "name": "ddtrace.contrib.starlette.patch",
  "levelname": "WARNING",
  "message": "datadog context not present in ASGI request scope, trace middleware may be missing",
  "lineno": 111,
  "module": "patch",
  "pathname": "/usr/local/lib/python3.12/site-packages/ddtrace/contrib/starlette/patch.py",
  "taskName": "Task-3",
  "dd.version": "",
  "dd.env": "",
  "dd.service": "",
  "dd.trace_id": "0",
  "dd.span_id": "0"
}

This warning effectively means that we don't have tracing enabled for our application.

Expected behaviour

We expect not to see the warning message and to have tracing enabled for the underlying Flask application.

Actual behaviour

We get the following warning and DD traces are disabled.

{
  "asctime": "2024-06-24 16:41:31,417",
  "name": "ddtrace.contrib.starlette.patch",
  "levelname": "WARNING",
  "message": "datadog context not present in ASGI request scope, trace middleware may be missing",
  "lineno": 111,
  "module": "patch",
  "pathname": "/usr/local/lib/python3.12/site-packages/ddtrace/contrib/starlette/patch.py",
  "taskName": "Task-3",
  "dd.version": "",
  "dd.env": "",
  "dd.service": "",
  "dd.trace_id": "0",
  "dd.span_id": "0"
}

Steps to reproduce

We create the application as follows:

import connexion

app = connexion.App( __name__, specification_dir='.')
...

# setup Datadog integration

ddtrace.tracer.configure(hostname=app.app.config.DD_APM_HOST, port=8126)
ddtrace.tracer.set_tags({'env':  'the-env')})
ddtrace.config.flask['analytics_enabled'] = True
ddtrace.config.flask['trace_signals'] = False
ddtrace.patch_all(logging=True)

# we have also tried adding the tracing middleware here
# app.add_middleware(TraceMiddleware)

# we also tried patching here instead
ddtrace.patch_all(logging=True, starlette=True)

Then, the application runs via docker-compose:

CMD ["gunicorn", "-c", "gunicorn.conf.py", "-k", "uvicorn.workers.UvicornWorker", "myproject.__main__:create_app()"]

This was not happening when we were using the previous version of Connexion (connexion = "2.14.2"), Flask and dd-trace (ddtrace = "2.7.3")

Solutions Tried

One of the many things I have tried was to use the TraceMiddleware as follows and, patching the application in different places.

import connexion
from ddtrace.contrib.asgi import TraceMiddleware
from ddtrace.contrib.starlette import patch

patch()  # patching here
app = connexion.App( __name__, specification_dir='.')

# patch()  # or patching here
app.add_middleware(TraceMiddleware)
# patch()  # or patching here

Via the logs I can see that starlette has been patched correctly

Configured ddtrace instrumentation for 59 integration(s). The following modules have been patched: aioredis,aiomysql,aredis,asyncio,boto,botocore,bottle,cassandra,celery,consul,django,dramatiq,elasticsearch,algoliasearch,futures,gevent,graphql,grpc,httpx,kafka,mongoengine,mysql,mysqldb,pymysql,mariadb,psycopg,pylibmc,pymemcache,pymongo,redis,rediscluster,requests,rq,sanic,sqlite3,aiohttp,aiohttp_jinja2,aiopg,vertica,molten,jinja2,mako,flask,flask_login,starlette,falcon,pyramid,logging,pynamodb,pyodbc,fastapi,dogpile_cache,yaaredis,asyncpg,aws_lambda,openai,langchain,subprocess,unittest

I have followed the docs for dd-trace integration but I still can't make it work.

The application does work well. Meaning that I'm able to make requests to the API it implements.

Additional info:

The most important and relevant would libraries that we need are:

connexion==3.1.0
Flask==3.0.3
gunicorn==20.1.0
asgiref==3.8.1
anyio==3.7.1
a2wsgi==1.10.4
Werkzeug==3.0.3
httpx==0.23.0
Jinja2==3.1.3
inflection==0.5.1
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
requests==2.32.3
urllib3==1.26.18
uvicorn==0.29.0
uvloop==0.19.0
datadog==0.40.0
ddtrace==2.9.2
starlette==0.37.2
swagger_ui_bundle==1.1.0
sentry-sdk==1.45.0

Output of the commands:

• python --version

3.12.3

• pip show connexion | grep "^Version\:"

3.1.0

Does anyone know how to solve this issue? Have anyone else run into the same problem? Thanks.

[Lewiscowles1986]

We use the datadog lambda layer without issue at work, which AFAIK, redirects STDOUT to datadog.
From the code I am reading here; it looks like DataDog is trying to manually instrument the framework, which I think means it needs to be instantiated prior to the framework code.

Note that your minimal example with starlette, you import and initialise dd_agent before Starlette, but in your example here it comes after.

I'm also not seeing a datadog API key anywhere, even referenced by environment. Can you check that you have either DD_AGENT or DD_API_KEY set in the environment and that the values are correct?