Connexion v3, Flask v3 integration with dd-trace fails: datadog context not present in ASGI request scope, trace middleware may be missing

[whyisdifficult]

Summary of problem

Running the latest version of the Connexion framework with a Flask application I get the following warning while trying to log messages to Datadog:

datadog context not present in ASGI request scope, trace middleware may be missing

{
  "asctime": "2024-06-24 16:41:31,417",
  "name": "ddtrace.contrib.starlette.patch",
  "levelname": "WARNING",
  "message": "datadog context not present in ASGI request scope, trace middleware may be missing",
  "lineno": 111,
  "module": "patch",
  "pathname": "/usr/local/lib/python3.12/site-packages/ddtrace/contrib/starlette/patch.py",
  "taskName": "Task-3",
  "dd.version": "",
  "dd.env": "",
  "dd.service": "",
  "dd.trace_id": "0",
  "dd.span_id": "0"
}

This warning effectively means that we don't have tracing enabled for our application.

Which version of dd-trace-py are you using?

ddtrace==2.9.2

Which version of pip are you using?

• pip=24.0
• poetry==1.7.1

Which libraries and their versions are you using?

The most important and relevant would be these:

connexion==3.1.0
Flask==3.0.3
gunicorn==20.1.0
asgiref==3.8.1
anyio==3.7.1
a2wsgi==1.10.4
Werkzeug==3.0.3
httpx==0.23.0
Jinja2==3.1.3
inflection==0.5.1
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
requests==2.32.3
urllib3==1.26.18
uvicorn==0.29.0
uvloop==0.19.0
datadog==0.40.0
ddtrace==2.9.2
starlette==0.37.2
swagger_ui_bundle==1.1.0
sentry-sdk==1.45.0

How can we reproduce your problem?

We create the application as follows:

import connexion

app = connexion.App( __name__, specification_dir='.')
...

# setup Datadog integration

ddtrace.tracer.configure(hostname=app.app.config.DD_APM_HOST, port=8126)
ddtrace.tracer.set_tags({'env':  'the-env')})
ddtrace.config.flask['analytics_enabled'] = True
ddtrace.config.flask['trace_signals'] = False
ddtrace.patch_all(logging=True)

# we have also tried adding the tracing middleware here
# app.add_middleware(TraceMiddleware)

# we also tried patching here instead
ddtrace.patch_all(logging=True, starlette=True)

Then, the application runs via docker-compose:

CMD ["gunicorn", "-c", "gunicorn.conf.py", "-k", "uvicorn.workers.UvicornWorker", "myproject.__main__:create_app()"]

This was not happening when we were using the previous version of Connexion (connexion = "2.14.2"), Flask and dd-trace (ddtrace = "2.7.3")

Solutions Tried

One of the many things I have tried was to use the TraceMiddleware as follows and, patching the application in different places.

import connexion
from ddtrace.contrib.asgi import TraceMiddleware
from ddtrace.contrib.starlette import patch

patch()  # patching here
app = connexion.App( __name__, specification_dir='.')

# patch()  # or patching here
app.add_middleware(TraceMiddleware)
# patch()  # or patching here

Via the logs I can see that starlette has been patched correctly

Configured ddtrace instrumentation for 59 integration(s). The following modules have been patched: aioredis,aiomysql,aredis,asyncio,boto,botocore,bottle,cassandra,celery,consul,django,dramatiq,elasticsearch,algoliasearch,futures,gevent,graphql,grpc,httpx,kafka,mongoengine,mysql,mysqldb,pymysql,mariadb,psycopg,pylibmc,pymemcache,pymongo,redis,rediscluster,requests,rq,sanic,sqlite3,aiohttp,aiohttp_jinja2,aiopg,vertica,molten,jinja2,mako,flask,flask_login,starlette,falcon,pyramid,logging,pynamodb,pyodbc,fastapi,dogpile_cache,yaaredis,asyncpg,aws_lambda,openai,langchain,subprocess,unittest

I have followed the docs for dd-trace integration but I still can't make it work.

The application does work well. Meaning that I'm able to make requests to the API it implements.

What is the result that you get?

We get the following warning and DD traces are disabled.

{
  "asctime": "2024-06-24 16:41:31,417",
  "name": "ddtrace.contrib.starlette.patch",
  "levelname": "WARNING",
  "message": "datadog context not present in ASGI request scope, trace middleware may be missing",
  "lineno": 111,
  "module": "patch",
  "pathname": "/usr/local/lib/python3.12/site-packages/ddtrace/contrib/starlette/patch.py",
  "taskName": "Task-3",
  "dd.version": "",
  "dd.env": "",
  "dd.service": "",
  "dd.trace_id": "0",
  "dd.span_id": "0"
}

What is the result that you expected?

We expect the application to be able to log messages to DD with traces enabled.

Does anyone else have run into the same issue? If so, what do you propose as a solution to this?

[emmettbutler]

Thanks for the report, @whyisdifficult. ddtrace doesn't directly support connexion, so I'm not surprised to hear that it has some issues when integrated with connexion and flask. We'll treat this as a request for connexion support and add it to the backlog. In the meantime, we may be able to suggest some custom code that would work around the issue you're experiencing.

cc @mabdinur

[whyisdifficult]

Thanks for the report, @whyisdifficult. ddtrace doesn't directly support connexion, so I'm not surprised to hear that it has some issues when integrated with connexion and flask. We'll treat this as a request for connexion support and add it to the backlog. In the meantime, we may be able to suggest some custom code that would work around the issue you're experiencing.

cc @mabdinur

Thanks @emmettbutler for the response. I have also created a ticket on the Connexion's repo (spec-first/connexion#1945) as I wasn't sure were the error originates.

We would really appreciate if you guys have a temporary workaround for this as we would like to upgrade our applications with the latest versions of these libs.

[whyisdifficult]

Hello guys, have you had the chance to look into this further to explore a possible (temporary) workaround for this issue?Thanks for the efforts. @emmettbutler @mabdinur

[whyisdifficult]

Any updates on this? @emmettbutler @mabdinur Thanks.

[lawrenceong]

Had a similar issue with python-socketio when used with fastapi. Disabled starlette integration altogether by setting two variables:

# Disabled due to issue with log spam
DD_TRACE_STARLETTE_ENABLED=${DD_TRACE_STARLETTE_ENABLED:-false}; export DD_TRACE_STARLETTE_ENABLED
# Also need to disable fastapi as it indirectly enables starlette!
DD_TRACE_FASTAPI_ENABLED=${DD_TRACE_FASTAPI_ENABLED:-false}; export DD_TRACE_FASTAPI_ENABLED

@whyisdifficult you may need to play around with what to disable, since we are using fastapi in place of flask.