Pillow with FastAPI broken after slimming (PIL.UnidentifiedImageError)

[jannichorst]

Expected Behavior

After slimming a container, Pillow throws a "PIL.UnidentifiedImageError" when reading an image. The code below is for an endpoint, but the behaviour is the same when reading from the file system. The code by itself is fine, as the not-slimmed container does work just fine. Similarly, reading an image from the file system using pillow that is not part of an FastAPI application works too in a slimmed container. Something about the combination slimmed container + FastAPI seems to break pillow.

Am I missing any shared libraries I have to specify manually or what could be the issue? Thanks already for the support!

from fastapi import FastAPI, File, UploadFile
from PIL import Image
from io import BytesIO
from pydantic import RootModel

app = FastAPI()

@app.post("/upload")
async def put_object(file: UploadFile = File(...)):
    request_object_content = await file.read()
    img = Image.open(BytesIO(request_object_content))
    
    width, height = img.size
    return {"width": width, "height": height}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

---

Actual Behavior

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/main.py", line 11, in put_object
    img = Image.open(BytesIO(request_object_content))
  File "/usr/local/lib/python3.9/site-packages/PIL/Image.py", line 3339, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0xffff82157d10>

---

Steps to Reproduce the Problem

1. paste code above in main.py
2. execute: slim --crt-api-version=1.25 build --dockerfile Dockerfile --include-exe tesseract --include-path /app --tag-fat test-container . (don't forget to delete the ".txt" from the attached Dockerfile...)
3. start container with docker run -it -p 8000:8000 test-container.slim
4. open http://0.0.0.0:8000/docs in the browser
5. upload an image (jpeg/png) in the /upload section of the swagger documentation

---

Specifications

• Version: 1.40.11
• Platform: MacOS ARM
  Dockerfile.txt
  requirements.txt

[kcq]

@jannichorst thanks for the extra context and for the snippets! trying to repro it

[kcq]

@jannichorst What's needed is a custom http probe in the slim command. I will share what it should look like with my repro.

[jannichorst]

@kcq Thanks Kyle! Looking forward to the example. It really has been bugging me not getting it done myself...

Just out of curiosity, are you guys open/looking for contributions of examples? Asking because if I get it done I wouldn't mind giving back by providing another example... Let me know if that is something that would be of value to the project.

[kcq]

@jannichorst yes, definitely will be valuable! Any contribution will be great

[kcq]

Here's a slightly hacky way to make it work (it's including the pillow/PIL package directory explicitly):

building the image separately just to keep it simple:
docker build -t repro-python-fastapi-pillow .

minify the target image:
mint slim --http-probe-cmd /docs --http-probe-apispec /openapi.json --include-path /usr/local/lib/python3.9/site-packages/PIL --include-exe tesseract repro-python-fastapi-pillow

Make sure to use the latest release

The next upcoming release will not require you to have --include-path /usr/local/lib/python3.9/site-packages/PIL. The http probes will handle multipart form uploads properly.

[jannichorst]

@kcq thanks for the example! I ran the commands and get the following output:

slim version

cmd=version info=app version='darwin/arm64|Transformer|1.40.11|latest|latest' container='false' dsimage='false' location='/opt/homebrew/bin' current='unknown' verdict='version status information is not available' cmd='version' 
cmd=version info=host release='23.3.0' sysname='darwin' cmd='version' osname='Sonoma (14.3)' osbuild='23D56' version=' Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000' 
cmd=version info=docker ostype='linux' server.version='24.0.7' architecture='aarch64' cmd='version' name='docker-desktop' kernel.version='6.5.11-linuxkit' operating.system='Docker Desktop' 
cmd=version info=dclient build.time='2023-10-26T09:08:15.000000000+00:00' git.commit='311b9ff' cmd='version' api.version='1.43' min.api.version='1.12' 

mint version

cmd=version info=app dsimage='false' location='/usr/local/bin' current='1.41.2' verdict='you have the latest version' cmd='version' version='darwin/amd64|Aurora|1.41.2|d6f9778dcd40dfd2108737070b85d929c9806de6|2024-05-30_03:14:56PM' container='false' 
cmd=version info=host osname='Sonoma (14.3)' osbuild='23D56' version=' Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000' release='23.3.0' sysname='darwin' cmd='version' 
cmd=version info=docker server.version='24.0.7' architecture='aarch64' cmd='version' name='docker-desktop' kernel.version='6.5.11-linuxkit' operating.system='Docker Desktop' ostype='linux' 
cmd=version info=dclient api.version='1.43' min.api.version='1.12' build.time='2023-10-26T09:08:15.000000000+00:00' git.commit='311b9ff' cmd='version' 

sudo mint slim --http-probe-cmd /docs --http-probe-apispec /openapi.json --include-path /usr/local/lib/python3.9/site-packages/PIL --include-exe tesseract repro-python-fastapi-pillow

cmd=slim state=started
cmd=slim info=cmd.input.params target.type='image' target.image='repro-python-fastapi-pillow' continue.mode='probe' rt.as.user='true' keep.perms='true' tags='' image-build-engine='internal' 
cmd=slim state=image.inspection.start
cmd=slim info=image id='sha256:de16ad88404836cfbf51873d8144723dc325270c3f4a4f2a25db7ffe6150bebd' size.bytes='287296525' size.human='287 MB' 
cmd=slim info=image.stack name='repro-python-fastapi-pillow:latest' id='sha256:de16ad88404836cfbf51873d8144723dc325270c3f4a4f2a25db7ffe6150bebd' index='0' 
cmd=slim info=image.exposed_ports list='8000/tcp' 
cmd=slim state=image.inspection.done
cmd=slim state=container.inspection.start
cmd=slim info=container status='created' name='mintk_3993_20240607063300' id='1fa920da9f1076093a76bb44fe6e86f19ee7bde06e6a268cda558f76dc694297' 
cmd=slim info=container status='running' name='mintk_3993_20240607063300' id='1fa920da9f1076093a76bb44fe6e86f19ee7bde06e6a268cda558f76dc694297' 
cmd=slim info=container message='obtained IP address' ip='172.17.0.2' 
cmd=slim info=cmd.startmonitor status='sent' 
cmd=slim info=event.startmonitor.done status='received' 
cmd=slim info=container name='mintk_3993_20240607063300' id='1fa920da9f1076093a76bb44fe6e86f19ee7bde06e6a268cda558f76dc694297' target.port.list='55211' target.port.info='8000/tcp => 0.0.0.0:55211' message='YOU CAN USE THESE PORTS TO INTERACT WITH THE CONTAINER' 
cmd=slim state=http.probe.starting message="WAIT FOR HTTP PROBE TO FINISH" 
cmd=slim info=continue.after mode='probe' message='no input required, execution will resume when HTTP probing is completed' 
cmd=slim prompt='waiting for the HTTP probe to finish'
cmd=slim state=http.probe.running
cmd=slim info=http.probe.ports count='1' targets='55211' 
cmd=slim info=http.probe.commands count='1' commands='GET /docs' 
cmd=slim info=http.probe.call status='error' method='GET' target='http://127.0.0.1:55211/docs' attempt='1' error='Get "http://127.0.0.1:55211/docs": read tcp 127.0.0.1:64409->127.0.0.1:55211: read: connection reset by peer' time='2024-06-07T06:33:15Z' 
cmd=slim info=http.probe.call status='error' method='GET' target='http://127.0.0.1:55211/docs' attempt='2' error='Get "http://127.0.0.1:55211/docs": read tcp 127.0.0.1:64410->127.0.0.1:55211: read: connection reset by peer' time='2024-06-07T06:33:23Z' 
cmd=slim info=http.probe.call target='http://127.0.0.1:55211/docs' attempt='3' error='Get "http://127.0.0.1:55211/docs": EOF' time='2024-06-07T06:33:31Z' status='error' method='GET' 
cmd=slim info=http.probe.call error='Get "http://127.0.0.1:55211/docs": read tcp 127.0.0.1:64412->127.0.0.1:55211: read: connection reset by peer' time='2024-06-07T06:33:47Z' status='error' method='GET' target='http://127.0.0.1:55211/docs' attempt='4' 
cmd=slim info=http.probe.call method='GET' target='http://127.0.0.1:55211/docs' attempt='5' error='Get "http://127.0.0.1:55211/docs": read tcp 127.0.0.1:64413->127.0.0.1:55211: read: connection reset by peer' time='2024-06-07T06:33:55Z' status='error' 
cmd=slim info=http.probe.summary total='5' failures='5' successful='0' 
cmd=slim state=http.probe.done warning=no.successful.calls 
cmd=slim info=event message='HTTP probe is done' 
cmd=slim error=probe.error message='no.successful.calls'
mint: container stdout:
mint: container stderr:
mint: end of container logs =============
cmd=slim state=exited code=-1
cmd=slim info=container.inspector.cleanup name='mintk_3993_20240607063300' id='1fa920da9f1076093a76bb44fe6e86f19ee7bde06e6a268cda558f76dc694297' 
cmd=slim state=container.target.shutdown.start
cmd=slim state=container.target.shutdown.done
cmd=slim info=report file='slim.report.json' 
cmd=slim info=exit code='-1' version='darwin/amd64|Aurora|1.41.2|d6f9778dcd40dfd2108737070b85d929c9806de6|2024-05-30_03:14:56PM' location='/usr/local/bin' 
app='mint' message='GitHub Discussions' info='https://github.com/mintoolkit/mint/discussions'
app='mint' message='Join the CNCF Slack channel to ask questions or to share your feedback' info='https://cloud-native.slack.com/archives/C059QP1RH1S'
app='mint' message='Join the Discord server to ask questions or to share your feedback' info='https://discord.gg/fAvq4ruKsG'

Using the same files as before... Am I missing something? Or is there anything else I have to check for updates?

[kcq]

The new release (1.41.3) now shouldn't need the --include-path flag for PIL.

This command should now produced a working image: mint slim --http-probe-full --http-probe-cmd /docs --http-probe-apispec /openapi.json YOUR_TARGET_IMAGE

Alternatively this should produce the same results: mint slim --http-probe-full --http-probe-cmd /docs --http-probe-cmd-upload generate.image:/upload YOUR_TARGET_IMAGE

The output you shared looks a bit odd because it shows that all http probe commands failed. Just in case, make sure to delete all sensor volumes. Do docker volume ls and then do docker volume rm AAA where AAA is mint-sensor.1.41.2 or something similar.

It'll also help to look at the logs for the temporary container mint creates (when you do docker ps that's the containers with themintk_ names, so you can do docker logs mintk_AAA). Adding the --debug flag should provide extra info too (e.g., mint --debug slim --http-probe-full --http-probe-cmd /docs --http-probe-apispec /openapi.json YOUR_TARGET_IMAGE).