Container for MLC LLM project using Apache TVM Unity with CUDA, cuDNN, CUTLASS, FasterTransformer, and FlashAttention-2 kernels.
To quantize and benchmark a model, run the benchmark.sh script from the host (outside container)
HUGGINGFACE_TOKEN=hf_abc123def ./benchmark.sh meta-llama/Llama-2-7b-hfThis will run the quantization and benchmarking in the MLC container, and save the performance data to jetson-containers/data/benchmarks/mlc.csv. If you are accessing a gated model, substitute your HuggingFace account's API key above. Omitting the model will benchmark a default set of Llama models. See benchmark.sh for various environment variables you can set.
AVERAGE OVER 3 RUNS, input=16, output=128
/data/models/mlc/0.1.0/Llama-2-7b-hf-q4f16_ft/params: prefill_time 0.025 sec, prefill_rate 632.8 tokens/sec, decode_time 2.731 sec, decode_rate 46.9 tokens/sec
The prefill time is how long the model takes to process the input context before it can start generating output tokens. The decode rate is the speed at which it generates output tokens. These results are averaged over the number of prompts, minus the first warm-up.
CONTAINERS
mlc:0.1.0 |
|
|---|---|
| Aliases | mlc |
| Requires | L4T ['>=36'] |
| Dependencies | build-essential cuda:12.2 cudnn python numpy cmake onnx pytorch:2.2 torchvision huggingface_hub rust transformers |
| Dependants | l4t-text-generation local_llm nano_llm:24.4 nano_llm:24.4.1 nano_llm:24.5 nano_llm:24.5.1 nano_llm:24.6 nano_llm:main |
| Dockerfile | Dockerfile |
| Images | dustynv/mlc:0.1.0-r36.3.0 (2024-06-18, 7.1GB) |
| Notes | mlc-ai/mlc-llm commit SHA 607dc5a |
mlc:0.1.0-builder |
|
|---|---|
| Aliases | mlc |
| Requires | L4T ['>=36'] |
| Dependencies | build-essential cuda:12.2 cudnn python numpy cmake onnx pytorch:2.2 torchvision huggingface_hub rust transformers |
| Dockerfile | Dockerfile |
| Notes | mlc-ai/mlc-llm commit SHA 607dc5a |
CONTAINER IMAGES
| Repository/Tag | Date | Arch | Size |
|---|---|---|---|
dustynv/mlc:0.1.0-r36.3.0 |
2024-06-18 |
arm64 |
7.1GB |
dustynv/mlc:0.1.1-r36.2.0 |
2024-04-18 |
arm64 |
7.4GB |
dustynv/mlc:0.1.1-r36.3.0 |
2024-06-18 |
arm64 |
7.4GB |
dustynv/mlc:3feed05-builder-r36.2.0 |
2024-02-16 |
arm64 |
10.8GB |
dustynv/mlc:3feed05-r36.2.0 |
2024-02-16 |
arm64 |
9.6GB |
dustynv/mlc:51fb0f4-builder-r35.4.1 |
2024-02-16 |
arm64 |
9.5GB |
dustynv/mlc:51fb0f4-builder-r36.2.0 |
2024-02-16 |
arm64 |
10.6GB |
dustynv/mlc:5584cac-r36.2.0 |
2024-02-22 |
arm64 |
9.6GB |
dustynv/mlc:607dc5a-r36.2.0 |
2024-02-27 |
arm64 |
9.6GB |
dustynv/mlc:c30348a-r36.2.0 |
2024-02-20 |
arm64 |
9.6GB |
dustynv/mlc:dev-r35.3.1 |
2023-10-30 |
arm64 |
9.0GB |
dustynv/mlc:dev-r35.4.1 |
2023-12-16 |
arm64 |
9.4GB |
dustynv/mlc:dev-r36.2.0 |
2023-12-16 |
arm64 |
10.6GB |
dustynv/mlc:r35.2.1 |
2023-12-16 |
arm64 |
9.4GB |
dustynv/mlc:r35.3.1 |
2023-11-05 |
arm64 |
8.9GB |
dustynv/mlc:r35.4.1 |
2024-01-27 |
arm64 |
9.4GB |
dustynv/mlc:r36.2.0 |
2024-03-09 |
arm64 |
9.6GB |
Container images are compatible with other minor versions of JetPack/L4T:
• L4T R32.7 containers can run on other versions of L4T R32.7 (JetPack 4.6+)
• L4T R35.x containers can run on other versions of L4T R35.x (JetPack 5.1+)
RUN CONTAINER
To start the container, you can use jetson-containers run and autotag, or manually put together a docker run command:
# automatically pull or build a compatible container image
jetson-containers run $(autotag mlc)
# or explicitly specify one of the container images above
jetson-containers run dustynv/mlc:0.1.1-r36.3.0
# or if using 'docker run' (specify image and mounts/ect)
sudo docker run --runtime nvidia -it --rm --network=host dustynv/mlc:0.1.1-r36.3.0
jetson-containers runforwards arguments todocker runwith some defaults added (like--runtime nvidia, mounts a/datacache, and detects devices)
autotagfinds a container image that's compatible with your version of JetPack/L4T - either locally, pulled from a registry, or by building it.
To mount your own directories into the container, use the -v or --volume flags:
jetson-containers run -v /path/on/host:/path/in/container $(autotag mlc)To launch the container running a command, as opposed to an interactive shell:
jetson-containers run $(autotag mlc) my_app --abc xyzYou can pass any options to it that you would to docker run, and it'll print out the full command that it constructs before executing it.
BUILD CONTAINER
If you use autotag as shown above, it'll ask to build the container for you if needed. To manually build it, first do the system setup, then run:
jetson-containers build mlcThe dependencies from above will be built into the container, and it'll be tested during. Run it with --help for build options.