Userguide changes

docs/Dockerfile.docs

@@ -68,11 +68,17 @@ RUN pip3 install \
       sphinx-book-theme \
       sphinx-copybutton \
       sphinx-design \
+      sphinx-mdinclude \
       sphinx-prompt \
       sphinx-sitemap \
       sphinx-tabs \
       sphinxcontrib-bibtex
 
+# install nvidia-sphinx-theme
+RUN pip3 install \
+      --index-url https://urm.nvidia.com/artifactory/api/pypi/ct-omniverse-pypi/simple/ \
+      nvidia-sphinx-theme
+
 # Set visitor script to be included on every HTML page
 ENV VISITS_COUNTING_SCRIPT="//assets.adobedtm.com/b92787824f2e0e9b68dc2e993f9bd995339fe417/satelliteLib-7ba51e58dc61bcb0e9311aadd02a0108ab24cc6c.js"
 

docs/README.md

@@ -124,9 +124,9 @@ Triton supports batching individual inference requests to improve compute resour
   - [Queuing Policies](user_guide/model_configuration.md#queue-policy)
   - [Ragged Batching](user_guide/ragged_batching.md)
 - [Sequence Batcher](user_guide/model_configuration.md#sequence-batcher)
-  - [Stateful Models](user_guide/architecture.md#stateful-models)
-  - [Control Inputs](user_guide/architecture.md#control-inputs)
-  - [Implicit State - Stateful Inference Using a Stateless Model](user_guide/architecture.md#implicit-state-management)
+  - [Stateful Models](user_guide/model_execution.md#stateful-models)
+  - [Control Inputs](user_guide/model_execution.md#control-inputs)
+  - [Implicit State - Stateful Inference Using a Stateless Model](implicit_state_management.md#implicit-state-management)
   - [Sequence Scheduling Strategies](user_guide/architecture.md#scheduling-strategies)
     - [Direct](user_guide/architecture.md#direct)
     - [Oldest](user_guide/architecture.md#oldest)

docs/_reference/tritonclient_api.rst

@@ -28,7 +28,7 @@
 Python tritonclient Package API
 ===============================
 
-tritonclient python package is hosted at the `pyPI.org <https://pypi.org/project/tritonclient/>`_. This package documentation for tritonclient is genenerated by sphinx autosummary extension.
+tritonclient python package is hosted at the `pyPI.org <https://pypi.org/project/tritonclient/>`_.
 
 .. autosummary::
    :toctree: tritonclient

docs/_static/switcher.json

@@ -0,0 +1,31 @@
+[
+    {
+      "version": "dev",
+      "url": "https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/"
+    },
+    {
+      "name": "2.49 (stable)",
+      "version": "2.49",
+      "url": "https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/"
+    },
+    {
+      "name": "2.48",
+      "version": "2.48",
+      "url": "https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/"
+    },
+    {
+      "name": "2.47",
+      "version": "2.47",
+      "url": "https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/"
+    },
+    {
+      "name": "2.46",
+      "version": "2.46",
+      "url": "https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/"
+    },
+    {
+      "name": "2.45",
+      "version": "2.45",
+      "url": "https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/"
+    }
+]

docs/backend/vllm.rst

@@ -0,0 +1,11 @@
+########
+vLLM
+########
+
+.. toctree::
+    :hidden:
+    :caption: vLLM
+    :maxdepth: 2
+
+    vllm_backend/README
+    Multi-LoRA <vllm_backend/docs/llama_multi_lora_tutorial>

docs/client_doc/api_reference.rst

@@ -0,0 +1,15 @@
+
+####
+API Reference
+####
+
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   .. Placeholder Openai Documentation
+
+   OpenAI API (BETA) <openai_README.md> 
+   kserve
+
+

docs/client_doc/in_process.rst

@@ -0,0 +1,40 @@
+####
+In-Process Triton Server API
+####
+
+
+The Triton Inference Server provides a backwards-compatible C API/ python-bindings/java-bindings that
+allows Triton to be linked directly into a C/C++/java/python application. This API
+is called the "Triton Server API" or just "Server API" for short. The
+API is implemented in the Triton shared library which is built from
+source contained in the `core
+repository <https://github.com/triton-inference-server/core>`__. On Linux
+this library is libtritonserver.so and on Windows it is
+tritonserver.dll. In the Triton Docker image the shared library is
+found in /opt/tritonserver/lib. The header file that defines and
+documents the Server API is
+`tritonserver.h <https://github.com/triton-inference-server/core/blob/main/include/triton/core/tritonserver.h>`__.
+`Java bindings for In-Process Triton Server API <../customization_guide/inprocess_java_api.html#java-bindings-for-in-process-triton-server-api>`__
+are built on top of `tritonserver.h` and can be used for Java applications that
+need to use Tritonserver in-process.
+
+All capabilities of Triton server are encapsulated in the shared
+library and are exposed via the Server API. The `tritonserver`
+executable implements HTTP/REST and GRPC endpoints and uses the Server
+API to communicate with core Triton logic. The primary source files
+for the endpoints are `grpc_server.cc <https://github.com/triton-inference-server/server/blob/main/src/grpc/grpc_server.cc>`__ and 
+`http_server.cc <https://github.com/triton-inference-server/server/blob/main/src/http_server.cc>`__. In these source files you can
+see the Server API being used.
+
+You can use the Server API in your own application as well. A simple
+example using the Server API can be found in
+`simple.cc <https://github.com/triton-inference-server/server/blob/main/src/simple.cc>`__.
+
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   C/C++ <../customization_guide/inprocess_c_api.md>
+   python
+   Java <../customization_guide/inprocess_java_api.md>
+

docs/client_doc/kserve.rst

@@ -0,0 +1,15 @@
+####
+KServe API
+####
+
+
+Triton uses the 
+`KServe community standard inference protocols <https://github.com/kserve/kserve/tree/master/docs/predict-api/v2>`__
+to define HTTP/REST and GRPC APIs plus several extensions.
+
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   HTTP/REST and GRPC Protocol <../customization_guide/inference_protocols.md> 
+   kserve_extension

docs/client_doc/kserve_extension.rst

@@ -0,0 +1,24 @@
+####
+Extensions
+####
+
+To fully enable all capabilities
+Triton also implements `HTTP/REST and GRPC
+extensions <https://github.com/triton-inference-server/server/tree/main/docs/protocol>`__
+to the KServe inference protocol.
+
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   Binary tensor data extension <../protocol/extension_binary_data.md>
+   Classification extension <../protocol/extension_classification.md>
+   Schedule policy extension <../protocol/extension_schedule_policy.md>
+   Sequence extension <../protocol/extension_sequence.md>
+   Shared-memory extension <../protocol/extension_shared_memory.md>
+   Model configuration extension <../protocol/extension_model_configuration.md>
+   Model repository extension <../protocol/extension_model_repository.md>
+   Statistics extension <../protocol/extension_statistics.md>
+   Trace extension <../protocol/extension_trace.md>
+   Logging extension <../protocol/extension_logging.md>
+   Parameters extension <../protocol/extension_parameters.md>

docs/client_doc/openai_README.md

@@ -0,0 +1,148 @@
+# OpenAI-Compatible Frontend for Triton Inference Server
+
+## Pre-requisites
+
+1. Docker + NVIDIA Container Runtime
+2. A correctly configured `HF_TOKEN` for access to HuggingFace models.
+    - The current examples and testing primarily use the
+      [`meta-llama/Meta-Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)
+      model, but you can manually bring your own models and adjust accordingly.
+
+## VLLM
+
+1. Build and launch the container:
+  - Mounts the `~/.huggingface/cache` for re-use of downloaded models across runs, containers, etc.
+  - Sets the [`HF_TOKEN`](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables#hftoken) environment variable to
+    access gated models, make sure this is set in your local environment if needed.
+
+```bash
+docker build -t tritonserver-openai-vllm -f docker/Dockerfile.vllm .
+
+docker run -it --net=host --gpus all --rm \
+  -v ${HOME}/.cache/huggingface:/root/.cache/huggingface \
+  -e HF_TOKEN \
+  tritonserver-openai-vllm
+```
+
+2. Launch the OpenAI-compatible Triton Inference Server:
+```bash
+cd openai/
+
+# NOTE: Adjust the --tokenizer based on the model being used
+python3 openai_frontend/main.py --model-repository tests/vllm_models/ --tokenizer meta-llama/Meta-Llama-3.1-8B-Instruct
+```
+
+3. Send a `/v1/chat/completions` request:
+  - Note the use of `jq` is optional, but provides a nicely formatted output for JSON responses.
+```bash
+MODEL="llama-3.1-8b-instruct"
+curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
+  "model": "'${MODEL}'",
+  "messages": [{"role": "user", "content": "Say this is a test!"}]
+}' | jq
+```
+
+4. Send a `/v1/completions` request:
+  - Note the use of `jq` is optional, but provides a nicely formatted output for JSON responses.
+```bash
+MODEL="llama-3.1-8b-instruct"
+curl -s http://localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{
+  "model": "'${MODEL}'",
+  "prompt": "Machine learning is"
+}' | jq
+```
+
+5. Benchmark with `genai-perf`:
+```bash
+MODEL="llama-3.1-8b-instruct"
+TOKENIZER="meta-llama/Meta-Llama-3.1-8B-Instruct"
+genai-perf \
+  --model ${MODEL} \
+  --tokenizer ${TOKENIZER} \
+  --service-kind openai \
+  --endpoint-type chat \
+  --synthetic-input-tokens-mean 256 \
+  --synthetic-input-tokens-stddev 0 \
+  --output-tokens-mean 256 \
+  --output-tokens-stddev 0 \
+  --streaming
+```
+
+6. Use the OpenAI python client directly:
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY",
+)
+
+model = "llama-3.1-8b-instruct"
+completion = client.chat.completions.create(
+    model=model,
+    messages=[
+        {
+            "role": "system",
+            "content": "You are a helpful assistant.",
+        },
+        {"role": "user", "content": "What are LLMs?"},
+    ],
+    max_tokens=256,
+)
+
+print(completion.choices[0].message.content)
+```
+
+7. Run tests (NOTE: The server should not be running, the tests will handle starting/stopping the server as necessary):
+```bash
+pytest -v tests/
+```
+
+8. For a list of examples, see the `examples/` folder.
+
+## TensorRT-LLM
+
+**NOTE**: The workflow for preparing TRT-LLM engines, model repository, etc. in order to
+load and test is not fleshed out in the README here yet. You can try using the Triton CLI
+or follow existing TRT-LLM backend examples to prepare a model repository, and point
+at the model repository accordingly when following the examples.
+
+0. Prepare your model repository for a TensorRT-LLM model, build the engine, etc.
+
+1. Build and launch the container:
+  - Mounts the openai source files to `/workspace` for simplicity, later on these will be shipped in the container.
+  - Mounts the `~/.huggingface/cache` for re-use of downloaded models across runs, containers, etc.
+  - Sets the [`HF_TOKEN`](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables#hftoken) environment variable to
+    access gated models, make sure this is set in your local environment if needed.
+
+```bash
+docker build -t tritonserver-openai-tensorrtllm -f docker/Dockerfile.tensorrtllm ./docker
+
+docker run -it --net=host --gpus all --rm \
+  -v ${PWD}:/workspace \
+  -v ${HOME}/.cache/huggingface:/root/.cache/huggingface \
+  -e HF_TOKEN \
+  -w /workspace \
+  tritonserver-openai-tensorrtllm
+```
+
+2. Launch the OpenAI server:
+```bash
+cd openai/
+
+# NOTE: Adjust the --tokenizer based on the model being used
+python3 openai_frontend/main.py --model-repository tests/tensorrtllm_models/ --tokenizer meta-llama/Meta-Llama-3.1-8B-Instruct
+```
+
+3. Send a `/v1/chat/completions` request:
+  - Note the use of `jq` is optional, but provides a nicely formatted output for JSON responses.
+```bash
+MODEL="tensorrt_llm_bls"
+curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
+  "model": "'${MODEL}'",
+  "messages": [{"role": "user", "content": "Say this is a test!"}]
+}' | jq
+```
+
+The other examples should be the same as vLLM, except that you should set `MODEL="tensorrt_llm_bls"`,
+everywhere applicable as seen in the example request above.

docs/client_doc/placeholder.rst

@@ -0,0 +1 @@
+

docs/client_doc/python.rst

@@ -0,0 +1,12 @@
+####
+Python
+####
+
+.. include:: python_readme.rst
+
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   Kafka I/O <../tutorials/Triton_Inference_Server_Python_API/examples/kafka-io/README.md>
+   Rayserve <../tutorials/Triton_Inference_Server_Python_API/examples/rayserve/README.md>