LiteLLM
Route requests to BlazeInfer through the LiteLLM proxy or SDK, with session IDs forwarded for better KV cache reuse.
LiteLLM is an open-source LLM gateway: a proxy server (and Python SDK) that gives your apps one OpenAI-compatible endpoint in front of many model providers. BlazeInfer plugs in as an OpenAI-compatible provider. This page was tested with LiteLLM 1.108.4.
Prerequisites
- A BlazeInfer account with credits and an API key (see Quickstart).
BLAZE_API_KEYset in the environment that runs LiteLLM.
Configure the proxy
File: config.yaml
model_list:
- model_name: deepseek-v4.1-flash
litellm_params:
model: openai/deepseek-ai/deepseek-v4.1-flash
api_base: https://blaze.e2enetworks.net/v1
api_key: os.environ/BLAZE_API_KEY
- model_name: glm-5.3-flash
litellm_params:
model: openai/zai-org/glm-5.3-flash
api_base: https://blaze.e2enetworks.net/v1
api_key: os.environ/BLAZE_API_KEY
general_settings:
forward_client_headers_to_llm_api: trueThe openai/ prefix tells LiteLLM to use its OpenAI-compatible client. The rest
of model is the BlazeInfer model ID; see the console
for every available ID.
Start the proxy:
litellm --config config.yamlForward client headers for session affinity
By default, LiteLLM does not pass your clients' request headers on to the
model provider. Turn on forward_client_headers_to_llm_api (shown above) so
that the session ID reaches BlazeInfer.
You do not need to send x-session-id specifically. BlazeInfer reads LiteLLM's
own session header, x-litellm-session-id, as is and uses it as the session ID,
so a client that already sets LiteLLM's session ID needs no change. If a request
carries both, x-session-id wins.
With the session ID forwarded, BlazeInfer keeps requests from the same session on the same model replica, so the KV cache is reused across turns. See Session affinity & caching.
To forward headers only for BlazeInfer models instead of every model on the
proxy, list them under model_group_settings:
litellm_settings:
model_group_settings:
forward_client_headers_to_llm_api:
- deepseek-v4.1-flash
- glm-5.3-flashLiteLLM forwards custom headers that start with x-. It does not forward
Authorization, so your clients' LiteLLM keys never reach BlazeInfer; the proxy
sends your BLAZE_API_KEY instead.
Call the proxy
Clients call LiteLLM as they would any OpenAI-compatible API, and send their session ID with each request in a conversation:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000",
api_key="sk-your-litellm-key",
)
response = client.chat.completions.create(
model="deepseek-v4.1-flash",
messages=[{"role": "user", "content": "Say hello in one sentence."}],
extra_headers={"x-litellm-session-id": "conv-8f3a2c"},
)
print(response.choices[0].message.content)Use the LiteLLM SDK directly
Without the proxy, pass the base URL, key and header on each call:
import os
import litellm
response = litellm.completion(
model="openai/deepseek-ai/deepseek-v4.1-flash",
api_base="https://blaze.e2enetworks.net/v1",
api_key=os.environ["BLAZE_API_KEY"],
messages=[{"role": "user", "content": "Say hello in one sentence."}],
extra_headers={"x-session-id": "conv-8f3a2c"},
)
print(response.choices[0].message.content)