BlazeInferDocs
Integrations

LiteLLM

Route requests to BlazeInfer through the LiteLLM proxy or SDK, with session IDs forwarded for better KV cache reuse.

SupportedTested with: LiteLLM 1.108.4 · Last verified: 2026-11-10

LiteLLM is an open-source LLM gateway: a proxy server (and Python SDK) that gives your apps one OpenAI-compatible endpoint in front of many model providers. BlazeInfer plugs in as an OpenAI-compatible provider. This page was tested with LiteLLM 1.108.4.

Prerequisites

  • A BlazeInfer account with credits and an API key (see Quickstart).
  • BLAZE_API_KEY set in the environment that runs LiteLLM.

Configure the proxy

File: config.yaml

model_list:
  - model_name: deepseek-v4.1-flash
    litellm_params:
      model: openai/deepseek-ai/deepseek-v4.1-flash
      api_base: https://blaze.e2enetworks.net/v1
      api_key: os.environ/BLAZE_API_KEY
  - model_name: glm-5.3-flash
    litellm_params:
      model: openai/zai-org/glm-5.3-flash
      api_base: https://blaze.e2enetworks.net/v1
      api_key: os.environ/BLAZE_API_KEY

general_settings:
  forward_client_headers_to_llm_api: true

The openai/ prefix tells LiteLLM to use its OpenAI-compatible client. The rest of model is the BlazeInfer model ID; see the console for every available ID.

Start the proxy:

litellm --config config.yaml

Forward client headers for session affinity

By default, LiteLLM does not pass your clients' request headers on to the model provider. Turn on forward_client_headers_to_llm_api (shown above) so that the session ID reaches BlazeInfer.

You do not need to send x-session-id specifically. BlazeInfer reads LiteLLM's own session header, x-litellm-session-id, as is and uses it as the session ID, so a client that already sets LiteLLM's session ID needs no change. If a request carries both, x-session-id wins.

With the session ID forwarded, BlazeInfer keeps requests from the same session on the same model replica, so the KV cache is reused across turns. See Session affinity & caching.

To forward headers only for BlazeInfer models instead of every model on the proxy, list them under model_group_settings:

litellm_settings:
  model_group_settings:
    forward_client_headers_to_llm_api:
      - deepseek-v4.1-flash
      - glm-5.3-flash

LiteLLM forwards custom headers that start with x-. It does not forward Authorization, so your clients' LiteLLM keys never reach BlazeInfer; the proxy sends your BLAZE_API_KEY instead.

Call the proxy

Clients call LiteLLM as they would any OpenAI-compatible API, and send their session ID with each request in a conversation:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-your-litellm-key",
)

response = client.chat.completions.create(
    model="deepseek-v4.1-flash",
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
    extra_headers={"x-litellm-session-id": "conv-8f3a2c"},
)
print(response.choices[0].message.content)

Use the LiteLLM SDK directly

Without the proxy, pass the base URL, key and header on each call:

import os
import litellm

response = litellm.completion(
    model="openai/deepseek-ai/deepseek-v4.1-flash",
    api_base="https://blaze.e2enetworks.net/v1",
    api_key=os.environ["BLAZE_API_KEY"],
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
    extra_headers={"x-session-id": "conv-8f3a2c"},
)
print(response.choices[0].message.content)

On this page