# Session routing

Hosted requests are load-balanced across many compute nodes. Sending a session identifier keeps a conversation on the node that already holds its KV cache, so the shared prefix is not recomputed on every turn.

## Send X-Session-ID

Set the `X-Session-ID` header to a value that is stable for the life of one conversation or agent run. Requests carrying the same value are consistently routed to the same worker.

```
curl https://api.ifm.ai/v1/chat/completions \
  -H "Authorization: Bearer $IFM_API_KEY" \
  -H "X-Session-ID: conv_8f2c14ab" \
  -H "Content-Type: application/json" \
  -d '{"model": "IFM/K2-Horizon-375B-A23B", "messages": [...]}'
```

With the OpenAI SDKs, pass it per request via `extra_headers`:

```
resp = client.chat.completions.create(
    model="IFM/K2-Horizon-375B-A23B", messages=messages,
    extra_headers={"X-Session-ID": session_id},
)
```

## Choosing a value

| Rule | Why |
| --- | --- |
| Stable per conversation | The whole point is that turn N+1 reaches the node that served turn N. |
| Opaque and unguessable | Use a random ID you mint — never an email, user name, or raw account ID. |
| Never shared across users | Two users on one ID land on one node and share nothing useful, while concentrating load. |
| New ID per agent run | Long-lived agent loops reuse a large prefix; a fresh run has a fresh prefix. |

## What it does and does not change

This is a **latency and efficiency** optimization, and it is best-effort. Results are identical with or without the header — no state is stored on the node between requests, and nothing about your conversation is retained server-side. Omit the header and requests are distributed by load as usual; every turn simply re-reads the full prompt.

Affinity can break at any time — nodes are added, drained, and restarted, and a cache entry can be evicted under memory pressure. Treat a cache hit as a bonus, never a guarantee, and never as a substitute for sending the full conversation history on each request.

Where it pays off most: long multi-turn chats, agent loops that resend a growing tool transcript, and any workload with a large shared system prompt.

## Under the hood

The hosted API balances traffic with the [vLLM Router](https://github.com/vllm-project/router) using its `consistent_hash` policy, which routes requests carrying the same session identifier to the same backend worker. See the router's [load-balancing documentation](https://github.com/vllm-project/router/tree/main/docs/load_balancing#consistent-hash) for the routing-key priority order and the full policy set.

_Last updated: September 2, 2026_