# Streaming

Stream tokens as they are generated using server-sent events in the OpenAI-compatible chunk schema. Existing SDK streaming code works unchanged.

## Enable streaming

Set `stream=True` to receive the response incrementally. Reasoning and content arrive as separate deltas — buffer each on its own, then append the finished turn when the stream ends.

```
stream = client.chat.completions.create(
    model=os.environ["IFM_MODEL"], messages=messages, stream=True,
)

content, reasoning = "", ""
for chunk in stream:
    delta = chunk.choices[0].delta
    reasoning += getattr(delta, "reasoning_content", None) or ""
    content   += getattr(delta, "content", None) or ""

messages.append({"role": "assistant", "content": content, "reasoning_content": reasoning})
```

## Streaming tool calls

Tool calls stream too, and `arguments` arrives in fragments across many chunks. Accumulate the fragments per `tool_calls[i].index` and only parse the JSON once the stream has finished — a partial fragment is almost never valid JSON on its own.

```
for chunk in stream:
    for tc in (chunk.choices[0].delta.tool_calls or []):
        buf = calls.setdefault(tc.index, {"name": "", "args": ""})
        if tc.function.name:      buf["name"] += tc.function.name
        if tc.function.arguments: buf["args"] += tc.function.arguments

# parse only after the stream ends
args = json.loads(calls[0]["args"])
```

## Ending a stream

The last content chunk carries `finish_reason` — `"stop"`, `"length"`, or `"tool_calls"` — followed by the `[DONE]` sentinel. Treat a stream that ends without a `finish_reason` as truncated and retry it; do not append a half-finished turn to your history.

Streams cannot be resumed. If the connection drops mid-generation, re-send the request from the last complete turn. Sending the same `X-Session-ID` makes the retry land on the compute node that still holds the prefix — see [Session routing](https://docs.ifm.ai/docs/session-routing.md).

_Last updated: September 2, 2026_