Fix chat hang: stop sending num_ctx to the LLM gateway

num_ctx makes Ollama reload the model at a different context size (an
unload+load). Against a resident qwen3:32b at ctx 32768, our request asked for
8192 — forcing a 29GB reload on every call, which stalls indefinitely whenever
the backend's eviction path is unhealthy.

A/B against the gateway, identical requests otherwise:
  stream, no num_ctx    -> 200 in 0.31s
  stream, num_ctx=8192  -> http 000, hangs 60s+

It was also counterproductive: it shrank the context the model already served.
This commit is contained in:
Ryan Malloy 2026-07-14 11:30:08 -06:00
parent 953b3efd58
commit 3891dd3566

View File

@ -97,7 +97,10 @@ async def chat_completion_stream(
"messages": messages,
"temperature": 0.3,
"max_tokens": settings.chat_max_tokens,
"num_ctx": settings.chat_max_tokens,
# No num_ctx: it makes Ollama reload the model at a different context
# size, which is an unload+load. That stalls whenever the backend's
# eviction path is unhealthy, and it would *shrink* the context the
# resident model already serves. Let the gateway pick the context.
"stream": True,
},
) as resp: