Yeah this happens a lot, especially if you’ve only compared the Python wrapper against the raw llama.cpp binary.

The wrapper adds extra cost just for being in Python. If it calls into C++ too often (say per-token callbacks or too much tokenization in Python), that overhead is noticeable when the prompt is short. The GIL can also cause delays if you mix threading or async badly.

Also, The first forward pass is always the heaviest. It needs to build KV caches, maybe dequantize weights, and on GPU it often has to spin up kernels for the first time. That’s why TTFT feels much higher than per-token latency. In C++ CLI they do some of this a bit more directly, so Python shows more of the raw cost.

If the wrapper is doing tokenization in Python, that can dominate when the prompt is short. Also if you aren’t using prompt caching, you pay the full cost every time even if the prefix is repeated.

How I’d debug it:

• Add simple perf_counter timers around tokenization, generation call, and first token callback. That tells me if Python or backend is eating the time.
• Compare with the C++ CLI on the same prompt to get the baseline. If CLI is faster, the wrapper layer is guilty.
• Run a warm-up generation right after loading the model. If the second run is much faster, that means initialization/dequantization was the big hit.
• If it’s GPU, I’d check with Nsight or nvprof to see if kernel launch overhead or transfers are the delay. On CPU, perf or just logging thread pool startup is enough.

How I’d fix it:

• Move tokenization into the C++ side if possible.
• Pre-warm the model on startup with a dummy forward pass so users never see that first-hit latency.
• Keep the model in memory and reuse it between calls.
• For streaming, batch tokens instead of sending them back to Python one by one.
• If I absolutely need really low TTFT, I’d consider serving from a thin C++ worker and only talk to it from Python, so Python isn’t in the hot path.

Long story short : Python overhead + cold start cost + tokenization are the usual suspects. The fix is mostly warm-up and pushing more work into native code.