Can you guys supply the Q5_K_M version?
I LOVE testing our your LLMs guys so let me just start by giving you props for the work you are doing.
I have a headless server PC with a RTX 4090 so I can USUALLY fit the Q5_K_M variant well since I do not have the large OS desktop overhead which frees up more VRAM.
If was wondering if you guys have the resources/time if you could output a Q5_K_M variant? I'm sure there are others who would greatly appreciate it besides me!
Keep up the inventive work!
We'll look into it 🙂
Hi, have you considered Q4_K_M?
AFAIK it is performing very decent - we aimed at keeping as much performance as possible and we also made sure it is not prolonging reasoning.
In my opinion it should be faster than Q5_K_M, it is not purely memory tradeoff at least for single request on RTX4090.
What do you expect the Q5_K_M to solve what Q4_K_M does not have?
Fair question. My understanding is that a higher quant stays closer to what the full unquantized model would output for the same prompt. So I figured Q5_K_M would lose a little less quality, especially on longer or trickier reasoning. If your testing shows Q4_K_M holds up that well, I'm happy to stick with it. Thanks for explaining!
Some numbers on what Q5_K_M would buy you on a 24 GB card. The Q4_K_M and Q6_K files here are within a few hundred bytes of ukisai's Swift-1.5-Qwen3.8-27B quants (17,442,400,352 vs 17,442,399,936 and 23,860,566,112 vs 23,860,565,696 bytes), so their Q5_K_M (20,923,877,056 bytes = 19.49 GiB) is a good predictor for a ThinkingCap Q5_K_M.
The KV cache is small for this architecture because only 16 of the 64 layers are full attention (the other 48 keep a fixed-size recurrent state): 16 × 4 KV heads × 256 × 2 (K,V) × 2 bytes = 64 KiB/token in f16, 34 KiB/token with q8_0. The files also carry the MTP layer (nextn_predict_layers = 1, blk.64). If you run it with --spec-type draft-mtp, that's one more attention layer, so 68 KiB/token.
On 24 GiB, budgeting 0.5 GiB of runtime plus 10% for compute buffers:
- Q4_K_M (16.24 GiB): ~83K context in f16, ~157K with q8_0 KV
- Q5_K_M (~19.49 GiB): ~30K in f16, ~57K with q8_0
- Q6_K (22.22 GiB): leaves 1.78 GiB for KV, compute buffers and the CUDA context, which is too tight for useful context
So on a headless 4090, Q5_K_M works if ~32K f16 or ~56K q8_0 is enough, and Q4_K_M is the one that keeps long context. The 10% is on the conservative side. llama.cpp's --fit keeps 1024 MiB free by default and may get a bit more out of it.
Disclosure: I maintain this free calculator, and the Qwen3.8 27B page has the same numbers for other cards: https://modelvram.com/llm-vram-calculator/qwen3.8-27b/
If you look closely, you'll notice that this isn't a standard Q4_K_M.🤔😀