Mixture-of-experts models as local decision models

A 20 GB file with 3 billion active parameters

The default file of qwen3.6:35b-a3b in our library is 20419565568 bytes. That is 20.4 GB on disk for q4_k_m. Then you open Qwen’s card and it says “Number of Parameters: 35B in total and 3B activated”. So which one am I running, a 35B model or a 3B one?

I ran it on JevBench’s public items to find out. It got 197 of 231 correct, with a 95% interval of 0.8 to 0.897 on accuracy. Its ECE is 0.034139, the lowest of the 15 complete runs on our benchmark page. Next is qwen3.8:27b at 0.04756.

Do not read too much into that gap. It is one run of the default quant, and our runner puts a bootstrap interval on accuracy only, so the ECE comes with no interval at all. Still, the best calibration number we measured belongs to a model that activates 3 billion parameters out of 35, and it is one of only two mixture-of-experts models in the library.

What a3b and a4b mean

Both names carry two sizes, and the letter A separates them. Gemma’s card says it directly: “The “A” in 26B A4B stands for “active parameters” in contrast to the total number of parameters the model contains.” Qwen’s card never expands A3B itself, but it gives the same pair of numbers: 35B in total and 3B activated.

The total is every expert stored in the file. Qwen3.6-35B-A3B has 256 experts and 40 layers, and a token is routed to 8 of those experts plus 1 shared. Gemma 4 26B A4B has 25.2B total parameters and 3.8B active ones, 30 layers, and 8 active experts out of 128 plus 1 shared. So each token touches only a small part of the weights. I computed the active share from the two cards: 3 of 35 is 8.6% for Qwen, and 3.8 of 25.2 is 15.1% for Gemma.

How is that different from a dense model? Qwen3.8-27B is the clean comparison, because its card calls it a “dense model” with 27B parameters and 64 layers. Its layout is (Gated DeltaNet → FFN), with a plain FFN where Qwen3.6 has MoE. There is no router there, so every token goes through all of the weights. Gemma 4 12B is dense in the same way, at 11.95B parameters and 48 layers.

How ardana reads a mixture-of-experts model

I did nothing special for these two. qwen3.6:35b-a3b and gemma-4:26b-a4b sit in the library as instruct models with the chat layout, the same kind as every other instruct model there. You pull them by name.

ardana pull qwen3.6:35b-a3b
ardana pull gemma-4:26b-a4b

Neither comes with a decider_config.json, so ardana gives each one the stock profile in the chat layout: temperature 1.0 and no per-type temperatures. ardana show prints it as chat (instruct model read zero-shot), word for word. The GGUF is loaded through llama.cpp, with all layers offloaded by default and n_ctx at 40,960.

So what does reading zero-shot look like? ardana renders the tokenizer’s chat template for one user turn, with enable_thinking=False when the template mentions it. The head, context and question blocks form that turn, then the tail and Answer: ( follow. A thinking block the template leaves open is closed before the answer. The runtime returns the label logits at the answer slot.

The request does not change either. It is POST /v1/systemone with model, state and questions, and each question carries type, instructions and criteria. The model reads everything once and never writes text, which you can see in the usage of the README’s sample response:

"usage": { "input_tokens": 72, "output_tokens": 0 }

Our benchmark runs went through the same path. Each one was an ardana pull and then a server started with the command below. JevBench’s TypeSafeAdapter sent one request at a time.

ardana serve --port 8011 --default-model <model> --keep-alive 2h

Two mixture-of-experts models beside the dense ones

I ran every model once through ardana 0.2.2 on JevBench’s 231 public items, 111 of them hard. All rows are default quants except decider:4b-q8_0. ECE is rounded to three places.

Model	Correct	Hard	ECE
qwen3.8:27b	200	82	0.048
gemma-4:26b-a4b	199	81	0.079
gemma-4:12b	198	80	0.089
qwen3.6:35b-a3b	197	79	0.034
decider:4b-q8_0	191	72	0.073
gemma-4:e4b	178	63	0.130

The top four are 200, 199, 198 and 197 correct, three items apart. Their 95% bootstrap intervals all overlap each other, and they overlap the interval of decider:4b-q8_0 as well. So who wins? Nobody, on this data. The two mixture-of-experts models sit inside the dense group, and that is all the intervals let me say.

Look at the two Gemma rows. gemma-4:26b-a4b has one more correct item than the dense gemma-4:12b, and it is on the hard tier, 81 against 80. Both get 70 of 72 on the standard tier. One item out of 231 is noise.

Calibration is the interesting column. qwen3.6:35b-a3b has the lowest ECE of the 15 complete runs, 0.034 with qwen3.8:27b next at 0.048, and it is lowest on the hard tier too, 0.081 against 0.104. On the standard tier it flips: 0.094, above gemma-4:12b at 0.016, gemma-4:26b-a4b at 0.023 and qwen3.8:27b at 0.054. ECE here is top-label over ten equal bins and comes with no interval, and each number is one run. I read it as one good measurement, and I would not rank on it.

Two things are absent from the table. gemma-4:31b stopped after 4 items with 3 errors, so it is no result. And only the default quants of the two mixture-of-experts models were measured.

You pay for every parameter in memory

So what does the small active count save you in RAM? For Gemma, Google’s docs answer it themselves: the model “only activates 4 billion parameters per token”, yet “all 26 billion parameters must be loaded into memory to maintain fast routing and inference speeds”. Their table of approximate memory to load the weights gives 14.4 GB at Q4_0 for the 26B A4B against 6.7 GB for the dense 12B, and 57.7 GB against 26.7 GB at BF16. Those figures carry a 20% overhead and leave the context window out. ardana’s default n_ctx is 40,960.

The files in the library tell the same story. gemma-4:26b-a4b at its default q4_0 is 14.6 GB, which is 2.02x the bytes of gemma-4:12b and 5.40x decider:4b. qwen3.6:35b-a3b at q4_k_m is 20.4 GB, 7.54x decider:4b and 1.08x the dense qwen3.8:27b. ardana loads that GGUF through llama.cpp, and the default gpu_layers is -1, so all layers are offloaded.

We keep llama.cpp’s default load mode, which memory-maps the whole GGUF where the device supports it and prefetches it. Every expert sits in the loaded tensors, none is read on demand, and the router picks per token. Hugging Face’s general rule for these models: “all parameters need to be loaded in RAM”.

Only the active parameters compute, 15.1% of gemma-4:26b-a4b and 8.6% of qwen3.6:35b-a3b, and every expert is in the file ardana maps. Qwen’s card states no memory figure, so the file is its only sourced size.

What about speed? The Gemma card says the model “runs almost as fast as a 4B-parameter model”. That is Google’s sentence, and it comes with no measured figure. Our benchmark run recorded no latency, no duration and no hardware per model, so I cannot confirm it under ardana. Bytes I can count, and by bytes gemma-4:26b-a4b is twice the dense 12B.

Which one to pull

I choose by the memory I have, and I read the file size before the parameter count. decider:4b-q8_0 is 4.5 GB and got 191 of 231. gemma-4:12b is 7.2 GB, gemma-4:26b-a4b is 14.6 GB and qwen3.6:35b-a3b is 20.4 GB. The accuracy intervals of all four overlap, so I cannot rank them on correct answers.

When a 7.2 GB file is the most I can load, I pull decider:4b-q8_0 or gemma-4:12b. The intervals give me no reason to spend 14.6 GB on gemma-4:26b-a4b while the dense 12b sits at half the bytes. If a 20.4 GB file fits and calibration matters most, qwen3.6:35b-a3b had the lowest ECE of the 15 complete runs, 0.034139, in one run with no interval.

ardana pull qwen3.6:35b-a3b

Only the default quants of the two mixture-of-experts models are measured, so the q8_0 files are still untested here. The benchmark overview prints each size’s accuracy, interval, Brier and ECE. The qwen3.6 and gemma-4 pages add file size and tier tables. Run it on your own states and tell me what you get!