The local-model sweet spot for Nowledge Mem: 12B, 26B, or 35B?

Republication note: Republished with the author’s permission from Versun’s original Chinese benchmark. The test data, conclusions, and first-person experience are the author’s. Nowledge Labs provided this edited and abridged English translation, formatting, terminology alignment, and links to the current documentation.

The short answer

Keep Qwen3.6 35B-A3B resident. Load Gemma 4 26B-A4B on demand. Choose Gemma 4 12B when memory matters more than warm-task speed.

Decision guide

Choose by loading pattern, not parameter count.

The surprising result is the tradeoff: 12B saves memory but is 2.8× slower on the warm task. The 26B and 35B models use almost the same RSS and take almost the same warm-task time; cold-load strategy separates them.

Keep resident

Qwen3.6 35B-A3B MLX

Fastest warm task and generation on the tested 64 GB Mac.

2.64 s · 78.1 tok/s

Load on demand

Gemma 4 26B-A4B MLX

Lower total wait than Qwen's 14.19 seconds when the model starts cold.

10.85 s load + task

Protect memory

Gemma 4 12B GGUF

Roughly 7 GiB below the two larger models, with a slower 7.38-second warm task.

10.65 GiB total RSS

All three scored 3/3 across three runs of one fixed nine-check memory task. That is not a general quality tie. The 16/24/32 GB guidance is inferred from this 64 GB RSS sample, not measured on those machines.

Why retest local models now

Nowledge Mem has been Local First from the beginning. Early versions could process memories with the small model bundled in the app. Version 0.5 brought longer and more demanding background work: organizing conversations, merging changes, handling conflicts, and supporting AI Now and Timeline. A 4B model could no longer cover every task, so subscriptions and bring-your-own-key providers became common daily setups.

Since version 0.10, the local runtime has expanded beyond the early M1 Mac setup to desktop and headless Linux. Official Docker images cover CPU, CUDA, and Vulkan. The bundled multimodal model already uses vision for Library document processing. Mem also supports Lemonade, Ollama, and LM Studio; community contributor abn did substantial work on the Lemonade integration.

Versun skipped the general benchmark suites. He used Nowledge Mem’s memory organization, update, noise-filtering, and long-context tasks to answer a narrower question: what should someone run locally today, given quality, waiting time, and memory pressure?

Why I moved Nowledge Mem to a local model

I recently changed Nowledge Mem’s memory processing from a cloud model to a local one. The reason was simple: I subscribe to different token plans, so the provider configured in Mem kept changing. A plan would run out, the intelligent processing would stop, and sometimes I would not notice for days.

A local model removes the quota problem. The remaining question was: which model is a good fit for Nowledge Mem?

My test machine and runtimes were:

• Mac mini with M4 Pro and 64 GB of unified memory
• LM Studio 0.4.19+2
• GGUF runtime: llama.cpp 2.25.2
• MLX runtime: 1.10.1

I had already published a quick first comparison. Wey from Nowledge Labs asked for more context, memory measurements, and a reproducible explanation in this follow-up thread, which led to the fuller test below.

Controlled memory test

Every run used the same settings:

• Context window: 32,768 tokens
• Temperature: 0
• Top P: 1
• Thinking / reasoning: off
• GPU offload: maximum
• Concurrent requests: 1
• Parallel model slots: LM Studio default of 4
• Maximum output: 700 tokens

The system prompt told the model to retain only information that would remain useful, distinguish temporary chat from old and current facts, reject one-off weather or mood, and return strict JSON.

The conversation contained six durable items: a preference for Chinese, an old agent_browser backend, its replacement by opencli, a 20-item automation limit, a rule requiring confirmation before publication, and a preference to retain superseded decisions. It also contained two pieces of noise: hot weather and drinking a glass of water.

The output passed only if it met all nine checks:

1. Exactly six memories.
2. The Chinese-language preference.
3. The old agent_browser decision.
4. The old decision marked superseded.
5. The new opencli decision marked current.
6. The 20-item limit.
7. The confirmation-before-publication rule.
8. The preference to preserve replaced decisions.
9. No weather or drinking-water memory.

Each model ran the same task three times. I checked JSON validity, total time, time to first token, generation speed, and the nine quality conditions. All three models passed all three runs of this fixed prompt. That result does not establish a general quality tie between the models.

The table reports the median memory-task time. Runs two and three benefited from LM Studio’s prompt cache, so “memory task” represents the normal speed of a resident model, not a cold start.

Long-context retrieval

I also generated a long project log and hid this durable fact in the middle:

The project code is Qinglan-7391, and publication requires user confirmation.

The same irrelevant log sentence appeared 180 times before and 180 times after it. When asked to return only the project code, every model answered Qinglan-7391. “Long-input TTFT” below measures the delay from receiving that text to producing the first token.

Measuring peak RSS

The memory sampler unloaded every model, waited for the inference process to exit, recorded an idle LM Studio baseline, then sampled macOS ps RSS every 0.08 seconds while loading the model, idling for 1.5 seconds, running three memory tasks, running the long-context task, and observing another 0.8 seconds.

I recorded both the inference process and the sum of LM Studio with its children. Shared pages may be counted more than once, so summed RSS is useful for comparing models; it is not the exact amount by which free system memory falls.

Results

All three extracted the six durable memories and rejected the temporary details. This test did not show a meaningful quality difference; speed and memory use separated them.

Why was 35B faster than 12B?

Qwen 35B-A3B and Gemma 26B-A4B are mixture-of-experts models. Although the total model is large, only a subset of experts participates in each token. Gemma 12B does not use the same design.

One way to picture it: the 35B model is a large company that calls only a roughly 3B expert group into each meeting. It can have a larger total capacity without doing 35B parameters’ worth of computation every time.

Which model would I choose?

• One default: Qwen3.6 35B-A3B MLX. It had a 2.64-second resident memory task, 78.1 tok/s generation, and the fastest long-input first token. Its 11.55-second cold load means it works best when kept resident.
• Frequent loading and unloading: Gemma 4 26B-A4B MLX. Cold load plus memory processing was about 10.85 seconds, versus 14.19 seconds for Qwen 35B, while generation remained close at 70.6 tok/s.
• For 32 to 64 GB, inferred from RSS: Gemma 4 26B-A4B MLX 4-bit. It is the more conservative memory choice when different tasks may activate more Qwen experts. This test did not run on a 32 GB machine.
• Below 32 GB, inferred from RSS: Gemma 4 12B GGUF. LM Studio’s total peak was about 10.65 GiB, roughly 7 GiB below the two larger models. That leaves more room for other desktop software, but this test did not run on 16 GB or 24 GB machines.

MLX or GGUF?

This was not a format-controlled comparison: the model architectures were different, so it would be wrong to conclude that MLX is always faster than GGUF.

The practical conclusion is narrower. On Apple Silicon, the MLX builds of Gemma 26B-A4B and Qwen 35B-A3B performed very well. I would prefer MLX for Mac-only use and GGUF when I also need to move the model to Windows, Linux, or other llama.cpp tools.

The takeaway

I used to assume that a smaller parameter count meant a faster model. This test went the other way:

• 12B used the least memory but was the slowest.
• 26B-A4B and 35B-A3B were faster.

This is one controlled prompt on one computer. It does not prove that Qwen3.6 35B is smarter on every task. For local memory management on my Mac, however, it is currently the best fit.

Nowledge Labs’ outlook

This benchmark gives today’s answer, not a permanent one. We expect models around 4B to cover memory organization, change merging, conflict handling, and background intelligence again someday. There is no schedule, and this three-model test cannot establish when that will happen.

The local multimodal path already uses vision for Library documents. PII filtering and audio-meeting processing remain future directions, outside the shipped and tested scope of this benchmark.

If you want to keep your own model calls on-device, continue with Use Nowledge Mem with fully local models.

---

Original discussion:

• Versun’s first test
• Wey’s republication request and follow-up questions
• Versun’s complete X Article