I also have a hypothesis that this model can be efficiently downsized not by pruning experts, but by using merges and LoRAs to downsize their unique parameter count. The merge would be most of the shared parameters, and the routing table need not change.
I'm building up a new version of my pipeline to test this hypothesis. I suspect it'd let us get most of the performance in <12B parameters.
This is a very cool release! I really enjoy the ShiningValiant series!
Do you see potential to prune experts or layers from the gpt-oss-20b model to downsize it, and then finetune?
The premise of getting things structured as soon as possible is quite sound. My reaction to CCL is - can you be agnostic to the many different ways things are processed? What is the most essential way that data and processes relate to each other for what you're doing?
Now imagine this as a hashtag generator and so a RAG search can find great context. :)
Neat! I've transitioned from wanting more from a model's one-shot answers to breaking things down and walking through the problem with cached context. This effectively means simulating most of the thinking block, but by tool usage and RAG.
I'm happily using our models from months ago to do it. If anything - even Lamarck 0.7's use of thinking blocks are a bit much. I'm using Lamarck 0.7 Fusion (my best GPQA model, though it didn't break your record and is best used where modest IFEVAL isn't a blocker) and /nothink with ValiantLab's Qwen3 models in concert.
I suspect I'll try some merges soon to give this toolchain better models, leaderboard or no leaderboard!
I've been using Esper3 8B and 14B for first-pass code review. I am quite pleased.
Have you considered fine-tuning a 1.7B or smaller model for autocomplete?
I've been thinking a lot about using small caches of embeddings for local RAG lately. Have you considered an HTTP caching proxy like Squid as a low-impact source? It would retrieve what a user is reading anyway, and what's in their field of interest. A browser extension to signal some limited ingestion when a page is bookmarked might fit a lot of use cases.
For many reasons, smart management of context windows is my top priority with AI now!
@Inschrift-Spruch-Raum ,I am looking through recent PRs to mergekit, and I am optimistic that Lamarck's recipes will be working again soon!
When that happens, there will be two efforts: one to make a compelling non-CoT model, and another to blend CoT in right amounts.
Lamarck's multilingual capabilities improved noticeably from light influence of Krystalan/DRT-14B in v0.6, and merging from other CoT models like DeepSeek R1 is a matter of careful moderation. I will always put the overall apparent quality of translation, prose, and reasoning first.
