InnerThink: compressing chain-of-thought into 6 continuous vectors

What this is

This project took first place at the Snowflake X Beta Fund X Evermind Agent & Token Economy Hackathon. It was held at the Silicon Valley AI Hub in San Francisco on agents and the token economy: every agent burns tokens, every token costs money, so how do you make that add up at scale.

We went after the most awkward line on the bill. Ask a reasoning model what a $40 jacket costs at 25% off and it writes a hundred-odd tokens of reasoning before telling you $30. You paid for those tokens and you won't glance at a single one.

Which turns the question into: can it still think, and just not write it down?

That's the deck we presented on the day — arrow keys to flip, f for fullscreen.

Why thinking costs so much

There's a detail about explicit chain-of-thought that's easy to skip past: its loop runs outside the network. The last layer produces a hidden state, that state gets decoded into a word, the word leaves the model as a token and comes back in through the input layer, and only then does the next step run. Every lap adds a line to the invoice — and the only thing that survives the lap is that one word.

Latent reasoning takes the obvious swing at this: pull the loop back inside. We reproduced CODI (arXiv:2502.21074, Shen et al., EMNLP 2025), where the final hidden state goes through a two-layer MLP and a LayerNorm straight back into the first hidden layer, loops six times between ⟨bot⟩ and ⟨eot⟩ without decoding anything, and only then starts writing the answer.

Training is self-distillation — one network doing two jobs. The teacher job writes the reasoning out step by step with a loss on every word. The student job replaces that reasoning with six continuous vectors and takes a loss on the answer only. Then, at the single token right before the answer, the two hidden states are aligned with an L1 term across all 36 layers, with no gradient flowing back through the teacher.

That last step is the clever bit. The student isn't learning the answer — it's learning what writing the reasoning out does to the inside of the network. Once the teacher has spelled every step out, its whole stack of hidden states at that position has been pushed around; the per-layer L1 is what makes the student reproduce that push. After training, the teacher path is thrown away and only the student runs.

Worth being straight about: six latent steps are still six forward passes. You don't save compute. You save tokens — decoding them, paying for them, and feeding them back in.

Which part we actually built

The model is the public checkpoint cds-jb/codi_qwen3-8b-answer_only: Qwen3-8B base, LoRA r=128, 0.35B of 8.2B parameters trained. We didn't retrain anything. We built the inference side.

That meant getting the latent loop running in FP16 on Apple Silicon MPS, wrapping it in a local FastAPI service, and routing the same question down three paths — direct (answer with no thinking), latent (six hidden steps), verbalized (write the full reasoning out) — so you can put the results side by side.

On top of that we added the part we actually wanted to play with: latent intervention. Take the hidden state at step 3, multiply it by 0.5 or zero it outright, let the model carry on, and watch the answer fall apart. This is one of the few things implicit reasoning gives you that explicit CoT doesn't — the thinking is a stack of tensors, so you can reach in and change it.

Wrapped around all of it is an MCP layer, so Cursor can hand a specific arithmetic sub-problem to this local model while the main model keeps writing the code.

250 questions later

250 questions sampled at random from the GSM8K test split, seed 11, greedy decoding, all three paths, all on the same Mac:

	Accuracy	Avg time	Total for 250
Direct (no thinking)	47.6%	0.59 s	2.5 min
Latent (6 hidden steps)	51.2%	1.23 s	5.1 min
Verbal (reasoning written out)	52.4%	5.20 s	21.7 min

The read is pretty clean: thinking silently buys 3.6 points over not thinking, gives up only 1.2 points against writing it out, and runs 4.2× faster, avoiding an average of 52 visible output tokens per question.

The other direction deserves saying too: latent is 2.1× slower than direct, because those six passes are real compute. So the actual trade is spend a bit more compute, skip most of the decoding. If your costs live in output tokens — which is how most APIs bill you — that's a good deal. If you're pinned on GPU time, run the numbers again.

This is also one deterministic sample, not a confidence interval, and no substitute for the benchmark on the model card. The checkpoint itself was trained mainly on grade-school math and CommonsenseQA, so it isn't a general chat model, and the verbalized path looks rough because training used an answer-only format.

One more thing worth stating up front: the hidden-state visualisations in the demo are measurements — L2 norms, cosine similarity between consecutive steps, a 64-bin activation fingerprint — not a translation of the model's private thoughts into English. Those two things are very easy to conflate.

Running it yourself

Everything is at github.com/Harvey-Yuan/innerthink. The README has the Apple Silicon setup script, curl examples for all three paths, the GSM8K eval harness and the Cursor MCP wiring. A 32 GB Mac will run it; the weights are about 34 GB.

If you want to push it further, the intervention seam is where the fun is. Sweep the zeroing across every step to find which ones actually carry the load — the model card says the middle ones matter most, which is testable. Or transplant hidden states from one question into another and see whether the answer follows or just collapses. We ran out of time for that one on the day, but the LatentHook seam is already there for it.

Thanks to Beta University / Beta Fund, EverMind (EverOS) and Snowflake for organising, sponsoring and hosting.