Synth-2 reads a search results page, answers only from it, and underlines the exact span it isn't sure about.
A mock-up, not a running model. Synth-2 is still in pretraining, so the answers and every u value above are hand-written to show the output format — nothing here was generated. What is real: u is predicted error, not token probability. A model pronounces an invented number as fluently as a true one, so logits only measure certainty about phrasing; the epistemic head is trained to predict the mistake instead, which is why it cannot be bolted onto a finished model afterwards.
A model that reads the results page, answers only from it, says "the sources don't say" when they don't — and shows exactly where it is unsure.
All three behaviours run off one signal: a scalar u produced at every layer, for every token, from the first step of pretraining.
Large assistants finish sentences — RLHF rewards willingness to help, not "that isn't in the sources." Synth-2 trains on the opposite: SQuAD 2.0, plus answerable questions deliberately broken by deleting the paragraph that held the answer.
Every statement carries a citation. No citation, no statement.
Not search-engine optimised — search-reading. The model is handed live results and answers from what's in front of it, so its 330M active parameters go into handling text rather than storing facts.
Knowledge lives in the sources. The model's job is to not exceed them.
Everyone else reports confidence for a whole answer — one number, usually none at all. Synth-2 carries one d_model → 1 projection on each of its 24 layers — 768 weights apiece, about 18K in total — so u arrives per token and a reader checks one clause instead of re-verifying a paragraph.
The same signal routes compute: how deep to go, whether to wake a second expert, whether to call search.
Routed experts hold 1104M of the 1.23B and contribute 255M of the active count; the MTP module adds 48M that costs nothing at inference. The whole-to-active ratio is 3.7×. Embeddings are tied, and the vocabulary is a multiple of 128 because the TPU wants it that way.
330M active parameters against models with hundreds of times more. Long synthesis, rare knowledge, fine language — not this weight class, and no architecture reverses that.
Google has an index of the whole web. Synth-2 works with whatever results it is handed.
The budget is thirteen billion tokens and the run is partway through it. The log below is the real state, not a launch countdown.
The turn markers are tokens in the vocabulary, not string formatting. Reasoning is separated, never shown, and never counts toward attribution.
<|user|>When was Canberra founded? <|result|>[1] Canberra is the capital of Australia... [2] The city was laid out in 1913... <|assistant|><|think|>[2] states the date outright. [1] has no date.</|think|> Canberra was laid out in 1913 [2].
| Stage | Target for u | Where the label comes from |
|---|---|---|
| B, C | Normalised entropy of the model's own distribution, detached | The model itself, free, no annotation |
| E | 1 − P(answer is correct) | Auto-checking generations against QA references |
| E | "Search needed" flag | Same signal: errs often ⇒ call the tool |
Not one label is drawn by hand. The head on early layers predicts the final entropy — by layer 5 the model already knows the token will be hard, which is what makes u a control signal rather than a report. Its weight in the loss, λ = 0.05, is still marked a hypothesis in the spec: push it higher and the head starts pulling the representations toward itself and spoiling the language.
Every number below is read from the run log stored with this page. Nothing is hard-coded: after each Kaggle session the new measurement is appended and this panel moves.
Every row is one held-out measurement taken at the end of a Kaggle session. The log is a JSON file versioned next to the training code, so a point appears here the moment the session that produced it is pushed — and the chart, the progress bars and the header all recompute from it.
Tokenizer, corpus, checkpoints, measurement harness.
done~50M model, configuration chosen by measurement.
done13B tokens, curriculum 2k → 4k → 8k.
running nowQuality pass, and Russian arrives late by design.
queuedStretch 8k → 32k, about 2% of the budget.
queuedAnswer only from sources; refuse without them.
ships claims 1 & 2Call search, read results, re-query when they're poor.
queuedu tied to attribution, with no hand-drawn labels.
ships claim 3ORPO.
queuedThe last half hour: it is Synth-2, from Sane Labs.
queuedOwn CUDA engine, MTP speculation, int8.
queuedThe headline number is not accuracy. It is the share of confident errors — how often the system answered firmly and was wrong — because a confident error is the one a reader cannot catch.
| Measurement | What it shows | Threshold |
|---|---|---|
| Refusal on unanswerable | SQuAD 2.0, questions with no answer in the passage | ≥ 75% |
| F1 on answerable | Whether caution wrecked the quality | ≥ 70 |
| AUROC(u, error) | Whether u predicts the model's own mistakes | ≥ 0.75 |
| Attribution accuracy | Statements actually supported by the source cited | ≥ 80% |
| Risk–coverage | Accuracy at 50 / 70 / 90% coverage | whole curve |
None of these have been measured — stage E is where they get taken. The thresholds sit deliberately below BERT-large's ~81 F1 and ~78% no-answer on SQuAD 2.0: the same weight class, but Synth-2 does the harder job of generating with citations rather than selecting a span. Alongside them runs a hand-scored benchmark of 100 questions the results page cannot answer, counting one thing — how many times each system invented an answer instead of refusing.
Not ambition — storage and quota. The compute is Kaggle's free TPU allowance; everything that has ended a session so far has been the disk under it.
One checkpoint is 9.3 GB, and the private tier fills after seven. Sessions have been spent wiping repositories instead of training.
Training runs in 9-hour Kaggle sessions on a weekly quota. Paid TPU time is what turns a five-week pretrain into a shorter one.
The refusal benchmark is scored by a human, by eye, on 100 questions — the one number that would make the claim checkable.
If a number on this page looks wrong, say so — the measurements are all reproducible and a correction is worth more than a compliment. Collaboration on grounded QA and calibration is welcome too.
Stage E is where the numbers get taken. Until then, the honest thing a landing page can show is the run itself.