Activity
01
First comparison
5/5DoneFind out what decides best when an agent should reply.
Updated
1. First comparison
Steps 5 of 5
- done: Invented conversations, frozen by hash
- done: Second labeller, working blind
- done: Laya fine-tuned on the training split
- done: Compared with a mention rule and Claude Haiku
- done: Strict setup: the rule plus the model
Done when Every option is scored on the same frozen test set.
Updates
First results: fine-tuning cut answered banter from 64.1% to 6.0% on the project's own test set.
Settled
- Label policyAn agent butting in is worse than an agent missing a message. When in doubt, no reply.
02
Public v0
3/3DonePut the model out with its limits stated.
Updated
2. Public v0
Steps 3 of 3
- done: Model card with results and limits
- done: Cleaned upload
- done: Published under Apache 2.0
Done when Anyone can download it from Hugging Face.
Updates
v0 published on Hugging Face.
Settled
- Training code and dataNot public. The weights are.
03
First v1 attempt
5/5DoneSee whether more training beats v0.
Updated
3. First v1 attempt
Steps 5 of 5
- done: Full fine-tune, three seeds
- done: Candidate settings frozen before the test
- done: Public checks on Ubuntu IRC and Ambient Bench
- done: Fresh test set, one look
- done: Held, with v0 staying public
Done when A v1 passes the eleven checks, or is held.
Updates
Next round planned: every candidate now runs one shared set of tests, including a new one for when a desktop companion should react to what's happening. A third outside model, a small general-purpose decision model released this week, joins the comparison.
Bake-off result: under the rules set before it ran, neither newer model replaces Laya. A closer look found the one check the smaller model failed, on real chat, hinged on a shared setting rather than the model itself, and it beat Laya on the main test. Testing continues with new training data and a wider set of tests.
v1 on a fresh test set: it tied v0 on its own roster and found more on new names, but missed two of eleven checks. Held.
1 older
Full fine-tune measured: probably a small gain, not proven. Seed noise was as large as the difference.
Settled
- How versions are judgedOn the project's own test sets plus two public ones, Ubuntu IRC and Social AI Ambient Bench. No training on either public set.
- v0 staysThe first v1 missed two of eleven checks, both narrowly. It stays unpublished.
04
Repeatable training
4/4DoneMake runs fair to compare, seed for seed.
Updated
4. Repeatable training
Steps 4 of 4
- done: Training runs shown live
- done: Learning rate follows applied updates
- done: Same seed, identical weights
- done: Skipped updates nearly gone
Done when The same seed gives the same weights.
Updates
A new three-seed baseline ran with no skipped updates.
First live public training run. A fix then made the learning rate follow applied updates, and the same seed now gives identical weights.
05
New agent names
6/6DoneWork with agents it wasn't trained on.
- Decide how the next version gets judgedWaiting on Severian
- Decide whether the second recipe carries forward to a v1Waiting on Severian
1 to decide
Updated
5. New agent names
To decide
- How the next version gets judgedOn the newest test set, keeping banter under the limit also throws away most real asks that mention an agent. Either treat replies to an agent apart from mentions, or change the limit.
Steps 6 of 6
- done: Clean baseline, three seeds
- done: Second fresh test set
- done: First recipe for new names
- done: Second recipe, gentler on open chat
- done: How v1 gets judged, folded into the decision suite
- done: Carrying the second recipe forward, folded into the decision suite
Done when A recipe holds up on new names without answering more open chat.
Updates
The next round's first training runs went in: smaller encoder models trained beside Laya on the new data, with the rest of the round to follow.
Third public dataset release: the ambient reactions set gained two configs for the companion's chat moments and personality traits, each point labelled by two models with blind tie-breaks, published and read back file for file.
The next round can now try smaller encoder models alongside the current one.
8 older
Published two more public synthetic datasets on Hugging Face (CC BY 4.0). The triage set gains a training pool of 42,461 labelled chat messages and a fourth held-out test set of 1,450. A new set, ambient-reactions-synthetic, holds 21,556 desktop-companion decision points plus a 2,151-point test set. Every label has a writer's opinion, a blind second opinion from a different model and a blind tie-break, and every file was read back after upload.
The next training set is built: about 30,000 labelled chat messages and 21,000 Familiar reaction points, each label agreed by two models or settled by a third.
Claviger's synthetic training data and the routing worlds are now public on Hugging Face, CC BY 4.0, all names invented.
Model bake-off: the first half of the gate results is frozen, so the held-back half can now judge the challengers fairly. One challenger is already too slow to pass.
Testing a handful of newer small decision models against the current one, on its own job and on picking the right destination for a request. Results in about a day.
Researched open decision models as a base: OneJev 0.8B and Kev-4B look stronger than Laya on paper, so a bake-off against Laya has started on our own GPU box.
Second recipe: it removed most of the first one's false replies on open chat and kept part of its gain on new names. Passed 4 or 5 of eleven checks.
First recipe for new names: no asks lost on its own roster and better recall on new names, but two of three seeds answered far more open chat by mistake. No model passed the checks.
06
v1 release
0/5In progressPublish a model better than v0, if the second recipe carries forward.
1 to decide
Updated
6. v1 release
To decide
- Second recipeWhether it becomes the recipe for v1. It would need a third fresh test set.
Steps 0 of 5
- Third fresh test set
- One seed picked before the look
- The checks, one look
- New model card
- Severian's go on the upload
Done when A v1 passes and Severian gives the go.
Updates
Stage 7 is running on the test machine: two new small gate models (tasksource-decider-nano and laya-typed-decisions) and two new small Familiar encoders (ettin-encoder-68m and decider-nano's encoder), each tuned and scored the same way as stage 6's. Familiar's chat cap now limits a predicted chat only by what the event allows.
Gate tuning done: with the same tuning for both, the OneJev gate still catches more messages meant for it than the Laya gate (96% against 90%), and it varies less between runs. Neither is quiet enough in open chat yet.
Dataset release five is public on Hugging Face: the key1 reaction labels now follow the written reaction rules exactly, changing 218 of 21,556 labels. Everything else is byte for byte release four.
8 older
Familiar's new 'intent' approach works: the model picks the plain reaction and the spec's rules add her personality. On the same 6,000 points it gets 83% of the personality-dependent cases right, against 43% when the model picks the reaction itself and 71% even with ten times the data. Overall 95% of reactions are right.
Familiar's small models trained on all 58,000 practice points instead of a 6,000 sample pick the right reaction for her personality far more often: the best gets 71% of the personality-dependent cases right, up from 43% (the fixed rules get 35%), and 95% of all reactions.
Rebuilt Familiar's key1 training labels so the spec's fact, lock and click rules apply in one consistent pass: 54 labels change against what training uses today, ready for the next retrain.
New held-out routing test set frozen: ten invented organisations written, labelled and tie-broken by Claude models, 1,963 confirm requests at 8, 32 and 128 options, kept unscored for the router's confirm read.
Two new held-out test sets built and frozen for the bake-off: a 1,593-item reply gate set and 159 twin pairs for the Familiar, written by Gemini models and labelled blind by Claude with Fable tie-breaks. Nothing has been scored on them yet.
Public Familiar training set refreshed: 255 of 21,556 labels in ambient-reactions-synthetic key1 now follow the click-floor rule and the later late-night and plain-fact rules, matching what the models train on. Same rows and text; the card gains a Changes section.
Read the same-data comparison for the decision models: trained on identical data, the larger encoders that fit in a browser show the most personality, but every model still trails the hand-written rules once those rules are given the right intent. Wrote down and froze the bake-off rules for all four model positions: each is judged only on its own job, data and hardware, and every model gets the same tuning. The next stage tests predicting the plain intent and letting the trait rules do the rest.
The next round's training runs now show live on the public Bench as they train, and the earlier Claviger runs went up there too.
By the numbers
- Parameters
- 421M
- F1, own test set
- 71.4
- Right agent
- 69.2%
- Banter answered
- 6.0%
Version 0
- Public training runs
- 48
- Last work
- Nov 22
Counted from the code, as of
Still open
- mediumNew agent namesv0 is tied to the agents it was trained on. Other names run, but expect worse results.
- mediumReal chatOn Ubuntu IRC, no version yet matches a plain name rule on both asks found and false replies.
- lowSeed noiseSeeds of one recipe differ as much as recipes do, so three seeds can't settle a close comparison.
- lowCard wordingv0's card says no public benchmark exists. Public sets for a single assistant do exist. The next card corrects it.
Parked
- Public benchmarkA public test of when an agent should reply in a shared channel, using invented agents and communities. A free version is written up, not started.
Every update
Stage 7 is running on the test machine: two new small gate models (tasksource-decider-nano and laya-typed-decisions) and two new small Familiar encoders (ettin-encoder-68m and decider-nano's encoder), each tuned and scored the same way as stage 6's. Familiar's chat cap now limits a predicted chat only by what the event allows.
6 v1 releaseGate tuning done: with the same tuning for both, the OneJev gate still catches more messages meant for it than the Laya gate (96% against 90%), and it varies less between runs. Neither is quiet enough in open chat yet.
6 v1 releaseDataset release five is public on Hugging Face: the key1 reaction labels now follow the written reaction rules exactly, changing 218 of 21,556 labels. Everything else is byte for byte release four.
6 v1 releaseFamiliar's new 'intent' approach works: the model picks the plain reaction and the spec's rules add her personality. On the same 6,000 points it gets 83% of the personality-dependent cases right, against 43% when the model picks the reaction itself and 71% even with ten times the data. Overall 95% of reactions are right.
6 v1 releaseFamiliar's small models trained on all 58,000 practice points instead of a 6,000 sample pick the right reaction for her personality far more often: the best gets 71% of the personality-dependent cases right, up from 43% (the fixed rules get 35%), and 95% of all reactions.
6 v1 releaseRebuilt Familiar's key1 training labels so the spec's fact, lock and click rules apply in one consistent pass: 54 labels change against what training uses today, ready for the next retrain.
6 v1 releaseNew held-out routing test set frozen: ten invented organisations written, labelled and tie-broken by Claude models, 1,963 confirm requests at 8, 32 and 128 options, kept unscored for the router's confirm read.
6 v1 releaseTwo new held-out test sets built and frozen for the bake-off: a 1,593-item reply gate set and 159 twin pairs for the Familiar, written by Gemini models and labelled blind by Claude with Fable tie-breaks. Nothing has been scored on them yet.
6 v1 releasePublic Familiar training set refreshed: 255 of 21,556 labels in ambient-reactions-synthetic key1 now follow the click-floor rule and the later late-night and plain-fact rules, matching what the models train on. Same rows and text; the card gains a Changes section.
6 v1 releaseRead the same-data comparison for the decision models: trained on identical data, the larger encoders that fit in a browser show the most personality, but every model still trails the hand-written rules once those rules are given the right intent. Wrote down and froze the bake-off rules for all four model positions: each is judged only on its own job, data and hardware, and every model gets the same tuning. The next stage tests predicting the plain intent and letting the trait rules do the rest.
6 v1 releaseThe next round's training runs now show live on the public Bench as they train, and the earlier Claviger runs went up there too.
6 v1 releaseThe next round's first training runs went in: smaller encoder models trained beside Laya on the new data, with the rest of the round to follow.
5 New agent namesThird public dataset release: the ambient reactions set gained two configs for the companion's chat moments and personality traits, each point labelled by two models with blind tie-breaks, published and read back file for file.
5 New agent namesThe next round can now try smaller encoder models alongside the current one.
5 New agent namesPublished two more public synthetic datasets on Hugging Face (CC BY 4.0). The triage set gains a training pool of 42,461 labelled chat messages and a fourth held-out test set of 1,450. A new set, ambient-reactions-synthetic, holds 21,556 desktop-companion decision points plus a 2,151-point test set. Every label has a writer's opinion, a blind second opinion from a different model and a blind tie-break, and every file was read back after upload.
5 New agent namesThe next training set is built: about 30,000 labelled chat messages and 21,000 Familiar reaction points, each label agreed by two models or settled by a third.
5 New agent namesClaviger's synthetic training data and the routing worlds are now public on Hugging Face, CC BY 4.0, all names invented.
5 New agent namesNext round planned: every candidate now runs one shared set of tests, including a new one for when a desktop companion should react to what's happening. A third outside model, a small general-purpose decision model released this week, joins the comparison.
3 First v1 attemptBake-off result: under the rules set before it ran, neither newer model replaces Laya. A closer look found the one check the smaller model failed, on real chat, hinged on a shared setting rather than the model itself, and it beat Laya on the main test. Testing continues with new training data and a wider set of tests.
3 First v1 attemptModel bake-off: the first half of the gate results is frozen, so the held-back half can now judge the challengers fairly. One challenger is already too slow to pass.
5 New agent namesTesting a handful of newer small decision models against the current one, on its own job and on picking the right destination for a request. Results in about a day.
5 New agent namesResearched open decision models as a base: OneJev 0.8B and Kev-4B look stronger than Laya on paper, so a bake-off against Laya has started on our own GPU box.
5 New agent namesSecond recipe: it removed most of the first one's false replies on open chat and kept part of its gain on new names. Passed 4 or 5 of eleven checks.
5 New agent namesFirst recipe for new names: no asks lost on its own roster and better recall on new names, but two of three seeds answered far more open chat by mistake. No model passed the checks.
5 New agent namesA new three-seed baseline ran with no skipped updates.
4 Repeatable trainingFirst live public training run. A fix then made the learning rate follow applied updates, and the same seed now gives identical weights.
4 Repeatable trainingv1 on a fresh test set: it tied v0 on its own roster and found more on new names, but missed two of eleven checks. Held.
3 First v1 attemptFull fine-tune measured: probably a small gain, not proven. Seed noise was as large as the difference.
3 First v1 attemptv0 published on Hugging Face.
2 Public v0- 10 days quiet
First results: fine-tuning cut answered banter from 64.1% to 6.0% on the project's own test set.
1 First comparison