New candidates next Updated Nov 22

Claviger

A small model that reads one Discord message and decides whether an AI agent should reply, and which one.

Right now

Both tuning studies are done. The next step is the first wave of new open models, then the picks.

The latest comparison is complete, and its rules are written down: each model is judged only on its own job, data and hardware, with the same tuning for all. Next, test models that predict plain intent and let the trait rules do the rest.

Steps 23 of 28

Back to now Open every stage

Public numbers

Laya-421M-Claviger-v0 Hugging Face

Downloads, 30 days
28
Likes
0

as of

Activity Oct 22 to today, 30 updates

Today
  1. 01

    First comparison

    5/5Done

    Find out what decides best when an agent should reply.

    Updated

    1. First comparison

    Steps 5 of 5

    • done: Invented conversations, frozen by hash
    • done: Second labeller, working blind
    • done: Laya fine-tuned on the training split
    • done: Compared with a mention rule and Claude Haiku
    • done: Strict setup: the rule plus the model

    Done when Every option is scored on the same frozen test set.

    Updates

    1. First results: fine-tuning cut answered banter from 64.1% to 6.0% on the project's own test set.

    Settled

    • Label policyAn agent butting in is worse than an agent missing a message. When in doubt, no reply.
  2. 02

    Public v0

    3/3Done

    Put the model out with its limits stated.

    Updated

    2. Public v0

    Steps 3 of 3

    • done: Model card with results and limits
    • done: Cleaned upload
    • done: Published under Apache 2.0

    Done when Anyone can download it from Hugging Face.

    Updates

    1. v0 published on Hugging Face.

    Settled

    • Training code and dataNot public. The weights are.
  3. 03

    First v1 attempt

    5/5Done

    See whether more training beats v0.

    Updated

    3. First v1 attempt

    Steps 5 of 5

    • done: Full fine-tune, three seeds
    • done: Candidate settings frozen before the test
    • done: Public checks on Ubuntu IRC and Ambient Bench
    • done: Fresh test set, one look
    • done: Held, with v0 staying public

    Done when A v1 passes the eleven checks, or is held.

    Updates

    1. Next round planned: every candidate now runs one shared set of tests, including a new one for when a desktop companion should react to what's happening. A third outside model, a small general-purpose decision model released this week, joins the comparison.

    2. Bake-off result: under the rules set before it ran, neither newer model replaces Laya. A closer look found the one check the smaller model failed, on real chat, hinged on a shared setting rather than the model itself, and it beat Laya on the main test. Testing continues with new training data and a wider set of tests.

    3. v1 on a fresh test set: it tied v0 on its own roster and found more on new names, but missed two of eleven checks. Held.

    1 older
    1. Full fine-tune measured: probably a small gain, not proven. Seed noise was as large as the difference.

    Settled

    • How versions are judgedOn the project's own test sets plus two public ones, Ubuntu IRC and Social AI Ambient Bench. No training on either public set.
    • v0 staysThe first v1 missed two of eleven checks, both narrowly. It stays unpublished.
  4. 04

    Repeatable training

    4/4Done

    Make runs fair to compare, seed for seed.

    Updated

    4. Repeatable training

    Steps 4 of 4

    • done: Training runs shown live
    • done: Learning rate follows applied updates
    • done: Same seed, identical weights
    • done: Skipped updates nearly gone

    Done when The same seed gives the same weights.

    Updates

    1. A new three-seed baseline ran with no skipped updates.

    2. First live public training run. A fix then made the learning rate follow applied updates, and the same seed now gives identical weights.

  5. 05

    New agent names

    6/6Done

    Work with agents it wasn't trained on.

    • Decide how the next version gets judgedWaiting on Severian
    • Decide whether the second recipe carries forward to a v1Waiting on Severian

    1 to decide

    Updated

    5. New agent names

    To decide

    • How the next version gets judgedOn the newest test set, keeping banter under the limit also throws away most real asks that mention an agent. Either treat replies to an agent apart from mentions, or change the limit.

    Steps 6 of 6

    • done: Clean baseline, three seeds
    • done: Second fresh test set
    • done: First recipe for new names
    • done: Second recipe, gentler on open chat
    • done: How v1 gets judged, folded into the decision suite
    • done: Carrying the second recipe forward, folded into the decision suite

    Done when A recipe holds up on new names without answering more open chat.

    Updates

    1. The next round's first training runs went in: smaller encoder models trained beside Laya on the new data, with the rest of the round to follow.

    2. Third public dataset release: the ambient reactions set gained two configs for the companion's chat moments and personality traits, each point labelled by two models with blind tie-breaks, published and read back file for file.

    3. The next round can now try smaller encoder models alongside the current one.

    8 older
    1. Published two more public synthetic datasets on Hugging Face (CC BY 4.0). The triage set gains a training pool of 42,461 labelled chat messages and a fourth held-out test set of 1,450. A new set, ambient-reactions-synthetic, holds 21,556 desktop-companion decision points plus a 2,151-point test set. Every label has a writer's opinion, a blind second opinion from a different model and a blind tie-break, and every file was read back after upload.

    2. The next training set is built: about 30,000 labelled chat messages and 21,000 Familiar reaction points, each label agreed by two models or settled by a third.

    3. Claviger's synthetic training data and the routing worlds are now public on Hugging Face, CC BY 4.0, all names invented.

    4. Model bake-off: the first half of the gate results is frozen, so the held-back half can now judge the challengers fairly. One challenger is already too slow to pass.

    5. Testing a handful of newer small decision models against the current one, on its own job and on picking the right destination for a request. Results in about a day.

    6. Researched open decision models as a base: OneJev 0.8B and Kev-4B look stronger than Laya on paper, so a bake-off against Laya has started on our own GPU box.

    7. Second recipe: it removed most of the first one's false replies on open chat and kept part of its gain on new names. Passed 4 or 5 of eleven checks.

    8. First recipe for new names: no asks lost on its own roster and better recall on new names, but two of three seeds answered far more open chat by mistake. No model passed the checks.

  6. 06

    v1 release

    0/5In progress

    Publish a model better than v0, if the second recipe carries forward.

    1 to decide

    Updated

    6. v1 release

    To decide

    • Second recipeWhether it becomes the recipe for v1. It would need a third fresh test set.

    Steps 0 of 5

    • Third fresh test set
    • One seed picked before the look
    • The checks, one look
    • New model card
    • Severian's go on the upload

    Done when A v1 passes and Severian gives the go.

    Updates

    1. Stage 7 is running on the test machine: two new small gate models (tasksource-decider-nano and laya-typed-decisions) and two new small Familiar encoders (ettin-encoder-68m and decider-nano's encoder), each tuned and scored the same way as stage 6's. Familiar's chat cap now limits a predicted chat only by what the event allows.

    2. Gate tuning done: with the same tuning for both, the OneJev gate still catches more messages meant for it than the Laya gate (96% against 90%), and it varies less between runs. Neither is quiet enough in open chat yet.

    3. Dataset release five is public on Hugging Face: the key1 reaction labels now follow the written reaction rules exactly, changing 218 of 21,556 labels. Everything else is byte for byte release four.

    8 older
    1. Familiar's new 'intent' approach works: the model picks the plain reaction and the spec's rules add her personality. On the same 6,000 points it gets 83% of the personality-dependent cases right, against 43% when the model picks the reaction itself and 71% even with ten times the data. Overall 95% of reactions are right.

    2. Familiar's small models trained on all 58,000 practice points instead of a 6,000 sample pick the right reaction for her personality far more often: the best gets 71% of the personality-dependent cases right, up from 43% (the fixed rules get 35%), and 95% of all reactions.

    3. Rebuilt Familiar's key1 training labels so the spec's fact, lock and click rules apply in one consistent pass: 54 labels change against what training uses today, ready for the next retrain.

    4. New held-out routing test set frozen: ten invented organisations written, labelled and tie-broken by Claude models, 1,963 confirm requests at 8, 32 and 128 options, kept unscored for the router's confirm read.

    5. Two new held-out test sets built and frozen for the bake-off: a 1,593-item reply gate set and 159 twin pairs for the Familiar, written by Gemini models and labelled blind by Claude with Fable tie-breaks. Nothing has been scored on them yet.

    6. Public Familiar training set refreshed: 255 of 21,556 labels in ambient-reactions-synthetic key1 now follow the click-floor rule and the later late-night and plain-fact rules, matching what the models train on. Same rows and text; the card gains a Changes section.

    7. Read the same-data comparison for the decision models: trained on identical data, the larger encoders that fit in a browser show the most personality, but every model still trails the hand-written rules once those rules are given the right intent. Wrote down and froze the bake-off rules for all four model positions: each is judged only on its own job, data and hardware, and every model gets the same tuning. The next stage tests predicting the plain intent and letting the trait rules do the rest.

    8. The next round's training runs now show live on the public Bench as they train, and the earlier Claviger runs went up there too.

By the numbers

Parameters
421M
F1, own test set
71.4
Right agent
69.2%
Banter answered
6.0%

Version 0

Public training runs
48
Last work
Nov 22

Counted from the code, as of

Still open

  • mediumNew agent namesv0 is tied to the agents it was trained on. Other names run, but expect worse results.
  • mediumReal chatOn Ubuntu IRC, no version yet matches a plain name rule on both asks found and false replies.
  • lowSeed noiseSeeds of one recipe differ as much as recipes do, so three seeds can't settle a close comparison.
  • lowCard wordingv0's card says no public benchmark exists. Public sets for a single assistant do exist. The next card corrects it.

Parked

  • Public benchmarkA public test of when an agent should reply in a shared channel, using invented agents and communities. A free version is written up, not started.

Every update

30, newest first
  1. Stage 7 is running on the test machine: two new small gate models (tasksource-decider-nano and laya-typed-decisions) and two new small Familiar encoders (ettin-encoder-68m and decider-nano's encoder), each tuned and scored the same way as stage 6's. Familiar's chat cap now limits a predicted chat only by what the event allows.

    6 v1 release
  2. Gate tuning done: with the same tuning for both, the OneJev gate still catches more messages meant for it than the Laya gate (96% against 90%), and it varies less between runs. Neither is quiet enough in open chat yet.

    6 v1 release
  3. Dataset release five is public on Hugging Face: the key1 reaction labels now follow the written reaction rules exactly, changing 218 of 21,556 labels. Everything else is byte for byte release four.

    6 v1 release
  4. Familiar's new 'intent' approach works: the model picks the plain reaction and the spec's rules add her personality. On the same 6,000 points it gets 83% of the personality-dependent cases right, against 43% when the model picks the reaction itself and 71% even with ten times the data. Overall 95% of reactions are right.

    6 v1 release
  5. Familiar's small models trained on all 58,000 practice points instead of a 6,000 sample pick the right reaction for her personality far more often: the best gets 71% of the personality-dependent cases right, up from 43% (the fixed rules get 35%), and 95% of all reactions.

    6 v1 release
  6. Rebuilt Familiar's key1 training labels so the spec's fact, lock and click rules apply in one consistent pass: 54 labels change against what training uses today, ready for the next retrain.

    6 v1 release
  7. New held-out routing test set frozen: ten invented organisations written, labelled and tie-broken by Claude models, 1,963 confirm requests at 8, 32 and 128 options, kept unscored for the router's confirm read.

    6 v1 release
  8. Two new held-out test sets built and frozen for the bake-off: a 1,593-item reply gate set and 159 twin pairs for the Familiar, written by Gemini models and labelled blind by Claude with Fable tie-breaks. Nothing has been scored on them yet.

    6 v1 release
  9. Public Familiar training set refreshed: 255 of 21,556 labels in ambient-reactions-synthetic key1 now follow the click-floor rule and the later late-night and plain-fact rules, matching what the models train on. Same rows and text; the card gains a Changes section.

    6 v1 release
  10. Read the same-data comparison for the decision models: trained on identical data, the larger encoders that fit in a browser show the most personality, but every model still trails the hand-written rules once those rules are given the right intent. Wrote down and froze the bake-off rules for all four model positions: each is judged only on its own job, data and hardware, and every model gets the same tuning. The next stage tests predicting the plain intent and letting the trait rules do the rest.

    6 v1 release
  11. The next round's training runs now show live on the public Bench as they train, and the earlier Claviger runs went up there too.

    6 v1 release
  12. The next round's first training runs went in: smaller encoder models trained beside Laya on the new data, with the rest of the round to follow.

    5 New agent names
  13. Third public dataset release: the ambient reactions set gained two configs for the companion's chat moments and personality traits, each point labelled by two models with blind tie-breaks, published and read back file for file.

    5 New agent names
  14. The next round can now try smaller encoder models alongside the current one.

    5 New agent names
  15. Published two more public synthetic datasets on Hugging Face (CC BY 4.0). The triage set gains a training pool of 42,461 labelled chat messages and a fourth held-out test set of 1,450. A new set, ambient-reactions-synthetic, holds 21,556 desktop-companion decision points plus a 2,151-point test set. Every label has a writer's opinion, a blind second opinion from a different model and a blind tie-break, and every file was read back after upload.

    5 New agent names
  16. The next training set is built: about 30,000 labelled chat messages and 21,000 Familiar reaction points, each label agreed by two models or settled by a third.

    5 New agent names
  17. Claviger's synthetic training data and the routing worlds are now public on Hugging Face, CC BY 4.0, all names invented.

    5 New agent names
  18. Next round planned: every candidate now runs one shared set of tests, including a new one for when a desktop companion should react to what's happening. A third outside model, a small general-purpose decision model released this week, joins the comparison.

    3 First v1 attempt
  19. Bake-off result: under the rules set before it ran, neither newer model replaces Laya. A closer look found the one check the smaller model failed, on real chat, hinged on a shared setting rather than the model itself, and it beat Laya on the main test. Testing continues with new training data and a wider set of tests.

    3 First v1 attempt
  20. Model bake-off: the first half of the gate results is frozen, so the held-back half can now judge the challengers fairly. One challenger is already too slow to pass.

    5 New agent names
  21. Testing a handful of newer small decision models against the current one, on its own job and on picking the right destination for a request. Results in about a day.

    5 New agent names
  22. Researched open decision models as a base: OneJev 0.8B and Kev-4B look stronger than Laya on paper, so a bake-off against Laya has started on our own GPU box.

    5 New agent names
  23. Second recipe: it removed most of the first one's false replies on open chat and kept part of its gain on new names. Passed 4 or 5 of eleven checks.

    5 New agent names
  24. First recipe for new names: no asks lost on its own roster and better recall on new names, but two of three seeds answered far more open chat by mistake. No model passed the checks.

    5 New agent names
  25. A new three-seed baseline ran with no skipped updates.

    4 Repeatable training
  26. First live public training run. A fix then made the learning rate follow applied updates, and the same seed now gives identical weights.

    4 Repeatable training
  27. v1 on a fresh test set: it tied v0 on its own roster and found more on new names, but missed two of eleven checks. Held.

    3 First v1 attempt
  28. Full fine-tune measured: probably a small gain, not proven. Seed noise was as large as the difference.

    3 First v1 attempt
  29. v0 published on Hugging Face.

    2 Public v0
  30. 10 days quiet
  31. First results: fine-tuning cut answered banter from 64.1% to 6.0% on the project's own test set.

    1 First comparison

Visits are counted with Google Analytics. Your choice ยท How Google uses this