Live, refinements paused Updated Nov 21

Palaemon

Tools that stream model tuning runs to a live public bench.

Right now

Bench live, 41 more runs on it

41 earlier training runs went public on the Bench in batches. Line smoothing is built but not live. The Bench now sits at citadel.severian.dev/bench, under the new front page.

Steps 19 of 31

Back to now Open every stage

Activity Nov 1 to today, 11 updates

Today
  1. 01

    Groundwork

    5/5Done

    Work out what to build, and give tuning its own place on the hub.

    Updated

    1. Groundwork

    Steps 5 of 5

    • done: Survey of existing run viewers and node tools
    • done: Proposal for nodes and a live stream
    • done: A separate space for tuning on the hub
    • done: One job queue
    • done: Three-seed test run

    Done when Three seeds run in order.

    Updates

    1. The first test passed: three seeds ran in order.

    2. Proposed after Claviger's first tuning runs on the hub. One seed misled an early report, so configs now run in threes.

  2. 02

    First stream

    5/5Done

    Get a real tuning run onto the public site.

    Updated

    2. First stream

    Steps 5 of 5

    • done: Wrapper for any training command
    • done: Runs reach the public site
    • done: Live view takes run data
    • done: A way to take a run back down
    • done: Real item counts on each run

    Done when A real run shows live on the bench and can be taken down.

    Updates

    1. The first run reached the public site and was taken down: its card showed more training items than it used. Runs now report their real count.

    2. Wrapper built: it reads a training command's output and writes the numbers to a file, with no change to the code.

  3. 03

    Instrument bench

    6/6Done

    Make the bench worth reading, live or quiet.

    Updated

    3. Instrument bench

    Recordings page with 14 runs on one loss chart and a list of runs by date

    Done when Bench, Recordings and run pages are live in both themes.

    Steps 6 of 6

    • done: Three designs tried, Instrument picked
    • done: Bench, Recordings and run pages
    • done: Phases, end time, learning rate, speed and peak memory
    • done: Power, temperature and per-GPU readings
    • done: Light and dark, following the device
    • done: One screen on desktop

    Updates

    1. The bench became one screen on desktop. The last three runs so far finished the same day.

    2. A real Claviger run proved the whole path, with GPU power, temperature and memory on the public page in both themes.

    3. The Instrument design went live, with Bench, Recordings and run pages.

    Settled

    • What the site isA general public view of what's on the bench.
    • LookInstrument: a full-screen panel of readouts in a film-interface style.
    • ThemeFollows each device, light or dark.
    1 more
    • Main readoutSegment digits, not plain type.
  4. 04

    Bench refinements

    1/5In progress

    Tidy the bench and rework the run page.

    • Pick the new visual direction for the front pageWaiting on Severian

    1 to decide

    Updated

    4. Bench refinements

    The live bench, quiet between runs, showing the last Claviger run

    Done when Smoothing and the new run page are live.

    To decide

    • Very small lossesShow 0.0000 as it is, or as a less-than mark.

    Steps 1 of 5

    • done: Smoothing on by default, built
    • Put smoothing live
    • Remove the footer
    • Bench strip on the new front page
    • Run page reworked as a dashboard

    Updates

    1. The Bench, where these runs are shown, now lives at citadel.severian.dev/bench; each run's own page keeps its address. A finished run with no eval now says "no eval" rather than "no eval yet", and a run's elapsed times no longer print over each other: when a stretch is too short to name in the bar, the times are listed under it.

    2. Past training runs can now be published to the Bench in batches. The 41 earlier runs of Claviger and the Familiar deciders went public this way. The training machine's job runner also gained a watchdog: a training job that stops printing for its limit is stopped and recorded as failed, so one stuck job can no longer hold the queue for hours.

    3. New training runs now go up on the public Bench by default and show live as they train, with GPU readings. The first was a run from the next Claviger round, on Nov 18.

    1 older
    1. Smoothing built: on by default, and every line but eval accuracy follows it. Not live yet.

  5. 05

    Drop-on node

    2/5Not started

    Stream runs from any machine used for tuning, not only the hub.

    5. Drop-on node

    Steps 2 of 5

    • done: GPU readings with NVIDIA and AMD backends
    • done: Hardware label on each run
    • Node that installs as a package or a container image
    • Test on one rented machine
    • Runs from a node reach the bench

    Done when A rented GPU machine streams to the bench after a short install.

    Settled

    • HardwareNodes will run on rented machines too, so readings can't assume one kind of GPU.
  6. 06

    Agent access

    0/2Later

    Let an agent run a tuning job on a node.

    6. Agent access

    Steps 0 of 2

    • Run a tune from an agent
    • Recipes for common runs

    Done when An agent runs a tune on a rented machine.

  7. 07

    Open release

    0/3Later

    Publish the training wrapper as a small open tool.

    • OK for the public repo and the first releaseWaiting on Severian

    1 to decide

    7. Open release

    To decide

    • NamesPalaemon is a working name, and the wrapper's name isn't final either.

    Steps 0 of 3

    • Final name
    • Public repo
    • First signed release

    Done when The wrapper has a public repo and a signed release.

    Settled

    • Open partOnly the training wrapper goes public, under Apache-2.0. The rest stays in citadel.

By the numbers

Runs on the bench
74
Projects
13
Configs
6
Best eval accuracy
0.8385
Last run
Nov 20
Public delay
2 min

Still open

  • lowOther hardwareAMD's own sensor tools and multi-GPU runs are untested on real machines.
  • lowBrowsersSafari and real phones are untested.

Parked

  • ExtrasAgent notes beside curves, approved synthetic samples, and cost from provider APIs. Each only if there's demand.

Every update

11, newest first
  1. The Bench, where these runs are shown, now lives at citadel.severian.dev/bench; each run's own page keeps its address. A finished run with no eval now says "no eval" rather than "no eval yet", and a run's elapsed times no longer print over each other: when a stretch is too short to name in the bar, the times are listed under it.

    4 Bench refinements
  2. Past training runs can now be published to the Bench in batches. The 41 earlier runs of Claviger and the Familiar deciders went public this way. The training machine's job runner also gained a watchdog: a training job that stops printing for its limit is stopped and recorded as failed, so one stuck job can no longer hold the queue for hours.

    4 Bench refinements
  3. New training runs now go up on the public Bench by default and show live as they train, with GPU readings. The first was a run from the next Claviger round, on Nov 18.

    4 Bench refinements
  4. 14 days quiet
  5. Smoothing built: on by default, and every line but eval accuracy follows it. Not live yet.

    4 Bench refinements
  6. The bench became one screen on desktop. The last three runs so far finished the same day.

    3 Instrument bench
  7. A real Claviger run proved the whole path, with GPU power, temperature and memory on the public page in both themes.

    3 Instrument bench
  8. The Instrument design went live, with Bench, Recordings and run pages.

    3 Instrument bench
  9. The first run reached the public site and was taken down: its card showed more training items than it used. Runs now report their real count.

    2 First stream
  10. Wrapper built: it reads a training command's output and writes the numbers to a file, with no change to the code.

    2 First stream
  11. The first test passed: three seeds ran in order.

    1 Groundwork
  12. Proposed after Claviger's first tuning runs on the hub. One seed misled an early report, so configs now run in threes.

    1 Groundwork

The live bench, quiet between runs, showing the last Claviger run

Recordings page with 14 runs on one loss chart and a list of runs by date

Visits are counted with Google Analytics. Your choice ยท How Google uses this