CITADEL
StandbyNothing is live right now.
Eval accuracylast measured Sat 21 Nov, 23:42 UTC
0%25%50%75%100%0%25%50%75%100%0%25%50%75%100%

Claviger

finishedRadeon 8060S

A small model that reads one Discord message and decides whether an agent should reply, and which one. Fine-tuned from Laya on invented chat.

started
Sat 21 Nov, 23:29 UTC
ran for
12:26
config
learning rate 0.00005, batch 8, seed 0

run20/22step336/339epoch2/3config10/11seed0lr0.00005batch8modellaya 421Mrev7b928d8items1800+130

Loss train 2.4943 ▲ +1.7929 eval 0.4691 lowest -0.4428 at step 143 run before, run 19 ▲ +2.5880 step 336 Laya's objective: soft cross-entropy plus a policy-gradient term, so it can go below zero.
Timebase
Range
Smoothing
Reference
66 readings, 5 min/div5 min/div2 min/div
-20246800:0005:0010:00phase starting, 0.0 s to 35.2 sphase training, 35.2 s to 12:25phase eval, 12:25 to 12:26epoch 1, 04:10epoch 2, 07:55epoch 3, 11:42trainingepoch 1epoch 2E3+1.7929since first0.7014Lowest last train loss across this project's 22 runs, run 19lowest last -0.0937, run 19339 steps2.4943-20246800:0005:0010:00phase starting, 0.0 s to 35.2 sphase training, 35.2 s to 12:25phase eval, 12:25 to 12:26epoch 1, 04:10epoch 2, 07:55epoch 3, 11:42trainingepoch 1epoch 2E30.7014Lowest last train loss across this project's 22 runs, run 19lowest last -0.0937, run 19eval 0.4691339 steps2.4943▲ +1.7929-20246800:0002:0004:0006:0008:0010:0012:00phase starting, 0.0 s to 35.2 sphase training, 35.2 s to 12:25phase eval, 12:25 to 12:26epoch 1, 04:10epoch 2, 07:55epoch 3, 11:42trainingepoch 1epoch 2epoch 30.7014Lowest last train loss across this project's 22 runs, run 19lowest last -0.0937, run 19eval 0.4691339 stepsrun 192.4943▲ +1.7929

Eval measures

Eval losslower is better
0.4691
02 →
ECEcalibration error, lower is better
0.0877
01
Brierlower is better
0.2341
02 →

Elapsed

226.2 s

epoch interval

336 of 339 steps, 2 of 3 epochs, 99.1%

1.99 s per step, 30.1 steps a minute

Against the other runs

8 of 22

by last eval accuracy

best so far 0.9231, run 19

▼ −0.0693 against run 19

Train minus eval

+2.0252

step 336

train 2.4943, eval 0.4691

Set-up

1800 train, 130 eval

synthetic, learning rate 0.00005, batch size 8, grad accum 2, epochs 3, loss rlcd, seed 0

Samples per second

2.36

▼ −5.24 since first

Learning rate

1.00e-1

▲ +1.00e-1 since first

GPU memory, this run

4.9 GB

On one axis

phase starting, 0.0 s to 35.2 sphase training, 35.2 s to 12:25phase eval, 12:25 to 12:26E1E2E3traininglowest last -0.0937Train loss2.4943▲ +1.79297.50-2.500.7014Samples per second2.36▼ −5.241007.60Learning rate1.00e-1▲ +1.00e-11.0e-10.0e+05.00e-5Power, whole chip, CPU and GPU118 W▲ +38 W140 W60 W80 WHottest, one board zone91 °C▲ +1 °C91 °C87 °C90 °C00:0002:0004:0006:0008:0010:0012:00
GPUpower: whole chip, CPU and GPUtemperature: one board zone

GPU 0

Wwhole chip, CPU and GPU

118 W whole chip, CPU and GPU

use100%
memory, whole GPU9.3 GB

this run's peak 8.0 GB

91 °C one board zone

history 0 to 124 W

Input

  • Runs128 runs
  • Agentsnone showing
  • Treesnone showing
  • Projectsnone showing

Check

Every entry is checked on its way out; some are held for review.

Not shown on any run: data samples, predictions, raw logs, file paths, machine names

Delay

2 min, set

Watching the agents work, about 2 min behind.

One way

Nothing here reaches the agents.

This page

This page updates by itself.

visits counted with Google Analytics · choose

Runsheight is eval accuracyall recordings ▸ | finished× failed■ stopped○ lost● live
0.950.750.950.75finished, eval accuracy 0.8077, config 1 of 11 (batch 8), seed 2, Mon 2 Nov, 13:19 UTCfinished, eval accuracy 0.8231, config 2 of 11 (batch 8), seed 2 of 3, Mon 2 Nov, 14:03 UTCfinished, eval accuracy 0.8231, config 3 of 11 (batch 8), seed 2, Mon 2 Nov, 14:09 UTCfinished, eval accuracy 0.7846, config 2 of 11 (batch 8), seed 0 of 3, Mon 2 Nov, 14:16 UTCfinished, eval accuracy 0.8385, config 2 of 11 (batch 8), seed 1 of 3, Mon 2 Nov, 14:22 UTCfinished, eval accuracy 0.7923, config 4 of 11 (batch 8), seed 0 of 3, Mon 2 Nov, 18:15 UTCfinished, eval accuracy 0.8000, config 4 of 11 (batch 8), seed 1 of 3, Mon 2 Nov, 18:22 UTCfinished, eval accuracy 0.8154, config 4 of 11 (batch 8), seed 2 of 3, Mon 2 Nov, 18:28 UTCfinished, eval accuracy 0.7923, config 5 of 11 (batch 8), seed 0 of 3, Mon 2 Nov, 18:46 UTCfinished, eval accuracy 0.8385, config 5 of 11 (batch 8), seed 1 of 3, Mon 2 Nov, 18:52 UTCfinished, eval accuracy 0.8000, config 5 of 11 (batch 8), seed 2 of 3, Mon 2 Nov, 18:59 UTCfinished, eval accuracy 0.8231, config 6 of 11 (batch 8), seed 0 of 3, Tue 3 Nov, 00:59 UTCfinished, eval accuracy 0.8231, config 6 of 11 (batch 8), seed 1 of 3, Tue 3 Nov, 01:13 UTCfinished, eval accuracy 0.8000, config 6 of 11 (batch 8), seed 2 of 3, Tue 3 Nov, 01:26 UTCfinished at step 336 of 339, eval accuracy 0.8769, config 7 of 11 (batch 8), seed 0 of 3, Thu 19 Nov, 21:49 UTCfinished at step 336 of 339, eval accuracy 0.8846, config 7 of 11 (batch 8), seed 1 of 3, Thu 19 Nov, 22:06 UTCfinished at step 336 of 339, eval accuracy 0.8846, config 7 of 11 (batch 8), seed 2 of 3, Thu 19 Nov, 22:21 UTCfinished at step 336 of 339, eval accuracy 0.8615, config 8 of 11 (learning rate 0.0001, batch 8), seed 0, Sat 21 Nov, 23:04 UTCfinished at step 336 of 339, eval accuracy 0.9231, config 9 of 11 (learning rate 0.00002, batch 8), seed 0, Sat 21 Nov, 23:17 UTCfinished at step 336 of 339, eval accuracy 0.8538, config 10 of 11 (learning rate 0.00005, batch 8), seed 0, Sat 21 Nov, 23:29 UTCfinished at step 336 of 339, eval accuracy 0.9231, config 11 of 11 (learning rate 0.00002, batch 8), seed 1 of 2, Sun 22 Nov, 03:49 UTCfinished at step 336 of 339, eval accuracy 0.8846, config 11 of 11 (learning rate 0.00002, batch 8), seed 2 of 2, Sun 22 Nov, 04:01 UTC0.950.75finished, eval accuracy 0.8077, config 1 of 11 (batch 8), seed 2, Mon 2 Nov, 13:19 UTCfinished, eval accuracy 0.8231, config 2 of 11 (batch 8), seed 2 of 3, Mon 2 Nov, 14:03 UTCfinished, eval accuracy 0.8231, config 3 of 11 (batch 8), seed 2, Mon 2 Nov, 14:09 UTCfinished, eval accuracy 0.7846, config 2 of 11 (batch 8), seed 0 of 3, Mon 2 Nov, 14:16 UTCfinished, eval accuracy 0.8385, config 2 of 11 (batch 8), seed 1 of 3, Mon 2 Nov, 14:22 UTCfinished, eval accuracy 0.7923, config 4 of 11 (batch 8), seed 0 of 3, Mon 2 Nov, 18:15 UTCfinished, eval accuracy 0.8000, config 4 of 11 (batch 8), seed 1 of 3, Mon 2 Nov, 18:22 UTCfinished, eval accuracy 0.8154, config 4 of 11 (batch 8), seed 2 of 3, Mon 2 Nov, 18:28 UTCfinished, eval accuracy 0.7923, config 5 of 11 (batch 8), seed 0 of 3, Mon 2 Nov, 18:46 UTCfinished, eval accuracy 0.8385, config 5 of 11 (batch 8), seed 1 of 3, Mon 2 Nov, 18:52 UTCfinished, eval accuracy 0.8000, config 5 of 11 (batch 8), seed 2 of 3, Mon 2 Nov, 18:59 UTCfinished, eval accuracy 0.8231, config 6 of 11 (batch 8), seed 0 of 3, Tue 3 Nov, 00:59 UTCfinished, eval accuracy 0.8231, config 6 of 11 (batch 8), seed 1 of 3, Tue 3 Nov, 01:13 UTCfinished, eval accuracy 0.8000, config 6 of 11 (batch 8), seed 2 of 3, Tue 3 Nov, 01:26 UTCfinished at step 336 of 339, eval accuracy 0.8769, config 7 of 11 (batch 8), seed 0 of 3, Thu 19 Nov, 21:49 UTCfinished at step 336 of 339, eval accuracy 0.8846, config 7 of 11 (batch 8), seed 1 of 3, Thu 19 Nov, 22:06 UTCfinished at step 336 of 339, eval accuracy 0.8846, config 7 of 11 (batch 8), seed 2 of 3, Thu 19 Nov, 22:21 UTCfinished at step 336 of 339, eval accuracy 0.8615, config 8 of 11 (learning rate 0.0001, batch 8), seed 0, Sat 21 Nov, 23:04 UTCfinished at step 336 of 339, eval accuracy 0.9231, config 9 of 11 (learning rate 0.00002, batch 8), seed 0, Sat 21 Nov, 23:17 UTCfinished at step 336 of 339, eval accuracy 0.8538, config 10 of 11 (learning rate 0.00005, batch 8), seed 0, Sat 21 Nov, 23:29 UTCfinished at step 336 of 339, eval accuracy 0.9231, config 11 of 11 (learning rate 0.00002, batch 8), seed 1 of 2, Sun 22 Nov, 03:49 UTCfinished at step 336 of 339, eval accuracy 0.8846, config 11 of 11 (learning rate 0.00002, batch 8), seed 2 of 2, Sun 22 Nov, 04:01 UTC

20 of 22 finished at step 336 of 339, 0.8538, config 10 of 11 (learning rate 0.00005, batch 8), seed 0, Sat 21 Nov, 23:29 UTC

Runs as a table
StartedStatusEval accuracyStepsConfig
Sun 22 Nov, 04:01 UTC| finished0.8846336 of 339learning rate 0.00002, batch 8, seed 2 of 2
Sun 22 Nov, 03:49 UTC| finished0.9231336 of 339learning rate 0.00002, batch 8, seed 1 of 2
Sat 21 Nov, 23:29 UTC| finished0.8538336 of 339learning rate 0.00005, batch 8, seed 0
Sat 21 Nov, 23:17 UTC| finished0.9231336 of 339learning rate 0.00002, batch 8, seed 0
Sat 21 Nov, 23:04 UTC| finished0.8615336 of 339learning rate 0.0001, batch 8, seed 0
Thu 19 Nov, 22:21 UTC| finished0.8846336 of 339batch 8, seed 2 of 3
Thu 19 Nov, 22:06 UTC| finished0.8846336 of 339batch 8, seed 1 of 3
Thu 19 Nov, 21:49 UTC| finished0.8769336 of 339batch 8, seed 0 of 3
Tue 3 Nov, 01:26 UTC| finished0.8000150 of 150batch 8, seed 2 of 3
Tue 3 Nov, 01:13 UTC| finished0.8231150 of 150batch 8, seed 1 of 3
Tue 3 Nov, 00:59 UTC| finished0.8231150 of 150batch 8, seed 0 of 3
Mon 2 Nov, 18:59 UTC| finished0.8000150 of 150batch 8, seed 2 of 3
Mon 2 Nov, 18:52 UTC| finished0.8385150 of 150batch 8, seed 1 of 3
Mon 2 Nov, 18:46 UTC| finished0.7923150 of 150batch 8, seed 0 of 3
Mon 2 Nov, 18:28 UTC| finished0.8154150 of 150batch 8, seed 2 of 3
Mon 2 Nov, 18:22 UTC| finished0.8000150 of 150batch 8, seed 1 of 3
Mon 2 Nov, 18:15 UTC| finished0.7923150 of 150batch 8, seed 0 of 3
Mon 2 Nov, 14:22 UTC| finished0.8385150 of 150batch 8, seed 1 of 3
Mon 2 Nov, 14:16 UTC| finished0.7846150 of 150batch 8, seed 0 of 3
Mon 2 Nov, 14:09 UTC| finished0.8231150 of 150batch 8, seed 2
Mon 2 Nov, 14:03 UTC| finished0.8231150 of 150batch 8, seed 2 of 3
Mon 2 Nov, 13:19 UTC| finished0.8077150 of 150batch 8, seed 2

Plate

Model
convaiinnovations/laya, 421M parameters revision 7b928d8
Data
Synthetic Discord triage set, 2,000-row draw, synthetic1800 train130 eval
Config
learning rate 0.00005, batch size 8, grad accum 2, epochs 3, loss rlcd, seed 0
Hardware
Radeon 8060S, ROCm

Log

Since startMark
0.0 sphase starting
13.4 swent public
35.2 sphase training
250.0 sepoch 1
475.9 sepoch 2
702.4 sepoch 3
745.2 sphase eval
745.2 seval 1

Readings

67 rows as a table
StepSecondsTrain lossEval lossEval accuracyECEBrierSamples per secondLearning rateGPU GB, this run
035.1-----7.5995.00e-56.5
545.10.7014----7.9774.99e-56.5
1055.00.6107----8.1164.98e-56.5
1565.00.7208----7.9544.96e-56.5
2075.20.0508----7.8734.94e-56.5
2585.31.2213----7.9394.91e-56.5
3094.70.5609----8.4754.88e-56.5
35104.71.2585----8.0174.84e-56.5
40114.40.2640----8.2154.80e-56.5
45125.51.3991----7.2194.75e-56.5
50135.20.6874----8.2764.70e-56.5
55145.30.3525----7.8574.64e-56.5
60155.50.6856----7.8564.58e-56.5
65165.40.2085----8.1234.52e-56.5
70175.30.6579----8.0354.45e-56.5
75185.00.0637----8.3064.37e-56.5
80194.70.6382----8.1974.29e-56.5
85205.50.8974----7.4564.21e-56.5
90215.30.7597----8.1454.13e-56.5
95225.02.0271----8.2624.04e-56.5
100234.61.1855----8.2733.95e-56.5
105244.50.7784----8.0943.85e-56.5
110255.10.8073----7.5253.76e-56.5
118265.10.6972----8.0043.66e-56.5
123275.00.6018----8.1563.55e-56.5
128284.90.5832----8.0063.45e-56.5
133294.70.8229----8.1783.34e-56.5
138304.80.4204----7.9093.23e-56.5
143314.4-0.4428----8.3283.12e-56.5
148323.9-0.0112----8.4863.01e-56.5
153334.30.5347----7.6962.90e-56.5
158344.30.7026----7.9962.79e-56.5
163354.41.2586----7.8732.67e-56.5
168364.30.2213----8.0992.56e-56.5
173374.1-0.0239----8.1362.45e-56.5
178384.50.3920----7.6932.33e-56.5
183394.50.0309----8.0532.22e-56.5
188404.60.2681----7.9422.11e-56.5
193414.70.1338----7.8722.00e-56.5
198424.70.1603----8.0501.89e-56.5
203434.90.8554----7.7861.78e-56.5
208445.40.2472----7.6591.67e-56.5
213455.50.2402----7.9331.57e-56.5
218464.70.7223----8.6561.47e-56.5
223484.80.2711----7.8631.27e-56.5
231494.80.0052----8.0061.17e-56.5
236504.20.5004----8.5441.08e-56.5
241514.4-0.1101----7.8509.90e-66.5
246524.60.0072----7.7969.04e-66.5
251535.0-0.1703----7.7518.22e-66.5
256544.7-0.4028----8.2297.43e-66.5
261554.80.0339----7.9086.68e-66.5
266565.10.0000----7.7895.98e-66.5
271575.20.0000----7.8895.31e-66.5
276585.80.0000----7.5224.69e-66.5
281595.4-0.0357----8.3324.11e-66.5
286605.01.9539----8.3293.58e-66.5
291615.30.3582----7.7853.10e-66.5
296625.30.1404----8.0362.66e-66.5
301634.8-0.0091----8.4462.28e-66.5
306645.10.0000----7.7411.94e-66.5
311654.90.0000----8.1571.65e-66.5
316664.70.0000----8.1631.42e-66.5
321674.50.0000----8.1731.24e-66.5
326684.8-0.3238----7.7901.11e-66.5
331695.0-0.1190----7.7901.03e-66.5
336746.52.49430.46910.85380.08770.23412.3561.00e-14.9

GPU readings

67 rows as a table
StepGPUUse %Power W, whole chip, CPU and GPUMemory GB, whole GPUTemp C, one board zone
0010080.33.589.8
150100118.29.087.8
30045122.39.089.8
450100122.09.088.8
60087119.99.089.8
80098114.19.089.8
95085115.49.189.8
1100100114.69.189.8
1280100118.69.190.8
1430100112.79.390.8
1580100116.29.390.8
173091106.39.390.8
193087123.59.390.8
208097118.79.390.8
22308166.49.390.8
241098119.19.390.8
256075118.99.389.8
271045118.89.389.8
286078120.19.390.8
306084117.49.390.8
3210100114.69.390.8
3360100118.09.390.8

Worked out

ReadoutValueWorked out from
Run complete99.1%336 steps over 339
Step time1.99 s659.9 s over 331 steps, between readings. Startup and intervals that hold an eval are left out
Steps per minute30.160 s over the step time of 1.99 s
Epoch interval226.2 sMean of 2 gaps between epoch marks
Train loss since first reading+1.79292.4943 at step 336 minus 0.7014 at step 5
Lowest train loss-0.4428Lowest of 66 readings, at step 143
Eval accuracy, as shown85.4%0.8538 times 100, to one decimal. Changes on that plate are percentage points
Train minus eval loss+2.02522.4943 minus 0.4691, both at step 336
Wrong answers19130 eval items minus 111 correct. 111 over 130 is the only count that gives 0.8538
Rank8 of 22By last eval accuracy, across this project's 22 runs with an eval
Best so far0.9231Highest last eval accuracy, run 19
Against the run before−0.06930.8538 minus 0.9231 on run 19
Lowest last train loss-0.0937Lowest last train loss across this project's 22 runs, run 19
Train loss against run 19+2.58802.4943 minus -0.0937 on run 19, both at step 336
Samples per second since first reading−5.242.36 minus 7.60
Learning rate since first reading+1.00e-11.00e-1 minus 5.00e-5
This run's peak memory per GPU8.0 GBAs the run reports it: its own memory, not the whole GPU's