ML Experiment Automation

idea → setup → compute → train → eval → report auto-refresh 15s

Ideas (1)

BatchNorm in a small MNIST CNN

mnist-cnn-batchnorm completed e2ecvmnistsmoke 58s ago
Insert BatchNorm2d after each convolution of a two-block MNIST CNN and compare against the identical network without it. Same data, same hyperparameters, same seed, same number of epochs - the only difference is the normalisation layer. Used as the end-to-end test of the automation pipeline, so it is deliberately small and fast.
Hypothesis: BatchNorm raises MNIST test accuracy over the plain CNN at an equal training budget, without increasing wall-clock time meaningfully.
repo: /Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn  |  baseline: plain-cnn
Run Status Pipeline Compute Metrics Updated
e2e-resume-001 completed
Idea Environment Baseline Setup Implement GPUs Venv Data Tests Train Monitor Eval Report
slurm / capella
1x H100 MIG
job 3922209
loss 0.0539 val_loss 0.0426 val_acc 0.9872 variant idea test_loss 0.040121 test_acc 0.9866 test_error_pct 1.34 eval_seconds 0.26 device cuda n_test 10000
base: variant=baseline test_loss=0.043111 test_acc=0.9862 test_error_pct=1.38 eval_seconds=0.23 device=cuda n_test=10000
57s ago
e2e-dresden-005 completed
Idea Environment Baseline Setup Implement GPUs Venv Data Tests Train Monitor Eval Report
slurm / capella
1x H100 MIG
job 3922209
loss 0.0534 val_loss 0.0428 val_acc 0.9882 variant idea test_loss 0.041019 test_acc 0.9857 test_error_pct 1.43 eval_seconds 0.34 device cuda n_test 10000
base: variant=baseline test_loss=0.044817 test_acc=0.9854 test_error_pct=1.46 eval_seconds=0.38 device=cuda n_test=10000
10m ago
e2e-local-006 completed
Idea Environment Baseline Setup Implement GPUs Venv Data Tests Train Monitor Eval Report
local
0x cpu
loss 0.0538 val_loss 0.0433 val_acc 0.9876 variant idea test_loss 0.04103 test_acc 0.9867 test_error_pct 1.33 eval_seconds 0.45 device mps n_test 10000
base: variant=baseline test_loss=0.04256 test_acc=0.9856 test_error_pct=1.44 eval_seconds=0.22 device=mps n_test=10000
47m ago

Recent activity

57s ago report e2e-resume-001 done report in 4.4s
58s ago report e2e-resume-001 report written to /Users/makartkar/.local/state/mlexp/runs/e2e-resume-001/report.md
1m ago report e2e-resume-001 start report
1m ago evaluate e2e-resume-001 done evaluate in 89.5s
1m ago evaluate e2e-resume-001 n_test: 10000 vs baseline 10000 (+0)
1m ago evaluate e2e-resume-001 eval_seconds: 0.26 vs baseline 0.23 (+0.03)
1m ago evaluate e2e-resume-001 test_error_pct: 1.34 vs baseline 1.38 (-0.04)
1m ago evaluate e2e-resume-001 test_acc: 0.9866 vs baseline 0.9862 (+0.0004)
1m ago evaluate e2e-resume-001 test_loss: 0.040121 vs baseline 0.043111 (-0.00299)
1m ago evaluate e2e-resume-001 eval_baseline metrics: {'variant': 'baseline', 'test_loss': 0.043111, 'test_acc': 0.9862, 'test_error_pct': 1.38, 'eval_seconds': 0.23, 'device': 'cuda', 'n_test': 10000}
1m ago evaluate e2e-resume-001 'eval_baseline' still running (4s): RUN_MARKER step=eval_baseline exit_code=0 end_epoch=1786969732
1m ago evaluate e2e-resume-001 launched 'eval_baseline' (pid 1545847) -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/eval_baseline.log
1m ago evaluate e2e-resume-001 eval_idea metrics: {'variant': 'idea', 'test_loss': 0.040121, 'test_acc': 0.9866, 'test_error_pct': 1.34, 'eval_seconds': 0.26, 'device': 'cuda', 'n_test': 10000}
2m ago evaluate e2e-resume-001 'eval_idea' still running (4s): RUN_MARKER step=eval_idea exit_code=0 end_epoch=1786969689
2m ago evaluate e2e-resume-001 launched 'eval_idea' (pid 1541805) -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/eval_idea.log
2m ago evaluate e2e-resume-001 start evaluate
2m ago monitor e2e-resume-001 done monitor in 62.9s
2m ago monitor e2e-resume-001 all training jobs finished successfully
2m ago monitor e2e-resume-001 train_baseline exited with rc=0
2m ago monitor e2e-resume-001 train_idea exited with rc=0
3m ago monitor e2e-resume-001 supervising 2 job(s); check every 40s, hard limit 20 min
3m ago monitor e2e-resume-001 start monitor
3m ago monitor e2e-resume-001 supervising 2 job(s); check every 40s, hard limit 20 min
3m ago monitor e2e-resume-001 start monitor
3m ago train e2e-resume-001 done train in 42.9s
3m ago train e2e-resume-001 baseline training started -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/train_baseline.log
4m ago train e2e-resume-001 training started -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/train_idea.log
4m ago train e2e-resume-001 start train
4m ago testing e2e-resume-001 done testing in 85.6s
4m ago testing e2e-resume-001 smoke_run passed
4m ago testing e2e-resume-001 'smoke_run' still running (4s): training finished in 0.4s
4m ago testing e2e-resume-001 launched 'smoke_run' (pid 1527521) -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/smoke_run.log
5m ago testing e2e-resume-001 unit_tests passed
5m ago testing e2e-resume-001 'unit_tests' still running (4s): RUN_MARKER step=unit_tests exit_code=0 end_epoch=1786969483
5m ago testing e2e-resume-001 launched 'unit_tests' (pid 1519682) -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/unit_tests.log
6m ago testing e2e-resume-001 start testing
6m ago data_prep e2e-resume-001 done data_prep in 8.8s
6m ago data_prep e2e-resume-001 data cache already prepared (/data/horse/ws/arma647i-mlexp/.data/mnist/mnist.npz)
6m ago data_prep e2e-resume-001 start data_prep
6m ago venv_prep e2e-resume-001 done venv_prep in 154.2s