e2e-resume-001

completed idea: BatchNorm in a small MNIST CNN updated 1h ago

Idea Environment Baseline Setup Implement GPUs Venv Data Tests Train Monitor Eval Report

orchestrator host: MBP-Artem-2.fritz.box

Compute

backend: slurm
cluster: capella
partition: capella-interactive
job_id: 3922209
gpus: 1
gpu_type: H100 MIG
cpus: 1
mem: 20G
nodes: 1
host: dresden-capella
workspace: /data/horse/ws/arma647i-mlexp
label: capella:3922209
interactive: True
source: dresden-holder
reused_job_name: mlexp-holder-mnist-cnn-batchnorm-i
time_left: 49:00
vram_gb: 16

Setup

baseline_source: train-alongside
run_baseline: True
train_data: MNIST train split, 55000 images
val_data: MNIST train split, last 5000 images held out
benchmarks: mnist-test
hyperparams: batch_size = 128
lr = 0.001
epochs = 2
seed = 0
scaling: global_batch_size = 128
scale_lr = sqrt
min_per_device_batch = 32
max_per_device_batch = 0
allow_step_scaling = False
adjustments: per-device batch 128 x 1 GPU(s) -> global batch 128
gpus: 1
global_batch_size: 128
requested_global_batch_size: 128
train_started_at: 1786969585.370067

Metrics

loss: 0.0539
val_loss: 0.0426
val_acc: 0.9872
variant: idea
test_loss: 0.040121
test_acc: 0.9866
test_error_pct: 1.34
eval_seconds: 0.26
device: cuda
n_test: 10000

Baseline metrics

variant: baseline
test_loss: 0.043111
test_acc: 0.9862
test_error_pct: 1.38
eval_seconds: 0.23
device: cuda
n_test: 10000

Artifacts

dashboard_url: https://main.research.makartkar.com/runs/e2e-resume-001
repo_commit: 83ac2db
repo_local_path: /Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn
implementation_diffstat: (clean tree)
compute_target: capella:3922209
compute_source: dresden-holder
venv_path: /data/horse/ws/arma647i-mlexp/projects/mnist-cnn/.venv
project_path: /data/horse/ws/arma647i-mlexp/worktrees/mnist-cnn-e2e-resume-001
python_version: Python 3.12.13
data_dir: /data/horse/ws/arma647i-mlexp/.data/mnist
data_cache: reused
unit_tests_log: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/unit_tests.log
smoke_run_log: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/smoke_run.log
train_log: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/train_idea.log
baseline_train_log: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/train_baseline.log
monitor_checks: 1
eval_idea_log: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/eval_idea.log
eval_metrics_file: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/results/idea_metrics.json
eval_baseline_log: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/eval_baseline.log
baseline_metrics_file: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/results/baseline_metrics.json
comparison: test_loss: 0.040121 vs baseline 0.043111 (-0.00299) | test_acc: 0.9866 vs baseline 0.9862 (+0.0004) | test_error_pct: 1.34 vs baseline 1.38 (-0.04) | eval_seconds: 0.26 vs baseline 0.23 (+0.03) | n_test: 10000 vs baseline 10000 (+0)
local_train.log: /Users/makartkar/.local/state/mlexp/runs/e2e-resume-001/train.log
local_baseline_train.log: /Users/makartkar/.local/state/mlexp/runs/e2e-resume-001/baseline_train.log
local_eval_idea.log: /Users/makartkar/.local/state/mlexp/runs/e2e-resume-001/eval_idea.log
data_cache_path: /data/horse/ws/arma647i-mlexp/.data/mnist
run_dir: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001
results_dir: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/results
setup_json: /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/setup.json
local_run_dir: /Users/makartkar/.local/state/mlexp/runs/e2e-resume-001
report_md: /Users/makartkar/.local/state/mlexp/runs/e2e-resume-001/report.md
report_json: /Users/makartkar/.local/state/mlexp/runs/e2e-resume-001/report.json

Event log (70)

1h ago report done report in 4.4s

1h ago report report written to /Users/makartkar/.local/state/mlexp/runs/e2e-resume-001/report.md

1h ago report start report

1h ago evaluate done evaluate in 89.5s

1h ago evaluate n_test: 10000 vs baseline 10000 (+0)

1h ago evaluate eval_seconds: 0.26 vs baseline 0.23 (+0.03)

1h ago evaluate test_error_pct: 1.34 vs baseline 1.38 (-0.04)

1h ago evaluate test_acc: 0.9866 vs baseline 0.9862 (+0.0004)

1h ago evaluate test_loss: 0.040121 vs baseline 0.043111 (-0.00299)

1h ago evaluate eval_baseline metrics: {'variant': 'baseline', 'test_loss': 0.043111, 'test_acc': 0.9862, 'test_error_pct': 1.38, 'eval_seconds': 0.23, 'device': 'cuda', 'n_test': 10000}

1h ago evaluate 'eval_baseline' still running (4s): RUN_MARKER step=eval_baseline exit_code=0 end_epoch=1786969732

1h ago evaluate launched 'eval_baseline' (pid 1545847) -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/eval_baseline.log

1h ago evaluate eval_idea metrics: {'variant': 'idea', 'test_loss': 0.040121, 'test_acc': 0.9866, 'test_error_pct': 1.34, 'eval_seconds': 0.26, 'device': 'cuda', 'n_test': 10000}

1h ago evaluate 'eval_idea' still running (4s): RUN_MARKER step=eval_idea exit_code=0 end_epoch=1786969689

1h ago evaluate launched 'eval_idea' (pid 1541805) -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/eval_idea.log

1h ago evaluate start evaluate

1h ago monitor done monitor in 62.9s

1h ago monitor all training jobs finished successfully

1h ago monitor train_baseline exited with rc=0

1h ago monitor train_idea exited with rc=0

1h ago monitor supervising 2 job(s); check every 40s, hard limit 20 min

1h ago monitor start monitor

1h ago monitor supervising 2 job(s); check every 40s, hard limit 20 min

1h ago monitor start monitor

1h ago train done train in 42.9s

1h ago train baseline training started -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/train_baseline.log

1h ago train training started -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/train_idea.log

1h ago train start train

1h ago testing done testing in 85.6s

1h ago testing smoke_run passed

1h ago testing 'smoke_run' still running (4s): training finished in 0.4s

1h ago testing launched 'smoke_run' (pid 1527521) -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/smoke_run.log

1h ago testing unit_tests passed

1h ago testing 'unit_tests' still running (4s): RUN_MARKER step=unit_tests exit_code=0 end_epoch=1786969483

1h ago testing launched 'unit_tests' (pid 1519682) -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/unit_tests.log

1h ago testing start testing

1h ago data_prep done data_prep in 8.8s

1h ago data_prep data cache already prepared (/data/horse/ws/arma647i-mlexp/.data/mnist/mnist.npz)

1h ago data_prep start data_prep

1h ago venv_prep done venv_prep in 154.2s

1h ago venv_prep venv created and dependencies installed

1h ago venv_prep launched 'venv_prep' (pid 1515234) -> /data/horse/ws/arma647i-mlexp/runs/e2e-resume-001/venv_prep.log

1h ago venv_prep materialising run tree /data/horse/ws/arma647i-mlexp/worktrees/mnist-cnn-e2e-resume-001

1h ago venv_prep syncing /Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn -> capella:3922209:/data/horse/ws/arma647i-mlexp/projects/mnist-cnn

1h ago venv_prep start venv_prep

1h ago gpu_select done gpu_select in 21.5s

1h ago gpu_select setup adjusted: per-device batch 128 x 1 GPU(s) -> global batch 128

1h ago gpu_select attached to capella job 3922209 with 1 GPU(s)

1h ago gpu_select capella: probing job 3922209 (mlexp-holder-mnist-cnn-batchnorm-i)

1h ago gpu_select trying compute backend: dresden-holder

1h ago gpu_select start gpu_select

1h ago implement done implement in 0.6s

1h ago implement verifying the implementation

1h ago implement no implement step configured; using the code as it is

1h ago implement start implement

1h ago setup done setup in 0.5s

1h ago setup benchmarks: ['mnist-test']

1h ago setup hyperparameters: {'batch_size': 128, 'lr': 0.001, 'epochs': 2, 'seed': 0}

1h ago setup start setup

1h ago baseline done baseline in 0.3s

1h ago baseline baseline 'plain-cnn' will be trained alongside the idea on the same machine

1h ago baseline start baseline

1h ago environment done environment in 0.6s

1h ago environment project source: /Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn @ 83ac2db

1h ago environment working tree has uncommitted changes: ?? ../../../ ?? ../../../../../how_to/ml-experiment-run-idea.md ?? ../../../../../knowledge/ml-experiment-automation.md ?? ../../../../../skills/ml-experiment-automation/

1h ago environment start environment

1h ago idea done idea in 0.7s

1h ago idea hypothesis: BatchNorm raises MNIST test accuracy over the plain CNN at an equal training budget, without increasing wall-clock time meaningfully.

1h ago idea idea registered: BatchNorm in a small MNIST CNN

1h ago idea start idea