e2e-dresden-005

completed idea: BatchNorm in a small MNIST CNN updated 1h ago

Idea Environment Baseline Setup Implement GPUs Venv Data Tests Train Monitor Eval Report

orchestrator host: MBP-Artem-2.fritz.box

Compute

backend: slurm
cluster: capella
partition: capella-interactive
job_id: 3922209
gpus: 1
gpu_type: H100 MIG
cpus: 1
mem: 20G
nodes: 1
host: dresden-capella
workspace: /data/horse/ws/arma647i-mlexp
label: capella:3922209
interactive: True
source: dresden-alloc
time_left: 59:48
vram_gb: 16
standby: cluster = capella
partition = capella
host = dresden-capella
job_id = 3922214
workspace = /data/horse/ws/arma647i-mlexp
role = standby-holder

Setup

baseline_source: train-alongside
run_baseline: True
train_data: MNIST train split, 55000 images
val_data: MNIST train split, last 5000 images held out
benchmarks: mnist-test
hyperparams: batch_size = 128
lr = 0.001
epochs = 2
seed = 0
scaling: global_batch_size = 128
scale_lr = sqrt
min_per_device_batch = 32
max_per_device_batch = 0
allow_step_scaling = False
adjustments: per-device batch 128 x 1 GPU(s) -> global batch 128
gpus: 1
global_batch_size: 128
requested_global_batch_size: 128
train_started_at: 1786969053.692113

Metrics

loss: 0.0534
val_loss: 0.0428
val_acc: 0.9882
variant: idea
test_loss: 0.041019
test_acc: 0.9857
test_error_pct: 1.43
eval_seconds: 0.34
device: cuda
n_test: 10000

Baseline metrics

variant: baseline
test_loss: 0.044817
test_acc: 0.9854
test_error_pct: 1.46
eval_seconds: 0.38
device: cuda
n_test: 10000

Artifacts

dashboard_url: https://main.research.makartkar.com/runs/e2e-dresden-005
repo_commit: 83ac2db
repo_local_path: /Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn
implementation_diffstat: (clean tree)
compute_target: capella:3922209
compute_source: dresden-alloc
standby_holder: capella:3922214
venv_path: /data/horse/ws/arma647i-mlexp/projects/mnist-cnn/.venv
project_path: /data/horse/ws/arma647i-mlexp/worktrees/mnist-cnn-e2e-dresden-005
python_version: Python 3.12.13
data_dir: /data/horse/ws/arma647i-mlexp/.data/mnist
data_prep_log: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/data_prep.log
data_cache: built
unit_tests_log: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/unit_tests.log
smoke_run_log: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/smoke_run.log
train_log: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/train_idea.log
baseline_train_log: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/train_baseline.log
monitor_checks: 1
eval_idea_log: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/eval_idea.log
eval_metrics_file: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/results/idea_metrics.json
eval_baseline_log: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/eval_baseline.log
baseline_metrics_file: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/results/baseline_metrics.json
comparison: test_loss: 0.041019 vs baseline 0.044817 (-0.003798) | test_acc: 0.9857 vs baseline 0.9854 (+0.0003) | test_error_pct: 1.43 vs baseline 1.46 (-0.03) | eval_seconds: 0.34 vs baseline 0.38 (-0.04) | n_test: 10000 vs baseline 10000 (+0)
local_train.log: /Users/makartkar/.local/state/mlexp/runs/e2e-dresden-005/train.log
local_baseline_train.log: /Users/makartkar/.local/state/mlexp/runs/e2e-dresden-005/baseline_train.log
local_eval_idea.log: /Users/makartkar/.local/state/mlexp/runs/e2e-dresden-005/eval_idea.log
data_cache_path: /data/horse/ws/arma647i-mlexp/.data/mnist
run_dir: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005
results_dir: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/results
setup_json: /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/setup.json
local_run_dir: /Users/makartkar/.local/state/mlexp/runs/e2e-dresden-005
report_md: /Users/makartkar/.local/state/mlexp/runs/e2e-dresden-005/report.md
report_json: /Users/makartkar/.local/state/mlexp/runs/e2e-dresden-005/report.json

Event log (82)

1h ago report done report in 10.0s

1h ago report released standby holder capella:3922214

1h ago report report written to /Users/makartkar/.local/state/mlexp/runs/e2e-dresden-005/report.md

1h ago report start report

1h ago evaluate done evaluate in 87.5s

1h ago evaluate n_test: 10000 vs baseline 10000 (+0)

1h ago evaluate eval_seconds: 0.34 vs baseline 0.38 (-0.04)

1h ago evaluate test_error_pct: 1.43 vs baseline 1.46 (-0.03)

1h ago evaluate test_acc: 0.9857 vs baseline 0.9854 (+0.0003)

1h ago evaluate test_loss: 0.041019 vs baseline 0.044817 (-0.003798)

1h ago evaluate eval_baseline metrics: {'variant': 'baseline', 'test_loss': 0.044817, 'test_acc': 0.9854, 'test_error_pct': 1.46, 'eval_seconds': 0.38, 'device': 'cuda', 'n_test': 10000}

1h ago evaluate 'eval_baseline' still running (4s): RUN_MARKER step=eval_baseline exit_code=0 end_epoch=1786969179

1h ago evaluate launched 'eval_baseline' (pid 1492430) -> /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/eval_baseline.log

1h ago evaluate eval_idea metrics: {'variant': 'idea', 'test_loss': 0.041019, 'test_acc': 0.9857, 'test_error_pct': 1.43, 'eval_seconds': 0.34, 'device': 'cuda', 'n_test': 10000}

1h ago evaluate 'eval_idea' still running (4s): RUN_MARKER step=eval_idea exit_code=0 end_epoch=1786969136

1h ago evaluate launched 'eval_idea' (pid 1488585) -> /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/eval_idea.log

1h ago evaluate start evaluate

1h ago monitor done monitor in 60.6s

1h ago monitor all training jobs finished successfully

1h ago monitor train_baseline exited with rc=0

1h ago monitor train_idea exited with rc=0

1h ago monitor supervising 2 job(s); check every 40s, hard limit 20 min

1h ago monitor start monitor

1h ago train done train in 38.6s

1h ago train baseline training started -> /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/train_baseline.log

1h ago train training started -> /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/train_idea.log

1h ago train start train

1h ago testing done testing in 80.5s

1h ago testing smoke_run passed

1h ago testing 'smoke_run' still running (3s): model parameters: 207018

1h ago testing launched 'smoke_run' (pid 1476358) -> /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/smoke_run.log

1h ago testing unit_tests passed

1h ago testing 'unit_tests' still running (3s): RUN_MARKER inner host=c2 gpus=5

1h ago testing launched 'unit_tests' (pid 1472373) -> /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/unit_tests.log

1h ago testing start testing

1h ago data_prep done data_prep in 47.1s

1h ago data_prep data prepared

1h ago data_prep 'data_prep' still running (4s): downloading https://storage.googleapis.com/cvdf-datasets/mnist/train-labels-idx1-ubyte.gz

1h ago data_prep launched 'data_prep' (pid 1464766) -> /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/data_prep.log

1h ago data_prep start data_prep

1h ago venv_prep done venv_prep in 245.0s

1h ago venv_prep venv created and dependencies installed

1h ago venv_prep 'venv_prep' still running (107s): Downloaded nvidia-cuda-nvrtc

1h ago venv_prep 'venv_prep' still running (4s): Downloading cpython-3.12.13-linux-x86_64-gnu (download) (32.6MiB)

1h ago venv_prep launched 'venv_prep' (pid 1446694) -> /data/horse/ws/arma647i-mlexp/runs/e2e-dresden-005/venv_prep.log

1h ago venv_prep materialising run tree /data/horse/ws/arma647i-mlexp/worktrees/mnist-cnn-e2e-dresden-005

1h ago venv_prep syncing /Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn -> capella:3922209:/data/horse/ws/arma647i-mlexp/projects/mnist-cnn

1h ago venv_prep start venv_prep

1h ago gpu_select done gpu_select in 81.9s

1h ago gpu_select setup adjusted: per-device batch 128 x 1 GPU(s) -> global batch 128

1h ago gpu_select allocated capella-interactive job 3922209 (capella-interactive 1x1 GPU, 1 CPU, 1h)

1h ago gpu_select cancelled surplus submission 2320108

1h ago gpu_select cancelled surplus submission 2320107

1h ago gpu_select keeping capella job 3922214 as standby holder

1h ago gpu_select submitted holder 2320108: alpha 1x1 GPU, 6 CPU, 1h

1h ago gpu_select submitted holder 2320107: alpha-interactive 1x1 GPU, 1 CPU, 1h

1h ago gpu_select submitted holder 3922214: capella 1x1 GPU, 14 CPU, 1h

1h ago gpu_select submitted holder 3922209: capella-interactive 1x1 GPU, 1 CPU, 1h

1h ago gpu_select trying compute backend: dresden-alloc

1h ago gpu_select alpha: no running allocations

1h ago gpu_select capella: no running allocations

1h ago gpu_select trying compute backend: dresden-holder

1h ago gpu_select start gpu_select

1h ago implement done implement in 0.6s

1h ago implement verifying the implementation

1h ago implement no implement step configured; using the code as it is

1h ago implement start implement

1h ago setup done setup in 0.6s

1h ago setup benchmarks: ['mnist-test']

1h ago setup hyperparameters: {'batch_size': 128, 'lr': 0.001, 'epochs': 2, 'seed': 0}

1h ago setup start setup

1h ago baseline done baseline in 0.4s

1h ago baseline baseline 'plain-cnn' will be trained alongside the idea on the same machine

1h ago baseline start baseline

1h ago environment done environment in 0.6s

1h ago environment project source: /Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn @ 83ac2db

1h ago environment working tree has uncommitted changes: ?? ../../../ ?? ../../../../../how_to/ml-experiment-run-idea.md ?? ../../../../../knowledge/ml-experiment-automation.md ?? ../../../../../skills/ml-experiment-automation/

1h ago environment start environment

1h ago idea done idea in 0.7s

1h ago idea hypothesis: BatchNorm raises MNIST test accuracy over the plain CNN at an equal training budget, without increasing wall-clock time meaningfully.

1h ago idea idea registered: BatchNorm in a small MNIST CNN

1h ago idea start idea