ML Experiment Automation

idea → setup → compute → train → eval → report auto-refresh 15s

e2e-local-006

completed idea: BatchNorm in a small MNIST CNN updated 1h ago
Idea Environment Baseline Setup Implement GPUs Venv Data Tests Train Monitor Eval Report

orchestrator host: MBP-Artem-2.fritz.box

Compute

backend
local
label
local
source
local
gpus
0
gpu_type
cpu
vram_gb
0
workspace
/Users/makartkar/.cache/mlexp/local
cpus
11
host
localhost

Setup

train_started_at
1786966918.013539

Metrics

loss
0.0538
val_loss
0.0433
val_acc
0.9876
variant
idea
test_loss
0.04103
test_acc
0.9867
test_error_pct
1.33
eval_seconds
0.45
device
mps
n_test
10000

Baseline metrics

variant
baseline
test_loss
0.04256
test_acc
0.9856
test_error_pct
1.44
eval_seconds
0.22
device
mps
n_test
10000

Artifacts

dashboard_url
https://main.research.makartkar.com/runs/e2e-local-006
repo_commit
83ac2db
repo_local_path
/Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn
implementation_diffstat
(clean tree)
compute_target
local
compute_source
local
venv_path
/Users/makartkar/.cache/mlexp/local/projects/mnist-cnn/.venv
project_path
/Users/makartkar/.cache/mlexp/local/worktrees/mnist-cnn-e2e-local-006
python_version
Python 3.12.9
data_dir
/Users/makartkar/.cache/mlexp/local/.data/mnist
data_prep_log
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/data_prep.log
data_cache
built
unit_tests_log
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/unit_tests.log
smoke_run_log
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/smoke_run.log
train_log
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/train_idea.log
baseline_train_log
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/train_baseline.log
monitor_checks
1
eval_idea_log
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/eval_idea.log
eval_metrics_file
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/results/idea_metrics.json
eval_baseline_log
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/eval_baseline.log
baseline_metrics_file
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/results/baseline_metrics.json
comparison
test_loss: 0.04103 vs baseline 0.04256 (-0.00153) | test_acc: 0.9867 vs baseline 0.9856 (+0.0011) | test_error_pct: 1.33 vs baseline 1.44 (-0.11) | eval_seconds: 0.45 vs baseline 0.22 (+0.23) | n_test: 10000 vs baseline 10000 (+0)
local_train.log
/Users/makartkar/.local/state/mlexp/runs/e2e-local-006/train.log
local_baseline_train.log
/Users/makartkar/.local/state/mlexp/runs/e2e-local-006/baseline_train.log
local_eval_idea.log
/Users/makartkar/.local/state/mlexp/runs/e2e-local-006/eval_idea.log
data_cache_path
/Users/makartkar/.cache/mlexp/local/.data/mnist
run_dir
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006
results_dir
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/results
setup_json
/Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/setup.json
local_run_dir
/Users/makartkar/.local/state/mlexp/runs/e2e-local-006
report_md
/Users/makartkar/.local/state/mlexp/runs/e2e-local-006/report.md
report_json
/Users/makartkar/.local/state/mlexp/runs/e2e-local-006/report.json

Event log (68)

1h ago report done report in 0.8s
1h ago report report written to /Users/makartkar/.local/state/mlexp/runs/e2e-local-006/report.md
1h ago report start report
1h ago evaluate done evaluate in 14.2s
1h ago evaluate n_test: 10000 vs baseline 10000 (+0)
1h ago evaluate eval_seconds: 0.45 vs baseline 0.22 (+0.23)
1h ago evaluate test_error_pct: 1.33 vs baseline 1.44 (-0.11)
1h ago evaluate test_acc: 0.9867 vs baseline 0.9856 (+0.0011)
1h ago evaluate test_loss: 0.04103 vs baseline 0.04256 (-0.00153)
1h ago evaluate eval_baseline metrics: {'variant': 'baseline', 'test_loss': 0.04256, 'test_acc': 0.9856, 'test_error_pct': 1.44, 'eval_seconds': 0.22, 'device': 'mps', 'n_test': 10000}
1h ago evaluate launched 'eval_baseline' (pid 96655) -> /Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/eval_baseline.log
1h ago evaluate eval_idea metrics: {'variant': 'idea', 'test_loss': 0.04103, 'test_acc': 0.9867, 'test_error_pct': 1.33, 'eval_seconds': 0.45, 'device': 'mps', 'n_test': 10000}
1h ago evaluate 'eval_idea' still running (0s): RUN_MARKER step=eval_idea start_epoch=1786966960 host=MBP-Artem-2.fritz.box
1h ago evaluate launched 'eval_idea' (pid 96613) -> /Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/eval_idea.log
1h ago evaluate start evaluate
1h ago monitor done monitor in 41.1s
1h ago monitor all training jobs finished successfully
1h ago monitor train_baseline exited with rc=0
1h ago monitor train_idea exited with rc=0
1h ago monitor supervising 2 job(s); check every 40s, hard limit 20 min
1h ago monitor start monitor
1h ago train done train in 2.6s
1h ago train baseline training started -> /Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/train_baseline.log
1h ago train training started -> /Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/train_idea.log
1h ago train start train
1h ago testing done testing in 23.5s
1h ago testing smoke_run passed
1h ago testing 'smoke_run' still running (0s): RUN_MARKER step=smoke_run start_epoch=1786966903 host=MBP-Artem-2.fritz.box
1h ago testing launched 'smoke_run' (pid 96497) -> /Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/smoke_run.log
1h ago testing unit_tests passed
1h ago testing 'unit_tests' still running (0s): RUN_MARKER step=unit_tests start_epoch=1786966891 host=MBP-Artem-2.fritz.box
1h ago testing launched 'unit_tests' (pid 96457) -> /Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/unit_tests.log
1h ago testing start testing
1h ago data_prep done data_prep in 11.9s
1h ago data_prep data prepared
1h ago data_prep 'data_prep' still running (0s): downloading https://storage.googleapis.com/cvdf-datasets/mnist/train-images-idx3-ubyte.gz
1h ago data_prep launched 'data_prep' (pid 96417) -> /Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/data_prep.log
1h ago data_prep start data_prep
1h ago venv_prep done venv_prep in 8.3s
1h ago venv_prep venv created and dependencies installed
1h ago venv_prep launched 'venv_prep' (pid 96370) -> /Users/makartkar/.cache/mlexp/local/runs/e2e-local-006/venv_prep.log
1h ago venv_prep materialising run tree /Users/makartkar/.cache/mlexp/local/worktrees/mnist-cnn-e2e-local-006
1h ago venv_prep syncing /Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn -> local:/Users/makartkar/.cache/mlexp/local/projects/mnist-cnn
1h ago venv_prep start venv_prep
1h ago gpu_select done gpu_select in 0.8s
1h ago gpu_select setup adjusted: per-device batch 128 x 1 GPU(s) -> global batch 128
1h ago gpu_select using the local machine (CPU-sized experiment)
1h ago gpu_select trying compute backend: local
1h ago gpu_select start gpu_select
1h ago implement done implement in 0.6s
1h ago implement verifying the implementation
1h ago implement no implement step configured; using the code as it is
1h ago implement start implement
1h ago setup done setup in 0.6s
1h ago setup benchmarks: ['mnist-test']
1h ago setup hyperparameters: {'batch_size': 128, 'lr': 0.001, 'epochs': 2, 'seed': 0}
1h ago setup start setup
1h ago baseline done baseline in 0.6s
1h ago baseline baseline 'plain-cnn' will be trained alongside the idea
1h ago baseline start baseline
1h ago environment done environment in 0.6s
1h ago environment project source: /Users/makartkar/.agents/.claude/worktrees/ml-experiment-automation-116c14/clis/mlexp/examples/mnist_cnn/mnist-cnn @ 83ac2db
1h ago environment working tree has uncommitted changes: ?? ../../../
1h ago environment start environment
1h ago idea done idea in 0.7s
1h ago idea hypothesis: BatchNorm raises MNIST test accuracy over the plain CNN at an equal training budget, without increasing wall-clock time meaningfully.
1h ago idea idea registered: BatchNorm in a small MNIST CNN
1h ago idea start idea