DFlash Qwen3-0.6B — batch 24, 10 epochs

success diff on top of dflash-qwen3-0.6b-1024-b12-6ep setup qwen3-0.6b-perfectblend-1024 on cadence (H100 box) · updated 59d 0h ago

The 6-epoch run was underfitting, so this doubles per-GPU batch to 24 (global 192), extends to 10 epochs, and sqrt-scales the learning rate to 8.5e-4.

Workflow

live

Experiment description success finished 59d 0h ago #

What this experiment is: a baseline of its own, or a diff on top of an existing run. Carries the repository, the baseline it is measured against, and the setup it reuses.

repository https://github.com/MakArtKar/SpecForge

branchdflash-fast-setup

diff vs baseline

note9 commits rebuilt on main; opened as PR #15.

baseline DFlash Qwen3-0.6B — 1024 len, batch 12, 6 epochs dflash-qwen3-0.6b-1024-b12-6ep

setup Qwen3-0.6B PerfectBlend 1024 qwen3-0.6b-perfectblend-1024

What changed vs baseline

batch (per GPU / global): 12 / 96 -> 24 / 192 epochs: 6 -> 10 learning rate: 6e-4 -> 8.5e-4 (sqrt-scaled for the 2x batch) eval / save interval: 500 -> 250 (fewer total steps)

GPU server success finished 59d 0h ago #

Which machine or cluster runs the training, and what hardware it provides. Chosen manually for now.

server	cadence (H100 box)
kind	ssh
nodes	1
GPUs	8 × H100 80GB HBM3 per node
CPUs	96 per node
RAM	1.5 TB per node
disk	531 GB
notes	Chosen over the 8xA100-40GB box: the 787 GB hidden-state cache only fits in this one's 1 TB tmpfs.

Experiment preparation success #

Get code, environment, and data onto the chosen server.

Copy code to server success finished 59d 0h ago #

cadence ~/working_dir/SpecForge — branch dflash-fast-setup

git clone https://github.com/MakArtKar/SpecForge ~/working_dir/SpecForge && git checkout dflash-fast-setup

Create venv and install prerequisites success finished 59d 0h ago #

cadence ~/cadence-env — training venv, cpython 3.12.11

cadence ~/sglang-venv — separate SGLang venv — CUDA 13 compatibility trick keeps it off the training env

Download data and preprocess data cache success finished 59d 0h ago #

Regenerate assistant answers with the target model

Regenerate assistant answers with the target model 100.0%

100,000 / 100,000 samples ETA —

cadence /mnt/ramcache 787 GB — hidden-state cache, tmpfs — rebuilt from the S3 jsonl in ~20 min if lost

Stage 1 regenerated 100k PerfectBlend samples, stage 2 captured superset hidden states (3 aux layers + final). At 1024 tokens the cache is 839 GB by the sizing table, which is why 1280+ was ruled out: it would not fit the 1 TB tmpfs.

Other preparation success finished 59d 0h ago #

sudo mount -t tmpfs -o size=1000G,noatime tmpfs /mnt/ramcache

1 TB of the box's 1574 GB RAM, so the cache never touches the 531 GB root disk

AWS credentials were missing on this box, so the stage-2 jsonl upload and the stage-5 result upload both failed silently with 'Unable to locate credentials'. Copied them over and re-uploaded; check this first on any fresh box.

Sanity checks success #

Cheap checks that must pass before spending GPU hours on a full run.

Unit and integration tests success finished 59d 0h ago #

Cover the newly added code. For a diff on top of a baseline, run the affected tests rather than the full suite.

875 passed / 48 failed / 11 collection errors — set-diffed against main

success full suite (local Mac)

875 passed 48 failed 0 skipped of 934

pytest tests/

main is 855 passed / 49 failed / 11 errors. Set-diffing the failure names shows no new failures, and one pre-existing failure fixed (test_recipe_readme_names_every_typed_config_field). The rest are the known CUDA/triton-bound and parser failures that need a GPU.

Smoke test success finished 59d 0h ago #

A handful of training steps on a small setup with debug hyperparameters. Must finish within about 7 minutes.

CUDA_VISIBLE_DEVICES=7 python -m sglang.launch_server --model-path Qwen/Qwen3-8B \
  --dtype bfloat16 --port 30000 --mem-fraction-static 0.85

~/dflash-run/sglang-smoke.log

Booted one SGLang server on GPU 7 and drove a handful of steps to confirm the DFLASH path loads and the JIT kernels compile before committing all 8 GPUs to a full run.

Full training success #

The real training run and its health over time.

Launch and watch first steps success finished 59d 0h ago #

Start the full run and watch the first training steps to confirm it behaves like the smoke test did.

Watched the first steps: 61.2 GB/GPU (75% of 80 GB), 0.40 s/step, 479 samples/s, 606 W. Healthy, with 20 GB of headroom — batch 32 would have sat at 99%.

GPU memory 61.2 GB/GPU

step time 0.4 s

throughput 479 samples/s

Scheduled monitoring success finished 59d 0h ago #

Every 30 minutes, check the run is alive and its metrics are healthy. Restart it if either check fails.

makartkar/diffusion_self_spec/qwen3-0.6b-dflash-1024-b24-10ep finished synced just now sync error

val/acc0.4031

train/lr3.40e-08

val/loss2.238

train/acc0.4502

train/loss1.903

test/gsm8k/samples1,024

test/checkpoint_step4,750

healthy/train/updates2.00e-07

test/mt-bench/samples1,024

test/humaneval/samples1,024

healthy/train/grad_norm0.5656

healthy/val/step_time_s1.713

healthy/train/max_weight2.618

healthy/train/min_weight-0.9819

test/gsm8k/output_tokens282,081

healthy/train/step_time_s0.3716

val/greedy_accepted_length5.034

test/mt-bench/output_tokens293,246

test/gsm8k/spec_verify_count57,950

test/gsm8k/tokens_per_second348

test/humaneval/output_tokens301,580

train/greedy_accepted_length5.552

test/mt-bench/spec_verify_count111,918

test/mt-bench/tokens_per_second189.3

test/humaneval/spec_verify_count75,360

test/humaneval/tokens_per_second284.9

test/gsm8k/greedy_accepted_length5.154

test/mt-bench/greedy_accepted_length2.924

test/humaneval/greedy_accepted_length4.066

healthy/val/throughput_samples_per_second597.7

healthy/train/throughput_samples_per_second516.6

Embedded wandb charts

Training — 10 epochs, global batch 192

Training — 10 epochs, global batch 192 100.0%

10 / 10 epochs ETA —

val/greedy_accepted_length 5.037 +0.267 vs baseline

Best checkpoint at step 4,750: val/greedy_accepted_length 5.037, up 5.6% from the 6-epoch run's 4.770. Validation was still rising at the end, so this is data-limited rather than converged.

Full evaluation success #

The evaluation run, monitored the same way as training.

Launch and watch first steps success finished 59d 0h ago #

Start the evaluation and watch the first steps to confirm it is producing sane numbers.

Stage 4 launches one DFLASH server per GPU and runs all three benchmarks concurrently, so evaluation costs the slowest single dataset instead of their sum. Only 3 of 8 GPUs are usable: 'specforge benchmark' has no prompt-offset flag, so one dataset cannot be sharded across servers. Raising --concurrency would fill the other 5 but would break comparability with the concurrency-1 protocol the 8B numbers used.

Scheduled monitoring success finished 59d 0h ago #

Every 30 minutes, check the evaluation is alive and healthy. Restart it if either check fails.

Serving eval — block 16, greedy, 1024 samples each

Serving eval — block 16, greedy, 1024 samples each 100.0%

3 / 3 benchmarks ETA —

GSM8K acceptance 5.154

HumanEval acceptance 4.066 +0.111 vs baseline

MT-Bench acceptance 2.924

GSM8K throughput 348 tok/s

HumanEval throughput 284.9 tok/s

MT-Bench throughput 189.3 tok/s

Caveat

These numbers are not comparable to the 8B runs or to the paper: different draft model, different sequence length, and a regenerated dataset.

Log

59d 0h ago evaluation.monitor Pipeline complete: regenerate to eval in ~2 h wall clock, versus ~14 h for the 8B run
59d 0h ago evaluation.launch Stage 4 hung after benchmarks finished: bare 'wait' also waited on the SGLang servers, which never exit. Patched to wait on benchmark PIDs only.
59d 0h ago training.launch Training live at 61.2 GB/GPU, 0.40 s/step, 479 samples/s
59d 0h ago preparation.data Re-uploaded regenerated jsonl and all three result JSONs to S3
59d 0h ago preparation.data S3 uploads failed silently: AWS credentials were never on this box
59d 0h ago preparation.other Mounted 1 TB tmpfs at /mnt/ramcache