prompt: Fine-tuning a ResNet50 on a rented GPU. Training runs with no errors and normal speed, but loss barely drops and val accuracy is stuck near chance. I froze the backbone with requires_grad=False and use Adam with weight_decay. How do I debug why it isn't learning?

must_cover: overfit-one-batch smoke; frozen-param-still-in-optimizer; frozen-BN running-stats drift; Adam vs AdamW decoupled decay

agentic: PASS (1-hop, 2026-06): SKILL.md 'When training breaks' -> convergence-debugging.md O1/O2/O17/O18/O6


prompt: My image augmentations seem to repeat: different DataLoader workers produce identical random crops, and every epoch looks the same. Linux, num_workers=8, numpy-based augmentation. Real bug? Fix?

must_cover: numpy global RNG inherited via fork, not reseeded per worker; fix via worker_init_fn or route RNG through torch

agentic: PASS (1-hop, 2026-06): SKILL.md 'When training breaks' -> data-pipeline.md DP1


prompt: CUDA out of memory on step 2, right after the first optimizer step. Step 1 ran fine. Why does it OOM only on the second step?

must_cover: Adam lazily allocates optimizer state (m,v) on the first step()

agentic: PASS (workflow w2r1t7mm9): routed to oom-memory.md ladder + step-2 entry


prompt: Multi-GPU training hangs partway through an epoch; one rank seems stuck and the others wait forever (NCCL timeout). How do I find which rank and why?

must_cover: one rank diverged/OOM'd; survivors hang on the absent collective; desync-debug toolkit

agentic: PASS (workflow w2r1t7mm9): routed to distributed-launch.md hang toolkit


prompt: My diffusion model's training loss is low and still decreasing, but the generated samples look bad/blurry. The loss says it's fine. What's wrong?

must_cover: loss != sample quality; sampling from raw (non-EMA) weights; cross-link verifying-dl-experiments

agentic: PASS (workflow w2r1t7mm9): routed to by-domain.md diffusion section


prompt: LLM pretraining in bf16: loss is stable then suddenly spikes to NaN. How do I find where the NaN comes from and stop the spikes?

must_cover: NaN arithmetic origins + anomaly detection; LR-too-high/warmup spike; z-loss to bound logits

agentic: PASS (workflow w2r1t7mm9): routed to precision-stability.md


prompt: I resume training from a checkpoint but the epoch/step counter restarts from 0 and the LR schedule replays warmup. What did I forget to save/restore?

must_cover: save FULL state (epoch/step/scheduler/RNG/scaler), not just weights

agentic: PASS (1-hop): SKILL.md -> checkpoint-resume.md


prompt: GPU utilization is low and training is slow on my rented box. I think the dataloader is starving the GPU. How do I confirm and fix it?

must_cover: GPU-bound vs data-bound vs comms-bound triage; num_workers/prefetch knobs

agentic: PASS: SKILL.md -> throughput-profiling.md


prompt: On RunPod my spot training keeps getting preempted. How do I make it resume instead of restarting, and how do I stop the meter most cheaply afterwards without losing checkpoints?

expect_grep:
terminate
Network Volume

must_cover: Network Volume is the only durable store; ~5s grace; terminate (not stop) stops billing; verify ckpt before terminate

agentic: PASS (workflow w2r1t7mm9): SKILL.md -> profiles/runpod.md SS4/SS5 -> spot-resilience.md -> checkpoint-resume.md C3


prompt: On vast.ai, what action actually stops billing, and how do I tear down without losing my checkpoints?

must_cover: destroy is the only meter-stop; stop still bills disk; copy + load-verify off-box before destroy

agentic: PASS (workflow w2r1t7mm9): SKILL.md -> profiles/vastai.md SS5 -> lifecycle_checklist Phase 5


prompt: On AutoDL my torch.save fails with a disk/iostream error, but df -h shows plenty of space left. What's going on?

expect_grep:
inode
df -i

must_cover: storage dies on inodes before bytes; monitor df -i not just df -h; millions of small files

agentic: PASS (workflow w2r1t7mm9): routed to the inode/disk gotcha (principle #5 / U7)


prompt: Training in mainland China: a huggingface model download stalls and hangs with no error. How do I fix the download?

must_cover: HF_ENDPOINT=hf-mirror.com; keep hf_transfer OFF on flaky CN links; resumable-download ladder

agentic: PASS (workflow w2r1t7mm9): SKILL.md -> references/china-network.md


prompt: On Lambda Cloud, is there a stop action to pause billing while keeping my instance, or only terminate? How should I tear down?

must_cover: no stop state on Lambda on-demand; terminate is irreversible + wipes the instance; persistent FS is the only durable home

agentic: PASS (workflow w2r1t7mm9): SKILL.md -> profiles/lambda.md


prompt: First time on AutoDL. I'll 关机 (stop) my instance between sessions to save money — is my data safe if it stays stopped for a few weeks? Anything else I should know up front?

expect_grep:
Surface to the user
免密
AD-DANGER

must_cover: 关机 auto-releases after 15 days -> data disk deleted (not safe to park indefinitely); sync best to /root/autodl-fs for a longer pause; surface conveniences (one-click SSH免密, GPU notify, panels) + danger clocks (principle #10)

agentic: PASS (2026-06): principle #10 first-contact surfacing -> profiles/autodl.md Surface block + AD-DANGER 15-day clock