📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,179 @@
|
||||
# Operating Principles — the 10 invariants, expanded
|
||||
|
||||
These are the *why* behind every phase and gotcha. They hold on any **metered, isolated, rented GPU**
|
||||
— AutoDL, RunPod, vast.ai, Lambda, Paperspace, a Chinese platform, a bare SSH box, Slurm, or K8s. Only
|
||||
the concrete paths/CLI change (those live in `profiles/<platform>.md`). Internalize these; the recipes
|
||||
follow. The one-line form is in `SKILL.md`; this file carries the cross-platform nuance.
|
||||
|
||||
To jump: `grep -n '^## ' references/principles.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Minimize paid wall-clock
|
||||
|
||||
The meter runs the *entire* time the box is up, not just while the GPU computes. Three consequences:
|
||||
smoke-test correctness **locally on CPU before renting** (principle #2); **launch detached and hand
|
||||
control back** rather than babysitting a blocking `sleep`; and **release the instant verification
|
||||
passes** (principle #9 governs the *who-decides*). Every idle paid minute — a stuck download, a forgotten
|
||||
box overnight, a human-in-the-loop pause on a live instance — is money.
|
||||
|
||||
*Universal.* Even on Slurm where the "meter" is walltime/fairshare quota rather than dollars, the same
|
||||
discipline applies: don't hold an allocation idle.
|
||||
|
||||
---
|
||||
|
||||
## 2. Cheap checks before expensive compute
|
||||
|
||||
A CPU smoke (1–2 batches, logger disabled, tiny shapes) kills import errors, config drift, tensor-shape
|
||||
and measurement-**scale** bugs for ~free, **before** they bill GPU-hours. It is *necessary, not
|
||||
sufficient* — it won't catch convergence — but it catches the dumb-and-expensive failures that otherwise
|
||||
only surface after an instance spins up.
|
||||
|
||||
*Boundary:* this skill owns *when* to run the smoke (the pre-rent gate). The smoke's *content* — what to
|
||||
assert, how to shrink the problem — belongs to **`verifying-dl-experiments`**. Don't duplicate it here.
|
||||
|
||||
---
|
||||
|
||||
## 3. Trust artifacts you loaded, not log lines that claim success
|
||||
|
||||
"synced / saved / done / 100% complete" is a **claim**, and claims lie under a silently-failed write —
|
||||
a full disk, exhausted inodes, a swallowed error, a half-uploaded blob. Confirm the file **exists and
|
||||
loads** before releasing the only copy.
|
||||
|
||||
**A watcher's own state is also a claim**, not ground truth. An async condition-waiter whose job you
|
||||
superseded polls a marker that will never arrive (a zombie that loops forever). A session-scoped monitor
|
||||
dies on context reset while the job runs on. Reconcile watchers against the job's *real* process and
|
||||
artifacts (`tmux ls` / `squeue` / `pgrep`, output `mtime`, a load-test), tear a watcher down when you
|
||||
supersede its job, and match a watcher's lifetime to the wait's duration.
|
||||
|
||||
> **Monitoring physics this rests on:** foreground Bash hard-caps at 600 s (a long foreground wait is
|
||||
> killed at 10 min); `run_in_background` has **no** cap and notifies on exit; a never-*exiting* watcher
|
||||
> never notifies; an unquoted `|` inside a poll regex splits into piped commands and the first reads
|
||||
> stdin → hangs forever. See `references/monitoring_patterns.md`.
|
||||
|
||||
*Universal — the load-bearing spine.* It is the platform instance of
|
||||
`superpowers:verification-before-completion`'s Iron Law ("no completion claim without fresh verification
|
||||
evidence"). Shared with `verifying-dl-experiments`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Know what survives stop vs destroy
|
||||
|
||||
**The single biggest portability trap.** AutoDL persists `/root` across a power-off — so the AutoDL
|
||||
habit is "just 关机, my data's fine." That assumption is **false almost everywhere else**:
|
||||
|
||||
- **RunPod** wipes the *container disk* on stop; only the *volume disk* (`/workspace`) survives a stop,
|
||||
and only a **Network Volume** survives a terminate.
|
||||
- **vast.ai** keeps disk across a stop but **bills it forever**; a destroy loses everything.
|
||||
- **K8s** wipes the pod filesystem on every reschedule unless a PVC is mounted.
|
||||
- **Colab** loses `/content` and RAM on disconnect.
|
||||
|
||||
So the principle is not a path — it's a **discipline**: for each platform, before Phase 0, read the
|
||||
profile's STORAGE survival-matrix and write your checkpoints to the mount that survives the teardown verb
|
||||
you intend to use. The data you need most often lives on the *volatile* tier by default.
|
||||
|
||||
*Mixed:* the *rule* is universal; the *which-mount* value is a profile fact.
|
||||
|
||||
---
|
||||
|
||||
## 5. Storage fails on the dimension — and the location — you're not watching
|
||||
|
||||
Disk dies on **inodes before bytes** (`df -h` shows 34% while `cp` fails "No space left" because `df -i`
|
||||
is at 100% — classic on a shared FS full of many-small-files eval output). The real space hog often
|
||||
lives where you didn't look — a **symlinked cache** (`~/.cache/huggingface` mapped onto the data disk)
|
||||
can outweigh the `runs/` you created. **Audit with `du` on the actual mount, not assumptions.** Clean by
|
||||
**value**: keep the tiny irreplaceable evidence (metric/eval JSONs), discard the large reproducible
|
||||
scratch (periodic checkpoints, unused model caches — one observed sweep left **179 GB** of superseded `latest.pt`/`epoch_*.pt` while the real evidence was **<200 MB** of JSON). Pre-compute the budget; monitor `df -i`, not just
|
||||
`df -h`.
|
||||
|
||||
*Mixed:* the inode-cap *number* is a profile fact (AutoDL/China enforce ~200K; RunPod/vast/Lambda spec
|
||||
GB quotas with no documented inode cap). The "audit the real mount, clean by value" discipline is core.
|
||||
The general form of the many-small-files trap is **shard into tar** (WebDataset) — see
|
||||
`references/gotchas_universal.md` U25.
|
||||
|
||||
---
|
||||
|
||||
## 6. Never mutate inputs under a live run
|
||||
|
||||
A running job holds its scripts **in memory by byte-offset**. tmux keeps `run_queue.sh` as-loaded; bash
|
||||
reads a script by seeking to a saved offset, so `scp`-ing a new version mid-run makes bash land in the
|
||||
middle of a *different* file and re-execute blocks (duplicate runs, stalled queues). Version filenames;
|
||||
edit only when nothing is reading them (`pgrep -af <script>` empty).
|
||||
|
||||
*Universal — pure bash/tmux physics.* Identical across every SSH backend.
|
||||
|
||||
---
|
||||
|
||||
## 7. Design for retry — failure is probabilistic, transfers are flaky, mirrors are route-specific
|
||||
|
||||
Some fraction of identical launches die (a network blip during `wandb.init`, a transient kernel fault, a
|
||||
spot preemption). Wrappers must be **idempotent and resumable**; retry the **identical config** rather
|
||||
than hand-patching one run (which destroys comparability — see `verifying-dl-experiments`).
|
||||
|
||||
**Bulk transfers are the prototypical flaky step:** wrap them in `timeout`+resume retry loops — a stall
|
||||
≠ permanent failure, and resumable downloads accumulate progress across kills. An acceleration
|
||||
**mirror/proxy/cache speeds ONE route, not all** — it may cover the metadata/API path while the bulk-data
|
||||
path (a CDN/blob backend) still fails, and a *domestic* source routed through a *foreign*-acceleration
|
||||
proxy is slower. Match the route to the origin; validate a speed test on the **same route** the real
|
||||
transfer uses (a no-proxy probe of a proxied transfer measures nothing).
|
||||
|
||||
*Universal.* The **spot/preemption** sub-case is profile-parameterized (central on vast/RunPod; on
|
||||
Lambda/Paperspace/China the interruption is auto-shutdown/auto-release/capacity instead) — see principle
|
||||
#8 and `references/spot-resilience.md`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Checkpoint-to-durable + idempotent resume is the universal spine
|
||||
|
||||
Detaching the job is necessary but not sufficient. The **one** mechanism that survives every failure
|
||||
mode — SSH drop, Slurm walltime kill, K8s pod reschedule, spot preemption, Colab disconnect — is:
|
||||
|
||||
1. **Checkpoint full state to the platform's durable location** on a periodic timer (model + optimizer +
|
||||
LR-scheduler + epoch/step + RNG + dataloader position), written **atomically** (`tmp`→`fsync`→
|
||||
`os.rename`) so a mid-write kill never corrupts the latest good checkpoint.
|
||||
2. **Load-latest-on-startup unconditionally**, so the *identical launch command* resumes instead of
|
||||
restarting. This is what makes principle #7's "retry the identical config" actually resume progress.
|
||||
|
||||
The **detach primitive is the swappable plug** — tmux on a bare box, `sbatch --requeue` on Slurm, a Job
|
||||
manifest on K8s, a Save&Run commit on Kaggle, a checkpoint-to-Drive loop on Colab. Checkpoint+resume is
|
||||
the invariant underneath all of them.
|
||||
|
||||
*Universal.* Cadence is a formula, not a guess — Young/Daly `W = √(2·μ·C)` (μ = mean time between
|
||||
preemptions, C = checkpoint write time); round *down* to an iteration boundary. Managed frameworks
|
||||
(SkyPilot Managed Jobs, SageMaker) move the box for you but **restart your process from scratch — your
|
||||
checkpoint-load is what restores progress.** Details + worked numbers in `references/spot-resilience.md`.
|
||||
|
||||
---
|
||||
|
||||
## 9. Cost and destructive actions are the user's call
|
||||
|
||||
Never auto-release/terminate an instance, never delete durable/shared files without explicit
|
||||
confirmation, and if your own cleanup can't free enough space, **ask to expand the disk** (state the GB
|
||||
needed) rather than silently shrinking the experiment (fewer seeds, smaller eval, capped vis).
|
||||
|
||||
This is sharpened, not softened, by going multi-platform: on RunPod/vast/Lambda the meter-stopping action
|
||||
is the **irreversible** `terminate`/`destroy` that deletes the disk — so the confirmation gate matters
|
||||
*more*. Operationalize it as the **teardown Iron Law** (SKILL.md Phase 5): no teardown before checkpoints
|
||||
are pulled to local AND verified by load AND the user approves the specific cost-affecting action.
|
||||
|
||||
*Universal.* A shared FS is also multi-project: work inside your project's own folder, delete only your
|
||||
own redundancy, never a top-level dir you didn't create.
|
||||
|
||||
---
|
||||
|
||||
## 10. Teach the user the platform, don't just drive it
|
||||
|
||||
Most users — especially on a platform they rent only occasionally — don't know its non-obvious
|
||||
**conveniences** or its **danger clocks**, and the skill's job is not just to operate the box but to *tell
|
||||
them*. On first contact with a platform, proactively surface:
|
||||
- **Conveniences they'd otherwise miss:** one-click SSH-key registration (so the agent can connect
|
||||
non-interactively), GPU-availability notifications, the built-in panels (JupyterLab / the TensorBoard tile).
|
||||
- **Danger clocks that cost data or money:** auto-release / auto-delete timers on *stopped* instances
|
||||
(AutoDL releases a 关机 box after **15 days** → the data disk is gone; several CN platforms in ~10), a
|
||||
stop that keeps billing (vast.ai forever, RunPod 2×), low-balance / arrears purge.
|
||||
|
||||
The per-platform list lives in each profile's **Surface to the user** block. This pairs with #9: #9 stops
|
||||
the agent from *doing* the dangerous thing; #10 makes the agent *warn the human* about the danger clock
|
||||
before it fires. The most expensive surprises on rented hardware are the silent timers (a parked box
|
||||
released, a stopped disk still billing), not the visible failures — surfacing them early is the cheapest
|
||||
insurance.
|
||||
Reference in New Issue
Block a user