📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,365 @@
|
||||
---
|
||||
platform: paperspace # Paperspace (now under DigitalOcean): Gradient Notebooks + Core/Machines
|
||||
kind: cloud-api # web console + pspace/gradient CLI/SDK + REST; Core machines also reachable by SSH
|
||||
meter_stop_verb: shut-down # shut-down/power-off stops COMPUTE; only destroy/delete stops storage + IP
|
||||
meter_stop_irreversible: false # a stop is reversible; destroy/delete IS irreversible (loses block storage)
|
||||
detach_primitive: tmux # on Core VMs; Notebooks have no clean SSH-daemon story (Jupyter kernel + hard auto-shutdown ceiling)
|
||||
spot_available: false # no AWS-style spot/preemptible with a 2-min warning
|
||||
spot_grace: n/a # interruption is capacity-at-launch + a deterministic auto-shutdown clock, not eviction
|
||||
shared_fs: true # Gradient /storage is team-shared per storage region/cluster
|
||||
inode_cap: none # no documented inode cap on either /storage or Core block storage
|
||||
free_egress: true # no documented ingress/egress fee
|
||||
china_mirror_needed: false # US/global cloud, direct egress; no platform-provided proxy
|
||||
host_driver_cuda_max: "host-dependent" # ML-in-a-Box / template ships the CUDA+driver stack (often lagging)
|
||||
local_nvme: host-dependent # ephemeral workspace on Notebooks; block storage on Core
|
||||
---
|
||||
|
||||
# Paperspace (DigitalOcean) — platform profile
|
||||
|
||||
One-line purpose: substrate for running detached GPU jobs on Paperspace Gradient (managed Jupyter
|
||||
notebooks/deployments) and Paperspace Core (raw Linux VMs, "Machines") — what stops the meter, what
|
||||
survives a stop vs a destroy, and the auto-shutdown clock that ends every long run. Universal gotchas are
|
||||
NOT repeated here — see `references/gotchas_universal.md`.
|
||||
|
||||
> **Surface to the user up front (principle #10):** ⚠️ Danger clocks — an **auto-shutdown timer ends every Notebook/Core run** (set it consciously; Gradient free notebooks hard-cap at 6 h); **snapshots / block storage keep billing after a machine is destroyed** (orphan bleed). Heads-up — the **Gradient CLI/API was deprecated 15 Jul 2024** (pin `gradient<3.0`; the three-CLI mess, §1).
|
||||
|
||||
To jump: `grep -in '<keyword>' profiles/paperspace.md`.
|
||||
|
||||
## Table of contents
|
||||
1. LAUNCH — Gradient vs Core, the env contract, the three-CLI mess
|
||||
2. STORAGE MODEL — survival matrix, the stop-keeps-disk rule, pip-doesn't-persist
|
||||
3. NETWORK — public IP (static vs dynamic), ports, SSH flavor
|
||||
4. SPOT / INTERRUPTION + RESUME — the auto-shutdown clock, not spot
|
||||
5. TEARDOWN / BILLING — what actually stops the meter (the trap)
|
||||
6. DAEMON TOOL — tmux on Core; why Notebooks resist a daemon
|
||||
7. TOP GOTCHAS — `PS1`–`PS13`, platform-pinned + platform-specific debugging
|
||||
8. SCRIPT OVERRIDES — values for the `scripts/` templates
|
||||
|
||||
---
|
||||
|
||||
## 1. LAUNCH
|
||||
|
||||
Two product families, with opposite operating models:
|
||||
|
||||
- **Gradient** — the managed layer. **Notebooks** are a web Jupyter IDE on a shared persistent store;
|
||||
**Deployments** serve a container behind a REST endpoint (bring a Docker image `<user>/img:tag`);
|
||||
**Workflows** run GPU-backed DAG automation. Entry: web console, the CLI/SDK, or REST.
|
||||
- **Core / Machines** — raw Linux/Windows VMs with a persistent block disk, full root/SSH. OS templates
|
||||
include **ML-in-a-Box** (preinstalled CUDA + PyTorch/TensorFlow/RAPIDS/Jupyter; **terminal/SSH-only**,
|
||||
home `/home/paperspace`, shell `/bin/bash`). **Ubuntu 22.04 is required for H100 and recommended for
|
||||
A100; Ubuntu 20.04 is recommended for any other machine type** (verified github.com/Paperspace/ml-in-a-box
|
||||
README + DO machines docs 2026-06). This is the family that maps cleanly onto the AutoDL
|
||||
tmux-resilient-training pattern.
|
||||
|
||||
**Env contract.** The chosen image/template IS the Python env — do NOT `conda create` on a rental
|
||||
(principle: the prebuilt base is the env). On Core, run inside **ML-in-a-Box** directly; on Gradient
|
||||
Deployments, the env is the Docker image specified at create time. Because a *destroy* wipes the box, the
|
||||
durable analog of the env is a Docker image plus a `requirements.txt`/lock file kept off-box, so a recreate
|
||||
reproduces it. **On Notebooks, a plain `pip install` does NOT survive a restart** (writes to
|
||||
`/usr/local/lib`, ephemeral) — see §2 / `PS3`.
|
||||
|
||||
**The three-CLI mess (gates ALL automation).** The tooling fragmented across the DigitalOcean acquisition;
|
||||
the draft's "migrate to the current API/CLI" understates the trap (verified github.com/Paperspace 2026-06):
|
||||
- The **legacy Gradient REST API endpoints were deprecated 15 Jul 2024** — stale calls 404 or no-op.
|
||||
- **`gradient-cli` v2 is deprecated**; pin `pip install "gradient<3.0"` only to keep *old* scripts alive.
|
||||
- **`gradient-python` (github.com/digitalocean/gradient-python) is NOT the orchestration CLI** — it is the
|
||||
new DigitalOcean *Gradient AI / GenAI inference* SDK. **Name collision** — do not install it expecting
|
||||
notebook/machine control.
|
||||
- The **recommended tool for new work is the streamlined `pspace` CLI** (github.com/Paperspace/cli,
|
||||
releases ongoing into 2026; e.g. `pspace public-ip release <ip>`). Pin and verify the CLI binary +
|
||||
version in any automation; do not assume `gradient` ⇒ `pspace` command parity.
|
||||
|
||||
→ **verify:** `ssh <core-alias> 'python -c "import torch;print(torch.cuda.is_available())"'` on Core, or a
|
||||
`print(torch.cuda.is_available())` cell in a Notebook.
|
||||
|
||||
---
|
||||
|
||||
## 2. STORAGE MODEL *(survival matrix — principle #4)*
|
||||
|
||||
The defining fact: a **stop/shut-down keeps the disk** — Paperspace is one of the few profiles here that
|
||||
behaves like AutoDL's 关机 in this respect. Only **destroy/delete** removes storage.
|
||||
|
||||
**Gradient Notebooks** — `/storage` and `/notebooks` are **separate branches from `/`, NOT nested**
|
||||
(verified DO notebooks/details/storage-architecture 2026-06):
|
||||
- `/storage` — **shared persistent**, team-wide, scoped to a **storage region/cluster**. Survives stop.
|
||||
(Team-shared ⇒ never write secrets here — see §7 / `references/gotchas_universal.md`.)
|
||||
- `/notebooks` — **per-notebook persistent**, managed via the console File Manager. Survives stop.
|
||||
- everything else — **ephemeral workspace** (incl. `/usr/local/lib` where `pip` lands), wiped on stop.
|
||||
|
||||
**Core machines** — block storage **50 GB–2 TB**, persists across a stop; **expansion is one-way**
|
||||
("increasing block storage expands the filesystem and is not reversible"). Region-locked: storage and
|
||||
custom templates must be used in the **same datacenter**. **Snapshots** are a separate billed resource
|
||||
(`$0.29/GB/mo`, default policy is **"Never" / 0 stored** — they bill only if manually enabled, and a
|
||||
snapshot **survives a machine destroy**, so an orphaned snapshot keeps charging — see `PS9`).
|
||||
|
||||
| Tier | Path | Survives STOP? | Survives DESTROY/DELETE? | Cap / note |
|
||||
|---|---|---|---|---|
|
||||
| Notebook shared persistent | `/storage` | yes | yes (separate resource) | team-shared per region/cluster; billed until deleted |
|
||||
| Notebook per-notebook | `/notebooks` | yes | no (dies with the notebook) | per-notebook persistent; console File Manager |
|
||||
| Notebook workspace | everything else (incl. `/usr/local/lib`) | **no** | no | ephemeral; wiped on stop; `pip` lands here |
|
||||
| Core block storage | machine root + block vol | yes | **no** | 50 GB–2 TB; expansion irreversible; region-locked |
|
||||
| Core snapshot | (separate resource) | yes | **yes** (orphan-bills!) | `$0.29/GB/mo`; default policy Never/0; survives machine destroy |
|
||||
|
||||
**Mount checkpoints MUST go to (for the §5 teardown verb):** on Notebooks, `/storage` (cross-stop,
|
||||
cross-delete-of-the-notebook) — `/notebooks` dies if the notebook itself is deleted. On Core, the block
|
||||
disk survives a stop, but a *destroy* wipes it, so the Iron-Law pull-to-local before destroy still applies.
|
||||
No documented inode cap on either tier; still monitor `df -i` (universal, U7 / principle #5).
|
||||
|
||||
---
|
||||
|
||||
## 3. NETWORK
|
||||
|
||||
- **Egress.** Direct and unproxied to HF/GitHub/PyPI; no `network_turbo`-style accelerator and no
|
||||
documented egress fee. China-mirror relevance is **N/A as a platform feature** — relevant only when
|
||||
operating from inside China and supplying a private mirror (then `references/china-network.md`).
|
||||
- **Public IP.** Core machines are reached by **public IP**, of two kinds (verified DO
|
||||
machines/how-to/manage-public-ips 2026-06):
|
||||
- **Static** — "the same IP address every time it powers on … remains in your account until you delete
|
||||
it." Use it to pin stable SSH/endpoint addressing. **Billed until deleted** — *including while the
|
||||
machine is powered off* (see §5 / `PS6`). API/CLI can create/release a **static** IP but **cannot add a
|
||||
dynamic IP to an existing machine** — dynamic must be requested at machine-creation time.
|
||||
- **Dynamic** — "assigned automatically when a machine powers on and deleted when it powers off"; a **new
|
||||
IP on every start**, so a hard-coded SSH alias breaks after a restart. **Charged only while the machine
|
||||
runs** (auto-released on power-off → no idle IP cost).
|
||||
A machine with **no public IP** is internet-isolated (and avoids the IP charge). **Private networks**
|
||||
give team-isolated pools.
|
||||
- **Ports / services.** Firewall is self-managed — open ports to expose services. Tunnel Jupyter (8888) /
|
||||
TensorBoard (6006) over SSH on Core:
|
||||
`ssh -L 8888:localhost:8888 -L 6006:localhost:6006 paperspace@<machine-ip>`
|
||||
(placeholder host — substitute the machine's real IP/static address). In a Gradient Notebook, launch
|
||||
TensorBoard in-Jupyter and write logs under `/storage` (or they vanish on stop).
|
||||
- **SSH flavor.** Core = a standard Linux VM → full `ssh`/`scp`/`rsync` (ML-in-a-Box default user
|
||||
`paperspace`). Gradient Notebooks expose a **Jupyter sandbox**, not a clean persistent SSH daemon —
|
||||
there is no stable SSH-daemon story for a multi-day unattended run on a Notebook.
|
||||
|
||||
---
|
||||
|
||||
## 4. SPOT / INTERRUPTION + RESUME *(principle #7/#8)*
|
||||
|
||||
**No AWS-style spot/preemptible tier** with a 2-minute interruption warning. The two interruption modes are
|
||||
different in kind and BOTH are deterministic, not random eviction:
|
||||
|
||||
1. **Capacity-at-launch.** The desired GPU type may be unavailable when launching — a *launch-time*
|
||||
availability problem, not a runtime eviction. On free notebooks this surfaces as **"out of capacity" /
|
||||
the notebook sits "pending" in queue for the next free machine** (verified DO notebooks/how-to docs
|
||||
2026-06). Build **retry-launch-until-available** logic, not a 2-minute-grace flush handler; for assured
|
||||
access, a paid instance type bypasses the free queue.
|
||||
2. **Auto-shutdown clock — the hard ceiling on any long run.** The timer is the real killer:
|
||||
- **Gradient free** notebooks hard-stop at a **6-hour** maximum auto-shutdown (cannot be raised).
|
||||
- **Paid notebooks** default to **12-hour** auto-shutdown; range **1 hour – 1 week**.
|
||||
- **Core** machines allow a configurable **1 hour – 1 week** auto-shutdown.
|
||||
- **Trap (Core/Linux):** Core Linux auto-shutdown is **wall-clock, not idle-based** — "Linux machines
|
||||
shut down regardless of whether any users are connected" (only Windows waits for idle). An active
|
||||
SSH/tmux session does **not** keep a Linux Core machine alive past the timer (verified DO
|
||||
machines/how-to/manage-auto-shutdown 2026-06).
|
||||
- **Trap (API):** auto-shutdown **cannot be enabled/disabled via API or CLI on an existing machine** —
|
||||
"you can only manage the auto-shutdown feature via the Paperspace console" (same source). Set it
|
||||
deliberately at create time / in the console.
|
||||
|
||||
The window is deterministic, so plan around it: a tmux session inside a Notebook **still dies at the
|
||||
timeout** (§6). **Resume hook:** checkpoint full state to `/storage` (Notebooks) or the block disk
|
||||
(Core) *before* the auto-shutdown window, then restart and load-latest-on-startup unconditionally.
|
||||
Because the clock is known in advance, cadence can be planned rather than guessed — but the
|
||||
load-latest-on-startup spine (principle #8) is what makes the restart idempotent. Young/Daly cadence
|
||||
formula → `references/spot-resilience.md`.
|
||||
|
||||
---
|
||||
|
||||
## 5. TEARDOWN / BILLING *(principle #9 + the Iron Law — the most error-prone section)*
|
||||
|
||||
Per-hour billing (verified DO products/paperspace/pricing 2026-06). **A shut-down/power-off STOPS the
|
||||
compute (GPU) meter** while disk persists — this is the AutoDL-like part. **But it does NOT stop every
|
||||
meter.**
|
||||
|
||||
- **What a stop still bills (the trap):** "When a Paperspace machine is powered off, attached **storage**,
|
||||
**public IP addresses**, and other **add-ons** continue to be billed on an hourly basis until you destroy
|
||||
those resources." Gradient `/storage` over the plan allowance and Core block storage both keep charging
|
||||
while the machine is off.
|
||||
- **The monthly-cap softener (new fact):** non-GPU resources (storage, public IP, snapshots) have a
|
||||
**maximum monthly charge** — "once a non-GPU resource reaches its monthly maximum, it no longer incurs
|
||||
charges for the rest of the billing cycle." Static public IP caps at **$3.00/mo** ($0.0045/hr). So a
|
||||
forgotten static IP is a bounded ~$3/mo bleed, but a forgotten 2 TB block volume is **~$120/mo** until
|
||||
destroyed (verified DO pricing 2026-06).
|
||||
- **What actually stops the full meter:** **destroy the machine** AND **release the static IP** AND
|
||||
**delete the storage** (AND delete any **snapshot**) — separate actions. "To stop all charges for a
|
||||
machine and its add-ons, destroy the machine and any resources you no longer need." A stopped-but-not-
|
||||
destroyed machine with a Static IP, a 2 TB block volume, and a leftover snapshot is still spending money.
|
||||
- **Irreversible:** **destroy/delete** of a machine removes its block storage (no recovery); block-storage
|
||||
**expansion** is also one-way. A **shut-down is reversible** (resume later).
|
||||
|
||||
**Net contrast vs the other profiles:** Paperspace gives a real idle-cheap *stop* (unlike Lambda, which has
|
||||
no stop), but unlike AutoDL's 关机 the **storage + IP + snapshots keep billing** until each is explicitly
|
||||
destroyed/released. "Stopped" ≠ "free."
|
||||
|
||||
> **Iron Law (teardown gate):** NO destroy/delete of the machine, release of the IP, or deletion of
|
||||
> `/storage`/block-storage/snapshot until checkpoints are **pulled to local AND verified by load**, and the
|
||||
> user has **explicitly approved** the specific cost-affecting action. A destroy is irreversible — "it
|
||||
> looked done in the log" is not evidence (principle #3). General form →
|
||||
> `superpowers:verification-before-completion`.
|
||||
|
||||
---
|
||||
|
||||
## 6. DAEMON TOOL
|
||||
|
||||
- **Core machines** — full VMs ⇒ `tmux`/`screen`/`nohup` all available; SSH is as stable as any cloud VM.
|
||||
This is the closest analog to the AutoDL tmux-resilient pattern. tmux survives an SSH drop; it does NOT
|
||||
survive a machine **stop/restart** (the process is gone), and — critically on Core/Linux — a live tmux
|
||||
session does **not** defer the wall-clock auto-shutdown (§4), so durability still rests on
|
||||
checkpoint-to-disk + load-latest (principle #8), not on the detach primitive.
|
||||
- **Gradient Notebooks** — a managed Jupyter sandbox: **no clean persistent SSH-daemon story**, and the
|
||||
**auto-shutdown timer is a hard ceiling** — a tmux session started inside a Notebook **still dies at the
|
||||
timeout**. Notebooks are not built for unattended multi-day daemons.
|
||||
- **Platform-native long-job mechanisms** — **Workflows** (DAG automation) and **Deployments** (always-on
|
||||
serving). For training-as-a-daemon, prefer **Core + tmux**; treat Notebooks as interactive/short-run only.
|
||||
|
||||
If `tmux` is absent on a minimal image, fall back to `nohup <cmd> </dev/null >log 2>&1 &`.
|
||||
|
||||
---
|
||||
|
||||
## 7. TOP GOTCHAS (platform-pinned; universal ones → `references/gotchas_universal.md`)
|
||||
|
||||
- **PS1 — "Stopped the machine, still getting billed."**
|
||||
Symptom: GPU meter halted but the bill keeps climbing while the box is off.
|
||||
Root cause: shut-down stops only the **compute** meter; attached **storage** + **public IP** + add-ons +
|
||||
snapshots bill hourly until destroyed/released (verified DO pricing 2026-06).
|
||||
Fix: to truly stop the meter, **destroy the machine, release the Static IP, delete the storage and any
|
||||
snapshot** — separate teardown actions. Audit for orphaned storage/IPs/snapshots after every stop.
|
||||
|
||||
- **PS2 — A long run dies at a round-number wall-clock with no error.**
|
||||
Symptom: training vanishes at exactly 6 h / 12 h (or the configured Core window); no traceback.
|
||||
Root cause: the **auto-shutdown clock**, not a crash — free notebooks 6 h (hard cap), paid notebooks 12 h
|
||||
default, Core 1 h–1 wk. On Core/Linux the clock is **wall-clock, not idle** — an active SSH/tmux session
|
||||
does NOT extend it (verified DO manage-auto-shutdown 2026-06).
|
||||
Fix: checkpoint to `/storage` (Notebooks) or the block disk (Core) **before** the window; for Core, raise
|
||||
the auto-shutdown to the longest needed **in the console** (API/CLI cannot change it post-create);
|
||||
restart + load-latest to resume.
|
||||
|
||||
- **PS3 — `pip install` (or any non-`/storage` write) vanishes after a Notebook restart.**
|
||||
Symptom: packages installed in-session are gone next session; "saved" files disappear after stop/restart.
|
||||
Root cause: `pip` writes to `/usr/local/lib`, which is **ephemeral workspace** — only `/storage` and
|
||||
`/notebooks` persist (verified fast.ai forum + DO storage-architecture 2026-06). "Machines are snapshots,
|
||||
not servers," so in-session installs do not persist.
|
||||
Fix: install into a persisted dir — `pip install --user` (lands in the home dir under a persisted tree)
|
||||
or `pip install --target /storage/pyenv && export PYTHONPATH=/storage/pyenv`; write all
|
||||
checkpoints/logs/outputs under `/storage`; verify they landed (`ls`/checksum) before stop.
|
||||
|
||||
- **PS4 — Automation 404s / silently no-ops / installs the wrong SDK.**
|
||||
Symptom: a `gradient`-era create/stop call fails or does nothing; or `pip install gradient` (v3+) imports
|
||||
an inference SDK with no notebook/machine commands.
|
||||
Root cause: **legacy Gradient REST endpoints deprecated 15 Jul 2024**; **`gradient-cli` v2 deprecated**;
|
||||
**`gradient-python` v3 is the DigitalOcean Gradient AI inference SDK — a name collision**, not the
|
||||
orchestration CLI (verified github.com/Paperspace/gradient-cli + digitalocean/gradient-python 2026-06).
|
||||
Fix: for new work use the **`pspace` CLI** (github.com/Paperspace/cli); to keep old scripts alive pin
|
||||
`pip install "gradient<3.0"`. Pin and verify the CLI binary + version in any automation.
|
||||
|
||||
- **PS5 — Custom template / storage / volume "not found" in a different datacenter.**
|
||||
Symptom: a saved template or block volume is unavailable when launching elsewhere; block-storage resize
|
||||
can't be undone.
|
||||
Root cause: storage and templates are **region/DC-locked**, and **block-storage expansion is
|
||||
irreversible** (one-way filesystem grow).
|
||||
Fix: pick the datacenter deliberately and keep storage+compute+template co-located; size block storage
|
||||
with headroom up-front (cannot shrink).
|
||||
|
||||
- **PS6 — SSH alias breaks after every restart.**
|
||||
Symptom: the saved `ssh` host no longer connects after a machine restart.
|
||||
Root cause: a **Dynamic public IP** is released on power-off and reassigned on start (new IP each time).
|
||||
Fix: attach a **Static IP** for stable SSH/endpoint addressing (it bills until deleted, capped $3/mo —
|
||||
`PS1`), or re-resolve the address on each start before scripting. Note: API/CLI can manage a *static* IP
|
||||
but cannot add a *dynamic* one to an existing machine (request dynamic at create time).
|
||||
|
||||
- **PS7 — Free-tier notebook code is PUBLIC by default.**
|
||||
Symptom: proprietary/confidential code is world-readable in a Gradient free notebook.
|
||||
Root cause: free Gradient notebooks are **public by default; private notebooks require a paid plan**
|
||||
(verified Paperspace blog / pricing 2026-06).
|
||||
Fix: never put confidential code or any secret in a free notebook; upgrade to a paid plan for private
|
||||
notebooks. Treat the free tier as a public scratchpad. (Secrets hygiene → `references/gotchas_universal.md`.)
|
||||
|
||||
- **PS8 — Free notebook won't start / sits "pending."**
|
||||
Symptom: a free-GPU notebook stays pending or errors "out of capacity"; only one notebook will run.
|
||||
Root cause: free tier = **1 concurrent running notebook, ≤5 projects, 5 GB `/storage`**, and free machines
|
||||
are pooled — a pending notebook is queued for the next free machine (verified Paperspace free-instances
|
||||
docs + blog 2026-06).
|
||||
Fix: expect queueing on free; stop the other free notebook (only one runs); for assured access use a paid
|
||||
instance type, which skips the free queue.
|
||||
|
||||
- **PS9 — A destroyed machine keeps billing via a leftover snapshot.**
|
||||
Symptom: machine destroyed, yet a small monthly charge persists.
|
||||
Root cause: **snapshots are a separate resource that survives a machine destroy** and bills at
|
||||
`$0.29/GB/mo` until deleted; auto-snapshot defaults to "Never"/0 but a manually-enabled policy (daily by
|
||||
default, up to 10 stored) silently accrues (verified DO pricing + blog/automated-snapshots 2026-06).
|
||||
Fix: when tearing down, delete the snapshot too (console or CLI); audit the snapshots list after every
|
||||
machine destroy. Capped per-resource by the monthly maximum but still a bleed.
|
||||
|
||||
- **PS10 — Notebook upload/import fails on the 5 GB free cap.**
|
||||
Symptom: uploading a multi-GB dataset to `/storage` fails for an unpaid account.
|
||||
Root cause: free `/storage` allowance is **5 GB**; overage is **$0.29/GB/mo** (paid plans include more:
|
||||
e.g. 200 GB / 1 TB tiers) (verified Paperspace pricing + fast.ai forum 2026-06).
|
||||
Fix: stream/stage the dataset rather than uploading the whole thing, prune aggressively, or upgrade the
|
||||
plan; redirect HF/torch caches off `/storage` if they would push over the allowance.
|
||||
|
||||
- **PS11 — ML-in-a-Box CUDA/driver too old for current PyTorch on a new-arch GPU.**
|
||||
Symptom: `The NVIDIA driver on your system is too old (found version 110xx). Please update your GPU
|
||||
driver`, or `no kernel image is available for execution` on a fresh card.
|
||||
Root cause: the template's **host driver/CUDA stack lags newer PyTorch wheels**; on a rental the host
|
||||
driver is host-global and a tenant usually cannot upgrade it (verified github.com/Paperspace/ml-in-a-box
|
||||
issue #13 2026-06). This is the platform-pinned face of the universal CUDA-triangle (U28).
|
||||
Fix: install a torch build matching the box's CUDA (do not force-upgrade the host driver on a rental);
|
||||
pick a template whose Ubuntu/driver matches the GPU (22.04 for H100/A100). Full triangle → U28 in
|
||||
`references/gotchas_universal.md`.
|
||||
|
||||
- **PS12 — Gradient Deployment / custom image won't pull or drifts.**
|
||||
Symptom: a Deployment fails to pull `<user>/img:tag`, or "the same image" behaves differently over time.
|
||||
Root cause: a moving tag (`:latest`) resolves to a different layer set; private-registry creds missing.
|
||||
Fix: pin the image by digest (`@sha256:`) and supply registry creds as a Gradient **secret**, not inline.
|
||||
General form → U30 in `references/gotchas_universal.md`.
|
||||
|
||||
- **PS13 — Platform-specific debugging.** Commands + what to check (Core uses standard Linux tooling; the
|
||||
Notebook-only items are the platform delta):
|
||||
- **Confirm GPU + driver/torch match:** `nvidia-smi` (driver/CUDA version) then
|
||||
`python -c "import torch;print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"` —
|
||||
a mismatch here is `PS11`/U28, not a code bug.
|
||||
- **Find what is eating the 5 GB / over-allowance `/storage` (the platform's own recommended cmd):**
|
||||
`du -sch .[!.]* * | sort -h` (or `!du -sch …` in a cell); install `ncdu` for an interactive view
|
||||
(verified DO notebooks/how-to/manage-storage 2026-06). Check `df -h` AND `df -i` (inodes, U7).
|
||||
- **Is a Notebook write durable?** `df -h /storage /notebooks` and confirm the target is one of those two
|
||||
mounts — anything else (incl. `/usr/local/lib`) is ephemeral (`PS3`).
|
||||
- **Why did the run vanish?** Walk the universal ladder (U3): `dmesg | grep -iE 'killed process|out of
|
||||
memory'` (OOM?), `uptime` (recent reboot = auto-shutdown fired, `PS2`), `nvidia-smi` (GPU idle = died,
|
||||
not hung). A round-number `uptime`-near-window with a clean `dmesg` ⇒ auto-shutdown, not a crash.
|
||||
- **Detect a stuck/slow download:** watch the target file size grow
|
||||
(`watch -n5 'ls -l /storage/<file>'`); a flat size with a live process = stalled wire (U12 resumable
|
||||
loop). Egress is direct/unproxied here, so a stall is route/peer, not a missing proxy hook.
|
||||
- **Audit orphaned billables before declaring teardown done:** in the console (or `pspace`) list
|
||||
machines, **public IPs**, **storage/volumes**, and **snapshots** — `PS1`/`PS9` hide in the last two.
|
||||
|
||||
---
|
||||
|
||||
## 8. SCRIPT OVERRIDES
|
||||
|
||||
Values to parameterize the `scripts/` templates for Paperspace. Forward-slash paths; placeholders for any
|
||||
host/IP (never a real address). Core and Gradient differ — both shown.
|
||||
|
||||
```sh
|
||||
# --- Gradient Notebook ---
|
||||
DATA_DIR=/storage # team-shared persistent; survives stop AND notebook delete
|
||||
DURABLE_DIR=/storage # checkpoints land here (NOT /notebooks — dies with the notebook)
|
||||
SCRATCH=/tmp # ephemeral workspace; wiped on stop — never the only copy
|
||||
HF_HOME=/storage/.cache/huggingface # redirect cache off ephemeral workspace (watch the 5 GB free cap, PS10)
|
||||
PROXY_HOOK= # none — direct egress (no network_turbo)
|
||||
CRED_FILE="" # Paperspace keys are Gradient secrets / env vars, not files — WANDB_API_KEY/HF_TOKEN arrive via the secret/env (run_one's [ -n "$CRED_FILE" ] guard skips the file read); never write keys to /storage (team-shared)
|
||||
DETACH= # no clean tmux; Jupyter kernel + hard 6h/12h auto-shutdown ceiling
|
||||
# NOTE: pip into /storage to persist — pip install --target /storage/pyenv && export PYTHONPATH=/storage/pyenv (PS3)
|
||||
|
||||
# --- Core machine (preferred for daemonized training) ---
|
||||
DATA_DIR=/path/to/blockstore # placeholder — the attached block disk mount
|
||||
DURABLE_DIR=/path/to/blockstore/ckpts
|
||||
SCRATCH=/tmp
|
||||
HF_HOME=/path/to/blockstore/.cache/huggingface
|
||||
PROXY_HOOK= # none
|
||||
CRED_FILE="" # Paperspace keys are Gradient secrets / env vars, not files — WANDB_API_KEY/HF_TOKEN arrive via the secret/env (run_one's [ -n "$CRED_FILE" ] guard skips the file read); inject at launch, never inline
|
||||
DETACH=tmux # survives SSH drop, NOT a machine stop, and NOT the wall-clock auto-shutdown — rely on checkpoint+resume
|
||||
SSH_HOST=<machine-ip> # placeholder — ML-in-a-Box user is `paperspace`; pin a Static IP for a stable alias (PS6); dynamic IP changes every start
|
||||
```
|
||||
|
||||
Reminder: secrets referenced by env-var NAME or Gradient secret only — never inline a key, and never write
|
||||
one onto the team-shared `/storage` (universal secrets gotcha → `references/gotchas_universal.md`).
|
||||
Reference in New Issue
Block a user