📦 deps(thirdparty): update snapshots
This commit is contained in:
@@ -134,7 +134,7 @@ Two purchase modes, two distinct interruption vectors:
|
||||
- **RP9 — CUDA forward-compat error (host driver too old).** Symptom: container runs locally but on RunPod throws `CUDA failure 804: forward compatibility was attempted on non supported HW`, or `cuda>=12.x, please update your driver`, or `OCI runtime create failed`. Root cause: the assigned machine's NVIDIA host driver is older than the image's CUDA needs (e.g. driver 525.x under a CUDA 12.1 image). Fix: in the deploy dialog use **Additional filters → CUDA Version** to require a machine whose driver meets the image's minimum; or pick an image matching the available driver. (verified github.com/runpod/containers/issues/67 2026-06)
|
||||
- **RP10 — `ENTRYPOINT` in a custom image silences the template start command.** Symptom: a custom image deploys but never starts `sshd` / the handler / `/start.sh`; the container runs the wrong process and SSH never comes up. Root cause: an image `ENTRYPOINT` cannot be overridden by the RunPod template's "container start command" (which only overrides `CMD`). Fix: use `CMD ["/start.sh"]` (not `ENTRYPOINT`) in the Dockerfile so the template override works. (verified github.com/runpod/runpodctl/issues/170 2026-06)
|
||||
- **RP11 — Container disk (~5 GB) fills, not the volume disk.** Symptom: "No space left on device" mid-`pip install` / mid-download even though `/workspace` has free GB. Root cause: pip wheels, the HF cache, apt and conda default to `/` (the small ~5 GB overlay), not `/workspace`. Fix: raise container-disk size at create time, AND redirect caches onto the volume — `export HF_HOME=/workspace/hf PIP_CACHE_DIR=/workspace/.cache/pip`, install conda envs under `/workspace`. Diagnose with the §7-debug commands. (verified docs.runpod.io/pods/troubleshooting/storage-full 2026-06)
|
||||
- **RP12 — Env vars set on the Pod are missing inside a full-SSH (over-TCP) session.** Symptom: `WANDB_API_KEY` / `HF_TOKEN` / template env vars are empty when reached via full SSH, though they exist in the web terminal / basic SSH. Root cause: the SSH daemon's login shell does not inherit the container env set on PID 1 at startup. Fix: snapshot at boot in the start command (`env > /workspace/.env_vars.txt`) and source it in the SSH session, or write the vars into `/etc/environment` / `~/.bashrc`. (verified leimao.github.io Setting-Up-Environment-Variables-SSH-Over-TCP-Runpod 2026-06)
|
||||
- **RP12 — Env vars set on the Pod are missing inside a full-SSH (over-TCP) session.** Symptom: `WANDB_API_KEY` / `HF_TOKEN` / template env vars are empty when reached via full SSH, though they exist in the web terminal / basic SSH. Root cause: the SSH daemon's login shell does not inherit the container env set on PID 1 at startup. Fix: pass the few required non-secret values explicitly, or create a root-owned/session-only file on container disk with `umask 077` and named exports only. Never dump `env` wholesale, and never write secret snapshots under `/workspace` or a Network Volume. (verified leimao.github.io Setting-Up-Environment-Variables-SSH-Over-TCP-Runpod 2026-06)
|
||||
- **RP13 — `runpodctl send/receive` is only for small/medium files.** Symptom: a large dataset transfer via `runpodctl send` is slow or unreliable. Root cause: the one-time-code transfer is positioned for "quick, occasional, small-to-medium" exchanges, not bulk data. Fix: use full-SSH `rsync` (RP6) or the Network-Volume S3 API for large datasets; keep `send/receive` for keyless one-off pulls on no-public-IP Pods. (verified docs.runpod.io/runpodctl/transfer-files 2026-06)
|
||||
|
||||
### Platform-specific debugging
|
||||
@@ -158,7 +158,7 @@ Values to parameterize the `scripts/` templates for RunPod:
|
||||
- `DATA_DIR=` `/workspace` (the per-Pod volume disk) — stop-safe working state (code, conda/pip env, in-progress outputs survive a stop, not a terminate).
|
||||
- `DURABLE_DIR=` a **Network Volume** mount (`/workspace` on Pods, `/runpod-volume` on Serverless) — terminate-safe durable checkpoints. Point `DURABLE_DIR` at the Network Volume when `terminate` is the teardown verb so `best` checkpoints survive Pod deletion AND the low-balance auto-delete (RP8).
|
||||
- `PROXY_HOOK=` none. No China mirror. Instead `export HF_HUB_ENABLE_HF_TRANSFER=1` (after `pip install huggingface_hub[hf_transfer]`).
|
||||
- `CRED_FILE=""` — no credential file on disk; the key is a RunPod secret / env var injected at Pod creation, so `WANDB_API_KEY` / `HF_TOKEN` arrive via the platform env and `run_one`'s `[ -n "$CRED_FILE" ]` guard skips the file read. **Caveat (RP12):** a full-SSH-over-TCP login shell may NOT see these env vars — snapshot them at boot (`env > /workspace/.env_vars.txt`) and source in the SSH session if a script reads them there. **NEVER** write a key to a Network Volume — it is unencryptable and shared across every attached Pod.
|
||||
- `CRED_FILE=""` — no credential file on disk; the key is a RunPod secret / env var injected at Pod creation, so `WANDB_API_KEY` / `HF_TOKEN` arrive via the platform env and `run_one`'s `[ -n "$CRED_FILE" ]` guard skips the file read. **Caveat (RP12):** a full-SSH-over-TCP login shell may NOT see these env vars. Prefer platform secrets or pass named values directly to the command that needs them. If a temporary bridge file is unavoidable, create it on container disk with `umask 077`, write only named required exports, delete it after use, and never place it under `/workspace` or a Network Volume.
|
||||
- `SCRATCH=` periodic/`latest` checkpoints under the Network Volume; keep `best` only (`save_top_k` small). Pruning matters more here — the volume disk grows-only and stopped storage is double-priced (RP4).
|
||||
- `HF_HOME=` a path on the Network Volume (e.g. `/workspace/hf` on a Network-Volume-backed Pod) so model caches survive Pod churn instead of re-downloading — AND to keep the cache off the tiny ~5 GB container disk (RP11). Likewise `PIP_CACHE_DIR=/workspace/.cache/pip`.
|
||||
- `DETACH=` `tmux` (after `apt-get install -y tmux`); fall back to `nohup … </dev/null >log 2>&1 &`. Neither survives a Pod restart — checkpoint-to-Network-Volume is the resilience layer.
|
||||
|
||||
Reference in New Issue
Block a user