📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,234 @@
|
||||
# Debugging and iteration
|
||||
|
||||
How to iterate cheaply, read logs, and smoke-test. Dev mode + SSH exists as a last resort (covered at the bottom).
|
||||
|
||||
## The rung ladder
|
||||
|
||||
Pick the cheapest update mechanism that fits the change. Going one rung too high wastes 30 s – 15 min per cycle.
|
||||
|
||||
| Rung | When | Command | Cost |
|
||||
|---|---|---|---|
|
||||
| 1. Hot-reload | Pure Python edit on a Gradio Space (SDK 6.1+), **no new deps** | `hf spaces hot-reload <id> -f app.py` | seconds, no rebuild |
|
||||
| 2. `hf upload` | Code-only change hot-reload can't handle (`gr.Server`, Streamlit, Docker entrypoint, non-Python file) | `hf upload <id> . --include '<file>'` | 30–90 s app restart |
|
||||
| 3. Full rebuild | `requirements.txt`, `Dockerfile`, README frontmatter, or hardware change | `hf upload <id> . && hf spaces logs <id> --build --follow` | 1–15 min |
|
||||
| 4. Factory reboot | Container in inconsistent state (broken pip env, etc.) | `hf spaces restart <id> --factory-reboot` | full rebuild + cold start |
|
||||
|
||||
### How hot-reload works
|
||||
|
||||
`hf spaces hot-reload <id> -f app.py` patches the running Python process in place via `jurigged` (vendored inside the `spaces` package), then commits the change to the repo with a hot-reload marker that tells the platform to skip its usual restart. Seconds, no rebuild.
|
||||
|
||||
Applies cleanly to function-body changes and new top-level symbols. Does **not** rerun module-level imports or one-time init — the model was loaded when `app.py` first ran and jurigged won't re-execute that load. New pip deps, README frontmatter, `Dockerfile`, and hardware changes need a full rebuild (rung 3).
|
||||
|
||||
Independent of dev mode. Marked experimental in the CLI. Requires Gradio SDK 6.1+.
|
||||
|
||||
### Footguns
|
||||
|
||||
- **Hot-reload poisons factory reboot.** A commit that's only a hot-reload leaves runtime metadata that's valid only while the hot process is alive. `--factory-reboot` on top of one can fail with `fatal: could not read Username for 'https://huggingface.co'`. Recovery: push any normal `hf upload` commit (even a one-line no-op) first, then restart.
|
||||
- **`runtime.sha` lags repo SHA on restart.** `hf upload` succeeds → repo updates → `hf spaces info` keeps reporting the *previous* commit's SHA under `runtime` for several minutes while the new container loads. Poll `runtime.sha`, not just `stage`, and don't issue another restart until it flips.
|
||||
- **Concurrent uploads or restart-while-uploading collide.** Wait for one to finish.
|
||||
- **"Let me try locally first"** for anything that depends on the Space's Python / torch / CUDA env. The Space environment is the only one that matters. `python3 -m py_compile app.py` is the maximum local check worth doing before pushing.
|
||||
|
||||
## Reading logs
|
||||
|
||||
```bash
|
||||
hf spaces info <id> --expand runtime # stage at a glance
|
||||
hf spaces logs <id> --build --follow # build log, live
|
||||
hf spaces logs <id> --follow # run log, live
|
||||
hf spaces logs <id> --build --tail 500 # bigger window — default is small
|
||||
```
|
||||
|
||||
Find the **first** error in the build log, not the last. Cascading errors after the first are noise.
|
||||
|
||||
State machine (terminal states in bold):
|
||||
|
||||
```
|
||||
BUILDING → APP_STARTING → RUNNING
|
||||
↘ RUNTIME_ERROR
|
||||
↘ BUILD_ERROR
|
||||
↘ CONFIG_ERROR
|
||||
```
|
||||
|
||||
For stage-specific lookups, see [`known-errors.md`](known-errors.md).
|
||||
|
||||
## Smoke-test patterns
|
||||
|
||||
A Space isn't done until a `gradio_client` call against the live URL exercises the endpoint end-to-end. Four steps in order — keep `hf spaces logs <id> --follow` running in another terminal throughout, so any silent fallback (model snapping to a different size, missing optional dep, dtype downgrade) surfaces.
|
||||
|
||||
### A. Alive?
|
||||
|
||||
```bash
|
||||
hf spaces info <id> --expand runtime --format json \
|
||||
| python3 -c "import json,sys; r=json.load(sys.stdin)['runtime']; \
|
||||
print(r['stage'], r.get('hardware','?'))"
|
||||
# expect: RUNNING zero-a10g
|
||||
```
|
||||
|
||||
If `requested_hardware` is `cpu-basic` when you wanted GPU, your `--flavor` was rejected silently. Fix with `hf spaces settings <id> --hardware zero-a10g`.
|
||||
|
||||
### B. Logs clean post-boot?
|
||||
|
||||
```bash
|
||||
hf spaces logs <id> --tail 200
|
||||
```
|
||||
|
||||
Confirm the model finished loading, no import warnings, no "falling back to CPU" / dtype-downgrade messages, no failing health checks the platform forgave. Do this before calling the API — many silent failures (a config typo loading the wrong model, a missing optional dep, a one-time init that errored but didn't crash boot) are only visible here.
|
||||
|
||||
### C. API actually functions?
|
||||
|
||||
Default — sync `gr.Interface` / `gr.Blocks` / `gr.ChatInterface` / `gr.Server` `@app.api`:
|
||||
|
||||
```python
|
||||
from gradio_client import Client, handle_file
|
||||
import os
|
||||
|
||||
c = Client("<ns>/<name>", token=os.environ["HF_TOKEN"],
|
||||
httpx_kwargs={"timeout": 600}) # ≥ @spaces.GPU duration + 60s
|
||||
|
||||
print(c.view_api()) # discover endpoints — don't guess api_name
|
||||
|
||||
result = c.predict(
|
||||
handle_file("test.png"), # file inputs need handle_file()
|
||||
"short prompt",
|
||||
api_name="/generate", # matches @app.api(name=...) or the function name
|
||||
)
|
||||
```
|
||||
|
||||
**Streaming endpoints** (function uses `yield` or `TextIteratorStreamer`) — `.predict()` returns only the final value. Iterate chunks via `.submit()`:
|
||||
|
||||
```python
|
||||
job = c.submit("short prompt", api_name="/chat")
|
||||
for chunk in job: print(chunk, end="")
|
||||
# or job.result() for the final value
|
||||
```
|
||||
|
||||
**`gr.Server` custom `@app.get/post(...)` routes** don't appear in `view_api()`. Hit them with plain HTTP:
|
||||
|
||||
```python
|
||||
import httpx
|
||||
r = httpx.post(f"https://<subdomain>.hf.space/your_route",
|
||||
json={...}, timeout=600,
|
||||
headers={"Authorization": f"Bearer {os.environ['HF_TOKEN']}"})
|
||||
```
|
||||
|
||||
**OAuth-gated Spaces** (`hf_oauth: true` + `gr.LoginButton`) — anonymous `Client` can't authenticate. Test interactively after sign-in, or capture a session token and pass via `httpx_kwargs={"headers": {...}}`.
|
||||
|
||||
**MCP server mode** (`launch(mcp_server=True)`) — different protocol. Use an MCP client.
|
||||
|
||||
### D. Output bytes AND logs look right?
|
||||
|
||||
HTTP 200 ≠ correct output. Sniff both the returned file and the run log emitted during the call.
|
||||
|
||||
```python
|
||||
head = open(path, "rb").read(16)
|
||||
# b'glTF...' → glb
|
||||
# b'\x89PNG' → png
|
||||
# b'\xff\xd8' → jpeg
|
||||
# b'RIFF...WEBP' → webp
|
||||
# b'RIFF...WAVE' → wav
|
||||
# head[4:8]==b'ftyp' → mp4
|
||||
# b'ply\n' → ply
|
||||
```
|
||||
|
||||
For text: non-empty, not all `<think>...</think>` (thinking-model leak), length reasonable. For images: returned dimensions match what was requested (some models snap to nearest preset).
|
||||
|
||||
Look at the tailed run log alongside — silent fallbacks (model snapping resolution, missing optional dep falling back to a slower path, dtype downgrade) only show up there.
|
||||
|
||||
### What NOT to do
|
||||
|
||||
- **Don't launch Playwright / headless browser** to verify backend logic. The Gradio UI calls the same API `gradio_client` does — one `predict` tests both.
|
||||
- **Don't build mock-mode + local-server harnesses** before pushing. Local-green ≠ Space-green.
|
||||
- **Don't smoke-test with full-budget inputs.** Smallest input that exercises the GPU code path — short prompt, small image, low step count. You're verifying wiring, not quality.
|
||||
|
||||
## Iterating on the Space, not locally
|
||||
|
||||
The Space env is the only one that matters: Python, torch, CUDA, file paths, env vars, gradio version, the `spaces` hijack all differ from your laptop.
|
||||
|
||||
Workflow:
|
||||
|
||||
1. Decide SDK + hardware. Write the smallest `app.py` / `Dockerfile` + `requirements.txt` + README frontmatter — just enough that the entry point loads.
|
||||
2. Push immediately. Don't build a Playwright / mock harness first.
|
||||
3. Once `RUNNING`: verify with `gradio_client` against the real Space. That's your test loop.
|
||||
4. Iterate via the cheapest rung.
|
||||
|
||||
`python3 -m py_compile app.py` is the maximum local check worth doing.
|
||||
|
||||
## Last resort: dev mode + SSH
|
||||
|
||||
Use only when:
|
||||
|
||||
- A failure is non-deterministic (device-side asserts, OOM under specific shapes, race conditions).
|
||||
- You need `CUDA_LAUNCH_BLOCKING=1` or `gdb` to localize a CUDA error.
|
||||
- You'd burn 4+ build cycles trying variations from outside.
|
||||
|
||||
Reading logs + grepping [`known-errors.md`](known-errors.md) + a tight `gradio_client` smoke loop solves the vast majority of issues. Dev mode is a heavy hammer.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. **PRO / Team / Enterprise plan** — dev mode is a paid feature.
|
||||
2. **An SSH key registered on the user's HF profile.** Without this, SSH refuses the connection. If the user doesn't have one yet, they need to:
|
||||
- Generate a keypair locally: `ssh-keygen -t ed25519 -f ~/.ssh/hf_dev -N ''` (no passphrase keeps automation simple; user can pick differently if they prefer).
|
||||
- Add the **public** key (`~/.ssh/hf_dev.pub`) at https://huggingface.co/settings/keys.
|
||||
- Keep the **private** key (`~/.ssh/hf_dev`) on the machine they'll SSH from.
|
||||
3. **The Space must be in `RUNNING` or `RUNTIME_ERROR`** before dev mode lets you in — not `BUILD_ERROR`. If it's in build error, push a stub `app.py` that boots cleanly first (e.g. `import gradio as gr; gr.Interface(lambda: 'ok', None, 'text').launch()`), then enable dev mode.
|
||||
|
||||
### Enable
|
||||
|
||||
No `huggingface_hub` Python wrapper yet — use the REST endpoint:
|
||||
|
||||
```bash
|
||||
curl -s -X POST \
|
||||
-H "Authorization: Bearer $HF_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
"https://huggingface.co/api/spaces/<ns>/<space>/dev-mode" \
|
||||
-d '{"enabled": true}'
|
||||
```
|
||||
|
||||
### SSH in
|
||||
|
||||
```bash
|
||||
ssh -i ~/.ssh/hf_dev -o BatchMode=yes -o StrictHostKeyChecking=accept-new \
|
||||
<ns>-<space>@ssh.hf.space
|
||||
```
|
||||
|
||||
Username is `<namespace>-<space>` **lowercase**, with `-` replacing `/`. Dots in the Space name become hyphens too.
|
||||
|
||||
### Inside the VM
|
||||
|
||||
It's a normal container at `/home/user/app/`. You can edit files, `pip install`, run repros, call `@spaces.GPU`-decorated functions interactively (they get a real GPU window).
|
||||
|
||||
**`nvidia-smi` in the bare terminal will fail** with `NVML: Unknown Error`. Expected — ZeroGPU only exposes the real GPU inside `@spaces.GPU` calls. Don't assume the GPU is broken.
|
||||
|
||||
**Edits in `/home/user/app/` don't survive** a restart, sleep, or dev-mode-disable. Only commits persist.
|
||||
|
||||
### Smoke-test the fix inside the container
|
||||
|
||||
Before exiting dev mode, verify the fix actually works under a real GPU window:
|
||||
|
||||
```bash
|
||||
cat > /home/user/app/_devtest.py <<'PY'
|
||||
import spaces, torch
|
||||
from app import predict # or whatever your @spaces.GPU function is
|
||||
print(predict(<realistic-args>))
|
||||
PY
|
||||
python3 _devtest.py
|
||||
```
|
||||
|
||||
### Persist + exit
|
||||
|
||||
Commit + push from inside the container (`git config user.email / user.name` first; the HF git remote works). Then disable dev mode:
|
||||
|
||||
```bash
|
||||
curl -s -X POST -H "Authorization: Bearer $HF_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
"https://huggingface.co/api/spaces/<ns>/<space>/dev-mode" \
|
||||
-d '{"enabled": false}'
|
||||
```
|
||||
|
||||
**Factory-reboot** to apply the pushed state (in dev mode the Space won't rebuild on commits):
|
||||
|
||||
```python
|
||||
from huggingface_hub import HfApi
|
||||
HfApi(token=HF_TOKEN).restart_space("<ns>/<space>", factory_reboot=True)
|
||||
```
|
||||
|
||||
Then re-run the outside-the-container smoke test. Dev-mode success does **not** guarantee post-rebuild success — different image, different process tree.
|
||||
Reference in New Issue
Block a user