90 lines
4.3 KiB
Markdown
90 lines
4.3 KiB
Markdown
# Persistent storage with Buckets
|
|
|
|
Spaces are stateless. All data is wiped on restart / rebuild. For state that must survive (user uploads, generations, dynamic feeds, logs, growing databases): mount an HF **Bucket** — S3-like object storage living at `hf://buckets/<ns>/<bucket>`.
|
|
|
|
Buckets are paid (per-TB storage). Check `whoami.canPay` and confirm with the user before creating one. Pricing + free tier: https://huggingface.co/storage.
|
|
|
|
Full docs: https://huggingface.co/docs/hub/storage-buckets.
|
|
|
|
## Create + attach
|
|
|
|
```bash
|
|
hf buckets create <ns>/<bucket-name> # --private optional
|
|
hf spaces volumes set <ns>/<space> -v hf://buckets/<ns>/<bucket-name>:/data
|
|
```
|
|
|
|
After this, writes to `/data/` in the Space are durable. Reads come from the bucket via the Xet storage backend.
|
|
|
|
To make the bucket files publicly addressable: leave the bucket public. Public bucket files are served at `https://huggingface.co/buckets/<ns>/<bucket>/resolve/<path>` (HTTP 302 redirect to a signed CDN URL). The Space writes once and the public URL works forever — no streaming proxy needed.
|
|
|
|
## Write-durable, read-fast pattern
|
|
|
|
For a feed-style Space (e.g. a community jam where users save generations and browse a public timeline), don't re-scan disk on every request. Module-level disk scan → in-memory list → every write appends to both:
|
|
|
|
```python
|
|
import os, json, uuid
|
|
from datetime import datetime, timezone
|
|
|
|
BUCKET_ID = "<ns>/<bucket-name>"
|
|
BUCKET_URL = f"https://huggingface.co/buckets/{BUCKET_ID}/resolve"
|
|
_feed = []
|
|
|
|
def _load_feed():
|
|
root = "/data/songs"
|
|
if not os.path.isdir(root):
|
|
return
|
|
for sid in os.listdir(root):
|
|
meta = f"{root}/{sid}/meta.json"
|
|
if os.path.isfile(meta):
|
|
_feed.append(json.load(open(meta)))
|
|
_feed.sort(key=lambda s: s["created_at"], reverse=True)
|
|
|
|
_load_feed() # one scan at startup
|
|
|
|
@app.api(name="save", time_limit=60)
|
|
def save(audio_bytes: bytes, title: str):
|
|
sid = uuid.uuid4().hex[:12]
|
|
d = f"/data/songs/{sid}"; os.makedirs(d, exist_ok=True)
|
|
open(f"{d}/audio.wav", "wb").write(audio_bytes)
|
|
meta = {"id": sid, "title": title,
|
|
"url": f"{BUCKET_URL}/songs/{sid}/audio.wav",
|
|
"created_at": datetime.now(timezone.utc).isoformat()}
|
|
json.dump(meta, open(f"{d}/meta.json", "w"))
|
|
_feed.insert(0, meta) # cache stays current — no re-scan
|
|
return meta
|
|
|
|
@app.api(name="feed", concurrency_limit=10)
|
|
def feed(): return _feed[:50] # zero disk I/O
|
|
```
|
|
|
|
Reference Space using this pattern: https://huggingface.co/spaces/victor/ace-step-jam
|
|
|
|
## Anti-pattern: bucket as model-weights cache
|
|
|
|
**Do NOT** `snapshot_download(..., local_dir="/data/weights")` and load checkpoints from there. Bucket I/O is S3-paced; reading a 22 GB `safetensors` from `/data` during `from_pretrained` stalls past any `@spaces.GPU` duration cap.
|
|
|
|
For model weights, let HF Hub re-download to local container disk on each cold start. With `HF_HUB_ENABLE_HF_TRANSFER=1` (set in the runtime by default) this is fast — typically much faster than streaming the same bytes through bucket I/O at request time.
|
|
|
|
Bucket I/O is fine for occasional metadata reads (the feed pattern above) or saving user information. It is *not* fine as the path your model loader streams gigabytes through every cold start.
|
|
|
|
## Cache redirects
|
|
|
|
`/home/user/.cache` is read-only on ZeroGPU. Redirect transient caches at the top of `app.py`, before any library import that uses them:
|
|
|
|
```python
|
|
import os
|
|
os.environ.setdefault("HF_HOME", "/data/.cache/huggingface") # or /tmp on non-bucket Spaces
|
|
os.environ.setdefault("HF_MODULES_CACHE", "/tmp/hf_modules")
|
|
os.environ.setdefault("MPLCONFIGDIR", "/tmp/matplotlib")
|
|
```
|
|
|
|
Missing redirections fail silently or at first matplotlib / transformers / diffusers import.
|
|
|
|
## Write access from the Space
|
|
|
|
The Space's `HF_TOKEN` secret needs write permission on the bucket. Set via Settings → Secrets in the Space UI, or `hf spaces secrets set <id> HF_TOKEN=<token>`.
|
|
|
|
## Security note
|
|
|
|
Public bucket files are publicly accessible forever at their resolve URL. **Don't write PII** to a public bucket. If you need durable but private storage (e.g. per-user history requiring an HF login), keep the bucket private and gate reads through your Space's own auth.
|