Persistent storage with Buckets

Spaces are stateless. All data is wiped on restart / rebuild. For state that must survive (user uploads, generations, dynamic feeds, logs, growing databases): mount an HF Bucket — S3-like object storage living at hf://buckets/<ns>/<bucket>.

Buckets are paid (per-TB storage). Check whoami.canPay and confirm with the user before creating one. Pricing + free tier: https://huggingface.co/storage.

Full docs: https://huggingface.co/docs/hub/storage-buckets.

Create + attach

hf buckets create <ns>/<bucket-name>                                # --private optional
hf spaces volumes set <ns>/<space> -v hf://buckets/<ns>/<bucket-name>:/data

After this, writes to /data/ in the Space are durable. Reads come from the bucket via the Xet storage backend.

To make the bucket files publicly addressable: leave the bucket public. Public bucket files are served at https://huggingface.co/buckets/<ns>/<bucket>/resolve/<path> (HTTP 302 redirect to a signed CDN URL). The Space writes once and the public URL works forever — no streaming proxy needed.

Write-durable, read-fast pattern

For a feed-style Space (e.g. a community jam where users save generations and browse a public timeline), don't re-scan disk on every request. Module-level disk scan → in-memory list → every write appends to both:

import os, json, uuid
from datetime import datetime, timezone

BUCKET_ID = "<ns>/<bucket-name>"
BUCKET_URL = f"https://huggingface.co/buckets/{BUCKET_ID}/resolve"
_feed = []

def _load_feed():
    root = "/data/songs"
    if not os.path.isdir(root):
        return
    for sid in os.listdir(root):
        meta = f"{root}/{sid}/meta.json"
        if os.path.isfile(meta):
            _feed.append(json.load(open(meta)))
    _feed.sort(key=lambda s: s["created_at"], reverse=True)

_load_feed()                                       # one scan at startup

@app.api(name="save", time_limit=60)
def save(audio_bytes: bytes, title: str):
    sid = uuid.uuid4().hex[:12]
    d = f"/data/songs/{sid}"; os.makedirs(d, exist_ok=True)
    open(f"{d}/audio.wav", "wb").write(audio_bytes)
    meta = {"id": sid, "title": title,
            "url": f"{BUCKET_URL}/songs/{sid}/audio.wav",
            "created_at": datetime.now(timezone.utc).isoformat()}
    json.dump(meta, open(f"{d}/meta.json", "w"))
    _feed.insert(0, meta)                          # cache stays current — no re-scan
    return meta

@app.api(name="feed", concurrency_limit=10)
def feed(): return _feed[:50]                      # zero disk I/O

Reference Space using this pattern: https://huggingface.co/spaces/victor/ace-step-jam

Anti-pattern: bucket as model-weights cache

Do NOT snapshot_download(..., local_dir="/data/weights") and load checkpoints from there. Bucket I/O is S3-paced; reading a 22 GB safetensors from /data during from_pretrained stalls past any @spaces.GPU duration cap.

For model weights, let HF Hub re-download to local container disk on each cold start. With HF_HUB_ENABLE_HF_TRANSFER=1 (set in the runtime by default) this is fast — typically much faster than streaming the same bytes through bucket I/O at request time.

Bucket I/O is fine for occasional metadata reads (the feed pattern above) or saving user information. It is not fine as the path your model loader streams gigabytes through every cold start.

Cache redirects

/home/user/.cache is read-only on ZeroGPU. Redirect transient caches at the top of app.py, before any library import that uses them:

import os
os.environ.setdefault("HF_HOME", "/data/.cache/huggingface")   # or /tmp on non-bucket Spaces
os.environ.setdefault("HF_MODULES_CACHE", "/tmp/hf_modules")
os.environ.setdefault("MPLCONFIGDIR", "/tmp/matplotlib")

Missing redirections fail silently or at first matplotlib / transformers / diffusers import.

Write access from the Space

The Space's HF_TOKEN secret needs write permission on the bucket. Set via Settings → Secrets in the Space UI, or hf spaces secrets set <id> HF_TOKEN=<token>.

Security note

Public bucket files are publicly accessible forever at their resolve URL. Don't write PII to a public bucket. If you need durable but private storage (e.g. per-user history requiring an HF login), keep the bucket private and gate reads through your Space's own auth.

4.3 KiB Raw Blame History