📦 deps(thirdparty): update snapshots

2026-07-01 16:02:41 +00:00
parent 8301f01888
commit c824ba9d7b
2449 changed files with 555104 additions and 9259 deletions
@@ -1,9 +1,13 @@
 ---
-source: "https://github.com/huggingface/skills/tree/main/skills/huggingface-llm-trainer"
 name: hugging-face-model-trainer
-description: Train or fine-tune TRL language models on Hugging Face Jobs, including SFT, DPO, GRPO, and GGUF export.
-license: Complete terms in LICENSE.txt
+description: Train or fine-tune language and vision models using TRL (Transformer Reinforcement Learning) or Unsloth with Hugging Face Jobs infrastructure. Covers SFT, DPO, GRPO and reward modeling training methods, plus GGUF conversion for local deployment. Includes guidance on the TRL Jobs...
 risk: unknown
+source: https://github.com/huggingface/skills/tree/main/skills/huggingface-llm-trainer
+source_repo: huggingface/skills
+source_type: official
+date_added: 2026-07-01
+license: Apache-2.0
+license_source: https://github.com/huggingface/skills/blob/main/LICENSE
 ---

 # TRL Training on Hugging Face Jobs
@@ -76,7 +80,7 @@ Before starting any training job, verify:
 - Hugging Face Account with [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan (Jobs require paid plan)
 - Authenticated login: Check with `hf_whoami()`
 - **HF_TOKEN for Hub Push** ⚠️ CRITICAL - Training environment is ephemeral, must push to Hub or ALL training results are lost
- Token must have write permissions  
+- Token must have write permissions
 - **MUST pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config** to make token available (the `$HF_TOKEN` syntax
  references your actual token value)

@@ -422,6 +426,26 @@ Before submitting:

 **On timeout:** Job killed immediately, all unsaved progress lost, must restart from beginning

+## Choose a Base Model (Model Selection)
+
+**Identify models to train based on task type or benchmark results.**
+
+Use `scripts/hf_benchmarks.py` to identify top-performing models for specific tasks. This helps the user select a model as the base for training, whilst keeping size and hardware constraints in mind.
+
+```bash
+# Get help on the benchmarks command:
+uv run scripts/hf_benchmarks.py --help
+```
+
+### Example -- choosing an OCR base model
+```bash
+# Search for benchmarks containing whose name contains the text `ocr`
+uv run scripts/hf_benchmarks.py search --query ocr
+
+# Get the ranked leaderboard for the allenai/olmOCR-bench benchmark
+uv run scripts/hf_benchmarks.py leaderboard allenai/olmOCR-bench
+```
+
 ## Cost Estimation

 **Offer to estimate cost when planning jobs with known parameters.** Use `scripts/estimate_cost.py`:
@@ -467,7 +491,7 @@ These scripts demonstrate proper Hub saving, Trackio integration, checkpoint man
 - **Space ID**: `{username}/trackio` (use "trackio" as default space name)
 - **Run naming**: Unless otherwise specified, name the run in a way the user will recognize (e.g., descriptive of the task, model, or purpose)
 - **Config**: Keep minimal - only include hyperparameters and model/dataset info
- **Project Name**: Use a Project Name to associate runs with a particular Project 
+- **Project Name**: Use a Project Name to associate runs with a particular Project

 **User overrides:** If user requests specific trackio configuration (custom space, run naming, grouping, or additional config), apply their preferences instead of defaults.

@@ -617,9 +641,9 @@ See `references/training_patterns.md` for detailed examples including:
 ### Out of Memory (OOM)

 **Fix (try in order):**
-1. Reduce batch size: `per_device_train_batch_size=1`, increase `gradient_accumulation_steps=8`. Effective batch size is `per_device_train_batch_size` x `gradient_accumulation_steps`. For best performance keep effective batch size close to 128. 
+1. Reduce batch size: `per_device_train_batch_size=1`, increase `gradient_accumulation_steps=8`. Effective batch size is `per_device_train_batch_size` x `gradient_accumulation_steps`. For best performance keep effective batch size close to 128.
 2. Enable: `gradient_checkpointing=True`
-3. Upgrade hardware: t4-small → l4x1, a10g-small → a10g-large etc. 
+3. Upgrade hardware: t4-small → l4x1, a10g-small → a10g-large etc.

 ### Dataset Misformatted

@@ -692,6 +716,7 @@ Add to PEP 723 header:
 - `scripts/unsloth_sft_example.py` - Unsloth text LLM training template (faster, less VRAM)
 - `scripts/estimate_cost.py` - Estimate time and cost (offer when appropriate)
 - `scripts/convert_to_gguf.py` - Complete GGUF conversion script
+- `scripts/hf_benchmarks.py` - Search for benchmark results and leaderboards by task, alias or free text.

 ### External Scripts
 - [Dataset Inspector](https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py) - Validate dataset format before training (use via `uv run` or `hf_jobs`)
@@ -719,6 +744,7 @@ Add to PEP 723 header:
 10. **Choose appropriate hardware** for model size; use LoRA for models >7B

 ## Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
+
+- Use this skill only when the task clearly matches its upstream product or API scope.
+- Verify commands, API behavior, pricing, quotas, credentials, and deployment effects against current official documentation before making changes.
+- Do not treat generated examples as a substitute for environment-specific tests, security review, or user approval for destructive or costly actions.
@@ -41,41 +41,16 @@ Dependencies: All required packages are declared in PEP 723 header above.
 import os
 import sys
 import torch
-import re
-import shutil
 from transformers import AutoModelForCausalLM, AutoTokenizer
 from peft import PeftModel
 from huggingface_hub import HfApi
 import subprocess

-HF_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*(/[A-Za-z0-9][A-Za-z0-9._-]*)?$")
-SAFE_FILENAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
-
-
-def require_hf_id(value, name):
-    if not HF_ID_RE.match(value or ""):
-        raise ValueError(f"{name} must be a Hugging Face model/repo id")
-    return value
-
-
-def safe_filename(value, name):
-    if not SAFE_FILENAME_RE.match(value or ""):
-        raise ValueError(f"{name} must be a safe filename segment")
-    return value
-
-
-def safe_output_file(root, filename):
-    root_path = os.path.abspath(root)
-    target = os.path.abspath(os.path.join(root_path, filename))
-    if os.path.commonpath([root_path, target]) != root_path:
-        raise ValueError(f"Output path escapes {root_path}")
-    return target
-

 def check_system_dependencies():
    """Check if required system packages are available."""
    print("🔍 Checking system dependencies...")
-    
+
    # Check for git
    if subprocess.run(["which", "git"], capture_output=True).returncode != 0:
        print("  ❌ git is not installed. Please install it:")
@@ -83,18 +58,18 @@ def check_system_dependencies():
        print("     RHEL/CentOS: sudo yum install git")
        print("     macOS: brew install git")
        return False
-    
+
    # Check for make or cmake
    has_make = subprocess.run(["which", "make"], capture_output=True).returncode == 0
    has_cmake = subprocess.run(["which", "cmake"], capture_output=True).returncode == 0
-    
+
    if not has_make and not has_cmake:
        print("  ❌ Neither make nor cmake found. Please install build tools:")
        print("     Ubuntu/Debian: sudo apt-get install build-essential cmake")
        print("     RHEL/CentOS: sudo yum groupinstall 'Development Tools' && sudo yum install cmake")
        print("     macOS: xcode-select --install && brew install cmake")
        return False
-    
+
    print("  ✅ System dependencies found")
    return True

@@ -103,19 +78,24 @@ def run_command(cmd, description):
    """Run a command with error handling."""
    print(f"   {description}...")
    try:
-        args = [str(part) for part in cmd]
-        if not args or any("\0" in part for part in args):
-            raise ValueError("Command arguments must be non-empty strings without NUL bytes")
-        executable = args[0] if os.path.isabs(args[0]) else shutil.which(args[0])
-        if not executable:
-            raise FileNotFoundError(args[0])
-        return_code = os.spawnv(os.P_WAIT, executable, args)
-        if return_code == 0:
-            return True
-        print(f"   ❌ Command failed with exit code {return_code}: {' '.join(args)}")
+        result = subprocess.run(
+            cmd,
+            check=True,
+            capture_output=True,
+            text=True
+        )
+        if result.stdout:
+            print(f"   {result.stdout[:200]}")  # Show first 200 chars
+        return True
+    except subprocess.CalledProcessError as e:
+        print(f"   ❌ Command failed: {' '.join(cmd)}")
+        if e.stdout:
+            print(f"   STDOUT: {e.stdout[:500]}")
+        if e.stderr:
+            print(f"   STDERR: {e.stderr[:500]}")
        return False
-    except (FileNotFoundError, OSError, ValueError) as e:
-        print(f"   ❌ Command failed: {e}")
+    except FileNotFoundError:
+        print(f"   ❌ Command not found: {cmd[0]}")
        return False


@@ -128,11 +108,10 @@ if not check_system_dependencies():
    sys.exit(1)

 # Configuration from environment variables
-ADAPTER_MODEL = require_hf_id(os.environ.get("ADAPTER_MODEL", "evalstate/qwen-capybara-medium"), "ADAPTER_MODEL")
-BASE_MODEL = require_hf_id(os.environ.get("BASE_MODEL", "Qwen/Qwen2.5-0.5B"), "BASE_MODEL")
-OUTPUT_REPO = require_hf_id(os.environ.get("OUTPUT_REPO", "evalstate/qwen-capybara-medium-gguf"), "OUTPUT_REPO")
-username = require_hf_id(os.environ.get("HF_USERNAME", ADAPTER_MODEL.split('/')[0]), "HF_USERNAME")
-TRUST_REMOTE_CODE = os.environ.get("TRUST_REMOTE_CODE", "").strip().lower() in {"1", "true", "yes"}
+ADAPTER_MODEL = os.environ.get("ADAPTER_MODEL", "evalstate/qwen-capybara-medium")
+BASE_MODEL = os.environ.get("BASE_MODEL", "Qwen/Qwen2.5-0.5B")
+OUTPUT_REPO = os.environ.get("OUTPUT_REPO", "evalstate/qwen-capybara-medium-gguf")
+username = os.environ.get("HF_USERNAME", ADAPTER_MODEL.split('/')[0])

 print(f"\n📦 Configuration:")
 print(f"   Base model: {BASE_MODEL}")
@@ -148,7 +127,7 @@ try:
        BASE_MODEL,
        dtype=torch.float16,
        device_map="auto",
-        trust_remote_code=TRUST_REMOTE_CODE,
+        trust_remote_code=True,
    )
    print("   ✅ Base model loaded")
 except Exception as e:
@@ -170,7 +149,7 @@ except Exception as e:

 try:
    # Load tokenizer
-    tokenizer = AutoTokenizer.from_pretrained(ADAPTER_MODEL, trust_remote_code=TRUST_REMOTE_CODE)
+    tokenizer = AutoTokenizer.from_pretrained(ADAPTER_MODEL, trust_remote_code=True)
    print("   ✅ Tokenizer loaded")
 except Exception as e:
    print(f"   ❌ Failed to load tokenizer: {e}")
@@ -224,8 +203,7 @@ os.makedirs(gguf_output_dir, exist_ok=True)

 convert_script = "/tmp/llama.cpp/convert_hf_to_gguf.py"
 model_name = ADAPTER_MODEL.split('/')[-1]
-model_name = safe_filename(model_name, "model_name")
-gguf_file = safe_output_file(gguf_output_dir, f"{model_name}-f16.gguf")
+gguf_file = f"{gguf_output_dir}/{model_name}-f16.gguf"

 print(f"   Running conversion...")
 if not run_command(
@@ -281,7 +259,7 @@ quant_formats = [
 quantized_files = []
 for quant_type, description in quant_formats:
    print(f"   Creating {quant_type} quantization ({description})...")
-    quant_file = safe_output_file(gguf_output_dir, f"{model_name}-{quant_type.lower()}.gguf")
+    quant_file = f"{gguf_output_dir}/{model_name}-{quant_type.lower()}.gguf"

    if not run_command(
        [quantize_bin, gguf_file, quant_file, quant_type],
@@ -291,7 +269,7 @@ for quant_type, description in quant_formats:
        continue

    quantized_files.append((quant_file, quant_type))
-    
+
    # Get file size
    size_mb = os.path.getsize(quant_file) / (1024 * 1024)
    print(f"   ✅ {quant_type}: {size_mb:.1f} MB")
@@ -0,0 +1,659 @@
+#!/usr/bin/env -S uv run --script
+# /// script
+# requires-python = ">=3.10"
+# dependencies = []
+# ///
+"""
+Search Hugging Face benchmark datasets and fetch leaderboard results.
+
+This script is designed to be pipeline-friendly:
+  - search benchmark datasets by free text, alias, task, and modality
+  - fetch dataset leaderboards in normalized JSON / NDJSON / table form
+  - optionally read dataset ids from stdin for chaining
+
+It uses HF_TOKEN automatically when present.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import sys
+import textwrap
+import urllib.error
+import urllib.parse
+import urllib.request
+from typing import Any, Iterable
+
+
+BASE_URL = "https://huggingface.co"
+DEFAULT_TIMEOUT = 30
+
+ALIASES: dict[str, list[str]] = {
+    "ocr": [
+        "ocr",
+        "olmocr",
+        "pdf",
+        "image-to-text",
+        "screen",
+        "screenspot",
+        "markdown",
+        "text recognition",
+    ],
+    "coding": [
+        "code",
+        "coding",
+        "software engineering",
+        "programming",
+        "swe",
+        "terminal",
+        "patch",
+        "bug",
+        "cuda",
+    ],
+    "math": [
+        "math",
+        "reasoning",
+        "gsm8k",
+        "mmlu",
+        "gpqa",
+        "aime",
+        "hmmt",
+    ],
+    "retrieval": [
+        "retrieval",
+        "search",
+        "mteb",
+        "arguana",
+        "bright",
+    ],
+    "agents": [
+        "agent",
+        "agents",
+        "terminal",
+        "screen",
+        "computer use",
+        "tool use",
+    ],
+    "asr": [
+        "asr",
+        "speech",
+        "audio",
+        "transcribe",
+        "transcription",
+    ],
+}
+
+
+class HfApiError(RuntimeError):
+    pass
+
+
+class FullHelpArgumentParser(argparse.ArgumentParser):
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        super().__init__(*args, **kwargs)
+        self._search_parser: argparse.ArgumentParser | None = None
+        self._leaderboard_parser: argparse.ArgumentParser | None = None
+
+    def format_help(self) -> str:
+        text = super().format_help()
+        extra_sections: list[str] = []
+
+        if self._search_parser is not None:
+            extra_sections.append(
+                "\nsearch command options:\n"
+                + textwrap.indent(self._search_parser.format_help().strip(), "  ")
+            )
+
+        if self._leaderboard_parser is not None:
+            extra_sections.append(
+                "\nleaderboard command options:\n"
+                + textwrap.indent(self._leaderboard_parser.format_help().strip(), "  ")
+            )
+
+        if extra_sections:
+            text += "\n" + "\n".join(extra_sections) + "\n"
+        return text
+
+
+def auth_headers() -> dict[str, str]:
+    token = os.getenv("HF_TOKEN")
+    return {"Authorization": f"Bearer {token}"} if token else {}
+
+
+def http_get_json(path: str, params: dict[str, Any] | None = None) -> Any:
+    url = f"{BASE_URL}{path}"
+    if params:
+        pairs: list[tuple[str, str]] = []
+        for key, value in params.items():
+            if value is None:
+                continue
+            if isinstance(value, (list, tuple)):
+                for item in value:
+                    pairs.append((key, str(item)))
+            else:
+                pairs.append((key, str(value)))
+        if pairs:
+            url = f"{url}?{urllib.parse.urlencode(pairs)}"
+
+    req = urllib.request.Request(url, headers=auth_headers())
+    try:
+        with urllib.request.urlopen(req, timeout=DEFAULT_TIMEOUT) as resp:
+            return json.loads(resp.read().decode("utf-8"))
+    except urllib.error.HTTPError as exc:
+        body = exc.read().decode("utf-8", errors="replace")
+        raise HfApiError(f"{exc.code} {exc.reason} for {url}: {body[:500]}") from exc
+    except urllib.error.URLError as exc:
+        raise HfApiError(f"Request failed for {url}: {exc}") from exc
+
+
+def shorten(text: str, width: int) -> str:
+    text = " ".join((text or "").split())
+    if len(text) <= width:
+        return text
+    return text[: max(0, width - 1)] + "…"
+
+
+def first_text(value: Any) -> str:
+    if value is None:
+        return ""
+    if isinstance(value, str):
+        return value
+    if isinstance(value, list):
+        return " ".join(first_text(v) for v in value)
+    if isinstance(value, dict):
+        return " ".join(first_text(v) for v in value.values())
+    return str(value)
+
+
+def benchmark_catalog(limit: int = 500) -> list[dict[str, Any]]:
+    data = http_get_json(
+        "/api/datasets",
+        params={"filter": "benchmark:official", "limit": limit, "full": "true"},
+    )
+    if not isinstance(data, list):
+        raise HfApiError("Unexpected response while listing benchmark datasets")
+    return data
+
+
+def dataset_search_blob(dataset: dict[str, Any]) -> str:
+    card = dataset.get("cardData") or {}
+    parts = [
+        dataset.get("id", ""),
+        dataset.get("description", ""),
+        first_text(dataset.get("tags")),
+        first_text(card.get("pretty_name")),
+        first_text(card.get("tags")),
+        first_text(card.get("task_categories")),
+        first_text(card.get("task_ids")),
+    ]
+    return " ".join(parts).lower()
+
+
+def dataset_search_fields(dataset: dict[str, Any]) -> dict[str, str]:
+    card = dataset.get("cardData") or {}
+    return {
+        "id": first_text(dataset.get("id")).lower(),
+        "pretty_name": first_text(card.get("pretty_name")).lower(),
+        "tags": " ".join(
+            [
+                first_text(dataset.get("tags")),
+                first_text(card.get("tags")),
+                first_text(card.get("task_categories")),
+                first_text(card.get("task_ids")),
+                first_text(card.get("modality")),
+            ]
+        ).lower(),
+        "description": first_text(dataset.get("description")).lower(),
+    }
+
+
+def collect_prefixed_tags(dataset: dict[str, Any], prefixes: Iterable[str]) -> list[str]:
+    prefixes = tuple(prefixes)
+    tags = dataset.get("tags") or []
+    card = dataset.get("cardData") or {}
+
+    out: list[str] = []
+    for tag in tags:
+        if isinstance(tag, str) and tag.startswith(prefixes):
+            out.append(tag)
+
+    for key, prefix in (
+        ("task_categories", "task_categories:"),
+        ("task_ids", "task_ids:"),
+        ("modality", "modality:"),
+    ):
+        values = card.get(key)
+        if isinstance(values, list):
+            for value in values:
+                full_tag = f"{prefix}{value}"
+                if full_tag.startswith(prefixes):
+                    out.append(full_tag)
+
+    deduped: list[str] = []
+    seen: set[str] = set()
+    for tag in out:
+        if tag not in seen:
+            deduped.append(tag)
+            seen.add(tag)
+    return deduped
+
+
+def expand_aliases(aliases: list[str]) -> dict[str, list[str]]:
+    expanded: dict[str, list[str]] = {}
+    for alias in aliases:
+        terms = ALIASES.get(alias.lower(), [alias])
+        expanded[alias] = terms
+    return expanded
+
+
+def matches_term(blob: str, term: str) -> bool:
+    candidate = term.lower().strip()
+    if not candidate:
+        return False
+    if re.fullmatch(r"[a-z0-9_]+", candidate):
+        return re.search(rf"(?<![a-z0-9_]){re.escape(candidate)}(?![a-z0-9_])", blob) is not None
+    return candidate in blob
+
+
+def score_dataset(
+    dataset: dict[str, Any],
+    queries: list[str],
+    aliases: dict[str, list[str]],
+    tasks: list[str],
+    modalities: list[str],
+) -> dict[str, Any]:
+    blob = dataset_search_blob(dataset)
+    fields = dataset_search_fields(dataset)
+    task_tags = collect_prefixed_tags(dataset, ["task_categories:", "task_ids:"])
+    modality_tags = collect_prefixed_tags(dataset, ["modality:"])
+
+    score = 0
+    reasons: list[str] = []
+
+    for query in queries:
+        q = query.lower().strip()
+        if q and any(matches_term(value, q) for value in fields.values()):
+            score += 3
+            reasons.append(f"query:{query}")
+
+    for alias_name, terms in aliases.items():
+        matched_terms: list[str] = []
+        alias_score = 0
+        for term in terms:
+            strong_match = any(
+                matches_term(fields[field_name], term)
+                for field_name in ("id", "pretty_name", "tags")
+            )
+            desc_match = matches_term(fields["description"], term)
+            if strong_match:
+                alias_score += 2
+                matched_terms.append(term)
+            elif desc_match:
+                alias_score += 1
+                matched_terms.append(term)
+        if matched_terms:
+            score += alias_score
+            reasons.append(f"alias:{alias_name}=" + ",".join(matched_terms[:5]))
+
+    lower_task_tags = [t.lower() for t in task_tags]
+    for task in tasks:
+        task = task.lower().strip()
+        if not task:
+            continue
+        exact = [
+            tag
+            for tag in lower_task_tags
+            if tag == f"task_categories:{task}" or tag == f"task_ids:{task}"
+        ]
+        fuzzy = matches_term(blob, task)
+        if exact:
+            score += 5
+            reasons.append(f"task:{task}")
+        elif fuzzy:
+            score += 2
+            reasons.append(f"task~:{task}")
+
+    lower_modality_tags = [m.lower() for m in modality_tags]
+    for modality in modalities:
+        modality = modality.lower().strip()
+        if not modality:
+            continue
+        if f"modality:{modality}" in lower_modality_tags:
+            score += 4
+            reasons.append(f"modality:{modality}")
+
+    return {
+        "dataset_id": dataset.get("id"),
+        "score": score,
+        "reasons": reasons,
+        "task_tags": task_tags,
+        "modality_tags": modality_tags,
+        "benchmark_tags": collect_prefixed_tags(dataset, ["benchmark:"]),
+        "pretty_name": (dataset.get("cardData") or {}).get("pretty_name"),
+        "downloads": dataset.get("downloads"),
+        "description": " ".join((dataset.get("description") or "").split()),
+    }
+
+
+def search_benchmarks(
+    queries: list[str],
+    aliases: list[str],
+    tasks: list[str],
+    modalities: list[str],
+    limit: int,
+) -> list[dict[str, Any]]:
+    datasets = benchmark_catalog(limit=500)
+    alias_map = expand_aliases(aliases)
+
+    results = [score_dataset(ds, queries, alias_map, tasks, modalities) for ds in datasets]
+
+    active_filters = bool(queries or aliases or tasks or modalities)
+    if active_filters:
+        results = [row for row in results if row["score"] >= 2]
+
+    results.sort(
+        key=lambda row: (
+            -row["score"],
+            -(row["downloads"] or 0),
+            row["dataset_id"] or "",
+        )
+    )
+    return results[:limit]
+
+
+def parse_repo_id(repo_id: str) -> tuple[str, str]:
+    if "/" not in repo_id:
+        raise ValueError(f"Expected <namespace>/<repo>, got: {repo_id}")
+    namespace, repo = repo_id.split("/", 1)
+    return namespace, repo
+
+
+def get_leaderboard(repo_id: str, task_id: str | None = None) -> list[dict[str, Any]]:
+    namespace, repo = parse_repo_id(repo_id)
+    params = {"task_id": task_id} if task_id else None
+    data = http_get_json(f"/api/datasets/{namespace}/{repo}/leaderboard", params=params)
+    if not isinstance(data, list):
+        raise HfApiError(f"Unexpected leaderboard response for {repo_id}")
+
+    normalized: list[dict[str, Any]] = []
+    for row in data:
+        source = row.get("source") or {}
+        normalized.append(
+            {
+                "dataset_id": repo_id,
+                "task_id": task_id,
+                "rank": row.get("rank"),
+                "model_id": row.get("modelId"),
+                "value": row.get("value"),
+                "verified": row.get("verified"),
+                "lower_is_better": row.get("lower_is_better"),
+                "filename": row.get("filename"),
+                "notes": row.get("notes"),
+                "pull_request": row.get("pullRequest"),
+                "source_name": source.get("name"),
+                "source_url": source.get("url"),
+                "source_is_external": source.get("isExternal"),
+            }
+        )
+    return normalized
+
+
+def read_repo_ids_from_stdin() -> list[str]:
+    if sys.stdin.isatty():
+        return []
+
+    repo_ids: list[str] = []
+    for raw_line in sys.stdin:
+        line = raw_line.strip()
+        if not line:
+            continue
+        if line.startswith("{"):
+            try:
+                obj = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            candidate = obj.get("dataset_id") or obj.get("id")
+            if isinstance(candidate, str) and "/" in candidate:
+                repo_ids.append(candidate)
+            continue
+        if "/" in line:
+            repo_ids.append(line)
+    return repo_ids
+
+
+def print_json(data: Any) -> None:
+    json.dump(data, sys.stdout, indent=2, ensure_ascii=False)
+    sys.stdout.write("\n")
+
+
+def print_ndjson(rows: list[dict[str, Any]]) -> None:
+    for row in rows:
+        sys.stdout.write(json.dumps(row, ensure_ascii=False) + "\n")
+
+
+def print_search_table(rows: list[dict[str, Any]]) -> None:
+    if not rows:
+        print("No benchmark datasets matched.")
+        return
+
+    headers = ["dataset_id", "score", "modalities", "tasks", "reasons", "description"]
+    widths = [34, 5, 18, 24, 30, 68]
+    print("  ".join(h.ljust(w) for h, w in zip(headers, widths)))
+    print("  ".join("-" * w for w in widths))
+    for row in rows:
+        values = [
+            shorten(row.get("dataset_id") or "", widths[0]),
+            str(row.get("score", "")),
+            shorten(", ".join(row.get("modality_tags") or []), widths[2]),
+            shorten(", ".join(row.get("task_tags") or []), widths[3]),
+            shorten(", ".join(row.get("reasons") or []), widths[4]),
+            shorten(row.get("description") or "", widths[5]),
+        ]
+        print("  ".join(v.ljust(w) for v, w in zip(values, widths)))
+
+
+def print_leaderboard_table(rows: list[dict[str, Any]]) -> None:
+    if not rows:
+        print("No leaderboard rows returned.")
+        return
+
+    headers = ["dataset_id", "rank", "model_id", "value", "verified", "source"]
+    widths = [30, 5, 38, 10, 8, 28]
+    print("  ".join(h.ljust(w) for h, w in zip(headers, widths)))
+    print("  ".join("-" * w for w in widths))
+    for row in rows:
+        values = [
+            shorten(str(row.get("dataset_id") or ""), widths[0]),
+            str(row.get("rank") or ""),
+            shorten(str(row.get("model_id") or ""), widths[2]),
+            shorten(str(row.get("value") or ""), widths[3]),
+            str(row.get("verified")),
+            shorten(str(row.get("source_name") or ""), widths[5]),
+        ]
+        print("  ".join(v.ljust(w) for v, w in zip(values, widths)))
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = FullHelpArgumentParser(
+        prog="hf_benchmarks.py",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        description=textwrap.dedent(
+            """
+            Search benchmark datasets and fetch leaderboard results from the Hugging Face Hub.
+
+            Workflow ideas:
+              1) Discover candidate benchmarks:
+                   hf_benchmarks.py search --alias ocr
+                   hf_benchmarks.py search --alias coding
+                   hf_benchmarks.py search --task image-to-text --modality document
+
+              2) Inspect a leaderboard:
+                   hf_benchmarks.py leaderboard allenai/olmOCR-bench --top 10
+
+              3) Chain search -> leaderboard:
+                   hf_benchmarks.py search --alias coding --format ndjson \\
+                     | hf_benchmarks.py leaderboard --stdin --top 5 --format table
+            """
+        ),
+    )
+
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    search_parser = subparsers.add_parser(
+        "search",
+        help="Search benchmark datasets by query, alias, task, and modality",
+    )
+    search_parser.add_argument(
+        "--query",
+        action="append",
+        default=[],
+        help="Free-text query to match against benchmark dataset metadata. Repeatable.",
+    )
+    search_parser.add_argument(
+        "--alias",
+        action="append",
+        default=[],
+        help=(
+            "Convenience alias for common benchmark domains. Known aliases: "
+            + ", ".join(sorted(ALIASES))
+            + ". Repeatable."
+        ),
+    )
+    search_parser.add_argument(
+        "--task",
+        action="append",
+        default=[],
+        help="Task to match, e.g. text-generation, image-to-text, question-answering. Repeatable.",
+    )
+    search_parser.add_argument(
+        "--modality",
+        action="append",
+        default=[],
+        help="Modality to match, e.g. text, image, document, audio. Repeatable.",
+    )
+    search_parser.add_argument(
+        "--limit",
+        type=int,
+        default=20,
+        help="Maximum number of rows to print (default: 20).",
+    )
+    search_parser.add_argument(
+        "--format",
+        choices=["table", "json", "ndjson"],
+        default="table",
+        help="Output format (default: table).",
+    )
+
+    leaderboard_parser = subparsers.add_parser(
+        "leaderboard",
+        help="Fetch normalized leaderboard rows for one or more benchmark datasets",
+    )
+    leaderboard_parser.add_argument(
+        "datasets",
+        nargs="*",
+        help="Dataset repo ids (<namespace>/<repo>). Can also be supplied via stdin with --stdin.",
+    )
+    leaderboard_parser.add_argument(
+        "--stdin",
+        action="store_true",
+        help="Read dataset ids from stdin. Accepts plain repo ids or NDJSON with dataset_id/id fields.",
+    )
+    leaderboard_parser.add_argument(
+        "--task-id",
+        default=None,
+        help="Optional leaderboard task_id query parameter.",
+    )
+    leaderboard_parser.add_argument(
+        "--top",
+        type=int,
+        default=None,
+        help="Only keep the top N results per leaderboard.",
+    )
+    leaderboard_parser.add_argument(
+        "--format",
+        choices=["table", "json", "ndjson"],
+        default="table",
+        help="Output format (default: table).",
+    )
+
+    parser._search_parser = search_parser
+    parser._leaderboard_parser = leaderboard_parser
+
+    return parser
+
+
+def run_search(args: argparse.Namespace) -> int:
+    rows = search_benchmarks(
+        queries=args.query,
+        aliases=args.alias,
+        tasks=args.task,
+        modalities=args.modality,
+        limit=args.limit,
+    )
+
+    if args.format == "json":
+        print_json(rows)
+    elif args.format == "ndjson":
+        print_ndjson(rows)
+    else:
+        print_search_table(rows)
+    return 0
+
+
+def run_leaderboard(args: argparse.Namespace) -> int:
+    repo_ids = list(args.datasets)
+    if args.stdin:
+        repo_ids.extend(read_repo_ids_from_stdin())
+
+    deduped: list[str] = []
+    seen: set[str] = set()
+    for repo_id in repo_ids:
+        if repo_id not in seen:
+            deduped.append(repo_id)
+            seen.add(repo_id)
+    repo_ids = deduped
+
+    if not repo_ids:
+        print("Error: provide dataset ids or use --stdin.", file=sys.stderr)
+        return 2
+
+    rows: list[dict[str, Any]] = []
+    for repo_id in repo_ids:
+        dataset_rows = get_leaderboard(repo_id, task_id=args.task_id)
+        if args.top is not None:
+            dataset_rows = dataset_rows[: args.top]
+        rows.extend(dataset_rows)
+
+    if args.format == "json":
+        print_json(rows)
+    elif args.format == "ndjson":
+        print_ndjson(rows)
+    else:
+        print_leaderboard_table(rows)
+    return 0
+
+
+def main() -> int:
+    parser = build_parser()
+    args = parser.parse_args()
+
+    try:
+        if args.command == "search":
+            return run_search(args)
+        if args.command == "leaderboard":
+            return run_leaderboard(args)
+        parser.error(f"Unknown command: {args.command}")
+        return 2
+    except HfApiError as exc:
+        print(f"Error: {exc}", file=sys.stderr)
+        return 1
+    except ValueError as exc:
+        print(f"Error: {exc}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())