📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,107 @@
|
||||
---
|
||||
name: hasdata
|
||||
description: Use HasData APIs for web scraping and structured web data extraction.
|
||||
risk: safe
|
||||
source: official
|
||||
source_type: official
|
||||
source_repo: HasData/hasdata-cli
|
||||
license: MIT
|
||||
license_source: "https://github.com/HasData/hasdata-cli/blob/main/LICENSE"
|
||||
date_added: "2026-06-04"
|
||||
---
|
||||
|
||||
# HasData
|
||||
|
||||
Cloud platform for extracting public web data. One API key, three execution modes. All endpoints sit under `https://api.hasdata.com` and authenticate with `x-api-key`.
|
||||
|
||||
```bash
|
||||
curl -G 'https://api.hasdata.com/scrape/google/serp' \
|
||||
--data-urlencode 'q=coffee' \
|
||||
-H 'x-api-key: <your-api-key>'
|
||||
```
|
||||
|
||||
`401` invalid key, `403` quota exhausted, `429` concurrency cap, `500` server error (retry).
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when:
|
||||
|
||||
- The user needs web scraping.
|
||||
- The user needs search engine results.
|
||||
- The user needs structured data extraction.
|
||||
- The user needs ecommerce, travel, jobs, or local business data.
|
||||
- The user explicitly asks about HasData.
|
||||
|
||||
## Three execution modes
|
||||
|
||||
| Mode | Latency | When | Endpoint |
|
||||
|---|---|---|---|
|
||||
| **Web Scraping API** | seconds | Arbitrary URL — JS rendering, CSS/AI extraction, screenshots | `POST /scrape/web` |
|
||||
| **Scraper APIs** (sync) | seconds | Pre-parsed JSON for known platforms (Google, Amazon, Zillow, …) | `GET /scrape/<vertical>/<resource>` |
|
||||
| **Scraper Jobs** (async) | minutes–hours | Bulk extraction, recursive crawling, webhook fan-out | `POST /scrapers/<slug>/jobs` |
|
||||
|
||||
**Decision rule.** Default to a **Scraper API** when one exists for the platform (pre-parsed JSON, no selector maintenance). Use **Web Scraping** for arbitrary URLs not covered by an API. Reach for a **Scraper Job** only when no API equivalent exists — `crawler`, `contacts`, `sec-edgar`, `amazon-bestsellers`, `amazon-product-reviews` — *or* when async fan-out + webhooks save engineering time over a paginated client loop.
|
||||
|
||||
## Always-true response shape
|
||||
|
||||
```json
|
||||
{ "requestMetadata": { "id": "…", "status": "ok", "url": "…" }, "...": "endpoint-specific" }
|
||||
```
|
||||
|
||||
Treat data as valid only if `requestMetadata.status === "ok"`. HTTP 200 alone isn't enough.
|
||||
|
||||
## High-leverage patterns
|
||||
|
||||
- **SERP-first enrichment.** Google SERP can surface public snippets for company and professional-profile lookup. Use it for business or authorized research, avoid unnecessary direct scraping, and treat personal email/phone lookup as allowed only with a legitimate purpose and user authorization.
|
||||
- **AI Mode + verify.** `/scrape/google/ai-mode` for the answer + references → `/scrape/web` (markdown) on each reference URL → cited RAG context, no vector DB.
|
||||
- **Maps → leads.** `/scrape/google-maps/search` returns business websites and phones; collect contact details only from public, permitted sources and apply opt-out, rate, and privacy-law constraints before any outreach use.
|
||||
- **Crawler → corpus.** `crawler` Scraper Job with `outputFormat: ["markdown"]` + `includePaths: "/docs/.+"` produces an LLM-ready corpus in one submission.
|
||||
- **Pre-extracted via SERP rich snippets.** `knowledgeGraph`, `localResults`, `inlineShoppingResults`, `relatedQuestions` carry pre-parsed public facts. Always check them before considering direct page access.
|
||||
|
||||
## When to call from code (the wiring)
|
||||
|
||||
- **Auth:** `x-api-key` header on every request. Read from `HASDATA_API_KEY` env. Never hardcode, never log.
|
||||
- **Timeouts:** **set client timeout ≥ 300 s.** HasData's own deadline is 300 s; shorter clients produce phantom failures while still being billed on completion.
|
||||
- **Retries:** `429` and `5xx` only — exponential backoff, jitter. Never retry `4xx` (auth, validation).
|
||||
- **Concurrency:** cap at your plan limit. The free tier is 1; anything higher just generates `429`s.
|
||||
- **Async jobs:** the submit response handle is `body.id` (integer), **not `jobId`**. Persist it immediately. Poll `GET /scrapers/jobs/<id>` every 10–30 s with backoff; treat webhooks as best-effort and always pair with polling. On `finished` the status carries `data: {csv, json, xlsx}` short-lived URLs — download immediately.
|
||||
|
||||
See `references/code-recipes.md` for ready-to-paste Python and TypeScript clients with retry, backoff, bounded concurrency, and the full job lifecycle.
|
||||
|
||||
## Common gotchas
|
||||
|
||||
- **300 s server deadline.** Match client timeout.
|
||||
- **Disable `jsRendering` first**, enable only if the page needs it — most static pages parse fine without a headless browser.
|
||||
- **No `cookies` parameter** — cookies go through `headers["Cookie"]`.
|
||||
- **`includePaths` regex is case-sensitive.** `/blog/.+` won't match `/Blog/...`.
|
||||
- **Scraper Job `data` is double-wrapped.** Each row is `body.data[i].data`; outer wraps with `id`, `jobId`, `dataId`, `createdAt`, `updatedAt`.
|
||||
- **`requestMetadata.status === "ok"` is the only success signal.** HTTP 200 alone isn't enough.
|
||||
- **Webhooks are best-effort with 3 retries.** Always have a polling fallback.
|
||||
|
||||
## References
|
||||
|
||||
- [`references/web-scraping.md`](references/web-scraping.md) — `POST /scrape/web` parameters, JS scenarios, AI extraction, cookie auth.
|
||||
- [`references/search.md`](references/search.md) — Google SERP / Light / AI Mode / News / Shopping / Bing / Trends + pagination.
|
||||
- [`references/ecommerce.md`](references/ecommerce.md) — Amazon (product, search, seller, seller-products) and Shopify.
|
||||
- [`references/real-estate.md`](references/real-estate.md) — Zillow, Redfin (bracketed filters).
|
||||
- [`references/travel.md`](references/travel.md) — Airbnb, Booking, Google Flights (occupancy rules, token pagination, IATA codes).
|
||||
- [`references/local-business.md`](references/local-business.md) — Maps (search/place/reviews/photos/posts), Yelp, YellowPages.
|
||||
- [`references/jobs.md`](references/jobs.md) — Indeed and Glassdoor.
|
||||
- [`references/youtube.md`](references/youtube.md) — YouTube search / video / channel / transcript.
|
||||
- [`references/scraper-jobs.md`](references/scraper-jobs.md) — async submit/poll/results, Crawler, Contacts, SEC EDGAR, webhook receiver.
|
||||
- [`references/code-recipes.md`](references/code-recipes.md) — Python / TypeScript clients with retry, backoff, concurrency, polling.
|
||||
|
||||
## Resources
|
||||
|
||||
- Sitemap: <https://docs.hasdata.com/llms.txt>
|
||||
- API status codes: <https://docs.hasdata.com/api-codes>
|
||||
- Credits & concurrency: <https://docs.hasdata.com/credits-and-concurrency>
|
||||
- Dashboard: <https://app.hasdata.com>
|
||||
|
||||
## Limitations
|
||||
|
||||
* Requires access to HasData services and valid credentials.
|
||||
* Data quality and available fields depend on the target website and extraction method used.
|
||||
* JavaScript-heavy websites may require rendering, which can affect performance and cost.
|
||||
* Use only for public data or content the user is authorized to access; respect site terms, robots/access controls, privacy law, and rate limits.
|
||||
* Rate limits, quotas, and account restrictions may apply depending on the endpoint and subscription plan.
|
||||
@@ -0,0 +1,150 @@
|
||||
# Code recipes — wiring HasData into your code
|
||||
|
||||
## Ground rules
|
||||
|
||||
- **Base URL:** `https://api.hasdata.com`. Header `x-api-key` on every request.
|
||||
- **Methods:** Scraper APIs are `GET`; Web Scraping is `POST`; Scraper Jobs use `POST` (submit) + `GET` (status/results) + `DELETE` (stop).
|
||||
- **Key handling:** read from env (`HASDATA_API_KEY`). Never hardcode, never log.
|
||||
- **Timeouts:** **client timeout ≥ 300 s.** HasData's deadline is 300 s; shorter clients get phantom failures while still being billed.
|
||||
- **Retries:** `429` and `5xx` only with exponential backoff + jitter. Never retry `4xx`.
|
||||
- **Concurrency:** cap at plan limit. Free tier = 1.
|
||||
- **Success signal:** sync APIs require `body.requestMetadata.status === "ok"`. HTTP 200 alone isn't enough.
|
||||
|
||||
## Status codes
|
||||
|
||||
| Code | Meaning | Action |
|
||||
|---|---|---|
|
||||
| 200 + `status:"ok"` | OK | Use body |
|
||||
| 401 | Bad/missing key | Fix — don't retry |
|
||||
| 403 | Quota exhausted | Don't retry |
|
||||
| 429 | Concurrency cap | Backoff + retry |
|
||||
| 500 | Server error | Retry |
|
||||
|
||||
## Python — minimal client
|
||||
|
||||
```python
|
||||
import os, requests
|
||||
|
||||
class HasData:
|
||||
BASE = "https://api.hasdata.com"
|
||||
|
||||
def __init__(self, api_key=None, timeout=300):
|
||||
self.s = requests.Session()
|
||||
self.s.headers["x-api-key"] = api_key or os.environ["HASDATA_API_KEY"]
|
||||
self.timeout = timeout
|
||||
|
||||
def get(self, path, **params):
|
||||
r = self.s.get(f"{self.BASE}{path}", params=params, timeout=self.timeout)
|
||||
r.raise_for_status()
|
||||
body = r.json()
|
||||
if body.get("requestMetadata", {}).get("status") != "ok":
|
||||
raise RuntimeError(f"hasdata not-ok: {body.get('requestMetadata')}")
|
||||
return body
|
||||
|
||||
def post(self, path, body):
|
||||
r = self.s.post(f"{self.BASE}{path}", json=body, timeout=self.timeout)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
hd = HasData()
|
||||
serp = hd.get("/scrape/google/serp", q="coffee", num=20)["organicResults"]
|
||||
md = hd.post("/scrape/web", {"url": "https://example.com", "outputFormat": ["markdown"]})["markdown"]
|
||||
```
|
||||
|
||||
## Python — retry + bounded concurrency
|
||||
|
||||
```python
|
||||
import time, random
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from requests import HTTPError
|
||||
|
||||
def with_retry(fn, attempts=5, base=1.0, cap=60.0):
|
||||
for i in range(attempts):
|
||||
try:
|
||||
return fn()
|
||||
except HTTPError as e:
|
||||
code = e.response.status_code
|
||||
if code == 429 or 500 <= code < 600:
|
||||
time.sleep(min(cap, base * 2 ** i) + random.random())
|
||||
continue
|
||||
raise
|
||||
raise RuntimeError("retry exhausted")
|
||||
|
||||
def scrape_many(urls, workers=5):
|
||||
out = {}
|
||||
with ThreadPoolExecutor(max_workers=workers) as ex:
|
||||
futs = {ex.submit(lambda u=u: hd.post("/scrape/web", {"url": u, "outputFormat": ["markdown"]})): u
|
||||
for u in urls}
|
||||
for f in as_completed(futs):
|
||||
try:
|
||||
out[futs[f]] = f.result().get("markdown")
|
||||
except Exception as e:
|
||||
out[futs[f]] = e
|
||||
return out
|
||||
```
|
||||
|
||||
Cap `workers` at your plan's concurrency — anything higher just generates `429`s.
|
||||
|
||||
## TypeScript — minimal client
|
||||
|
||||
```typescript
|
||||
const BASE = "https://api.hasdata.com";
|
||||
const KEY = process.env.HASDATA_API_KEY!;
|
||||
|
||||
async function get<T = any>(path: string, params: Record<string, string | number> = {}): Promise<T> {
|
||||
const qs = new URLSearchParams(Object.entries(params).map(([k, v]) => [k, String(v)]));
|
||||
const r = await fetch(`${BASE}${path}?${qs}`, {
|
||||
headers: { "x-api-key": KEY },
|
||||
signal: AbortSignal.timeout(300_000),
|
||||
});
|
||||
if (!r.ok) throw new Error(`HasData ${r.status} ${await r.text()}`);
|
||||
const body = await r.json() as any;
|
||||
if (body?.requestMetadata?.status && body.requestMetadata.status !== "ok") {
|
||||
throw new Error(`HasData not-ok: ${JSON.stringify(body.requestMetadata)}`);
|
||||
}
|
||||
return body as T;
|
||||
}
|
||||
|
||||
async function post<T = any>(path: string, body: unknown): Promise<T> {
|
||||
const r = await fetch(`${BASE}${path}`, {
|
||||
method: "POST",
|
||||
headers: { "x-api-key": KEY, "Content-Type": "application/json" },
|
||||
body: JSON.stringify(body),
|
||||
signal: AbortSignal.timeout(300_000),
|
||||
});
|
||||
if (!r.ok) throw new Error(`HasData ${r.status} ${await r.text()}`);
|
||||
return r.json() as Promise<T>;
|
||||
}
|
||||
|
||||
// Bounded concurrency, no deps
|
||||
async function pool<T, R>(items: T[], n: number, fn: (x: T) => Promise<R>) {
|
||||
const out: R[] = []; let i = 0;
|
||||
await Promise.all(Array.from({ length: n }, async () => {
|
||||
while (i < items.length) { const k = i++; out[k] = await fn(items[k]); }
|
||||
}));
|
||||
return out;
|
||||
}
|
||||
```
|
||||
|
||||
## Pagination cheat sheet
|
||||
|
||||
| Endpoint family | Pagination |
|
||||
|---|---|
|
||||
| Google SERP / Light SERP / Bing | `start` + `num` (max 100) |
|
||||
| Google Maps Search | `start` (steps of 20) |
|
||||
| Yelp Search | `start` (steps of 10) |
|
||||
| Google Maps Reviews / Glassdoor / Airbnb | `nextPageToken` |
|
||||
| Indeed / YellowPages / Amazon Search | `start` or `page` |
|
||||
| Shopify Products | `page` (with `limit` ≤ 250) |
|
||||
| Scraper-Job results | `page` + `limit` (max 100) until `meta.currentPage >= meta.lastPage` |
|
||||
|
||||
## Pre-ship checklist
|
||||
|
||||
- [ ] Key from env, never logged.
|
||||
- [ ] All HTTP timeouts ≥ 300 s.
|
||||
- [ ] `requestMetadata.status === "ok"` checked on every sync response.
|
||||
- [ ] Backoff on 429 + 5xx; never on 4xx.
|
||||
- [ ] Concurrency capped at plan limit.
|
||||
- [ ] Job `id` (from submit response) persisted to durable storage immediately.
|
||||
- [ ] Webhooks paired with polling fallback.
|
||||
- [ ] Result files downloaded immediately on `scraper.job.finished`.
|
||||
@@ -0,0 +1,116 @@
|
||||
# E-commerce APIs — Amazon & Shopify
|
||||
|
||||
| Endpoint | Returns |
|
||||
|---|---|
|
||||
| `/scrape/amazon/product` | Single product (price, ratings, variants, other sellers, A+) |
|
||||
| `/scrape/amazon/search` | Search results (sponsored + organic) |
|
||||
| `/scrape/amazon/seller` | Seller profile |
|
||||
| `/scrape/amazon/seller-products` | Seller catalog |
|
||||
| `/scrape/shopify/products` | Products from any Shopify store |
|
||||
| `/scrape/shopify/collections` | Collections from any Shopify store |
|
||||
|
||||
All synchronous `GET`.
|
||||
|
||||
## Amazon Product
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/amazon/product",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"asin": "B0DHJ7SBDR", "domain": "www.amazon.com", "otherSellers": "true"},
|
||||
timeout=300,
|
||||
)
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `asin` | **Required.**. |
|
||||
| `domain` | `www.amazon.com` (default), `.co.uk`, `.de`, `.co.jp`, … |
|
||||
| `language` | Locale per domain. |
|
||||
| `deliveryZip` | Affects shipping/availability fields. |
|
||||
| `shippingLocation` | 2-letter country code. |
|
||||
| `otherSellers` | `true` (default) to include other-seller block. |
|
||||
|
||||
Response: top-level `requestMetadata` + `product`. The `product` object's keys (verified live): `asin`, `url`, `title`, `brand`, `isAvailable`, `primaryFeatures`, `features`, `featureBullets`, `description`, `badges`, `breadcrumbs`, `whatIsInTheBox`, `variants`, `totalImages`, `primaryImage`, `images`, `descriptionImages`, `totalVideos`, `primaryVideo`, `videos`, `specification`, `reviewsInfo` (rating + count + sample reviews live here, not at the root). Pricing fields are surfaced via `variants` and `specification`.
|
||||
|
||||
## Amazon Search
|
||||
|
||||
```python
|
||||
params = {"q": "mechanical keyboard", "domain": "www.amazon.com", "page": 1}
|
||||
```
|
||||
|
||||
Params: `q` (required), `domain`, `language`, `page`, `deliveryZip`, `shippingLocation`, `sortBy`.
|
||||
|
||||
## Amazon Seller / Seller Products
|
||||
|
||||
```python
|
||||
profile = requests.get(
|
||||
"https://api.hasdata.com/scrape/amazon/seller",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"sellerId": "A1MNOPQR", "domain": "www.amazon.com"},
|
||||
timeout=300,
|
||||
).json()
|
||||
|
||||
catalog = requests.get(
|
||||
"https://api.hasdata.com/scrape/amazon/seller-products",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"sellerId": "A1MNOPQR", "page": 1},
|
||||
timeout=300,
|
||||
).json()
|
||||
```
|
||||
|
||||
Use cases: counterfeit detection, MAP enforcement, competitor catalog mirroring.
|
||||
|
||||
## Shopify Products
|
||||
|
||||
Works on **any** Shopify storefront with no authentication.
|
||||
|
||||
```python
|
||||
def shopify_all(store_url):
|
||||
page, out = 1, []
|
||||
while True:
|
||||
batch = requests.get(
|
||||
"https://api.hasdata.com/scrape/shopify/products",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"url": store_url, "page": page, "limit": 250},
|
||||
timeout=300,
|
||||
).json().get("products", [])
|
||||
if not batch:
|
||||
return out
|
||||
out.extend(batch)
|
||||
page += 1
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `url` | **Required.** Storefront URL. |
|
||||
| `limit` | 1–250, default `1`. **Bump to 250** for catalog work. |
|
||||
| `page` | 1-indexed. |
|
||||
| `collection` | Collection handle filter. |
|
||||
|
||||
`/scrape/shopify/collections` has the same shape and returns the collection list.
|
||||
|
||||
## Patterns
|
||||
|
||||
### Cross-merchant price comparison
|
||||
|
||||
```python
|
||||
a = requests.get("https://api.hasdata.com/scrape/amazon/search",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": query}, timeout=300).json()
|
||||
g = requests.get("https://api.hasdata.com/scrape/google/shopping",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": query, "gl": "us"}, timeout=300).json()
|
||||
```
|
||||
|
||||
### Reviews & bestsellers go through Scraper Jobs
|
||||
|
||||
The Product API only includes a sample of reviews. For all reviews use the `amazon-product-reviews` Scraper Job. For bestseller ranks use `amazon-bestsellers` — there's no synchronous API. See `scraper-jobs.md`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Same ASIN ≠ same product across `domain`s.** `.com` vs `.co.uk` can differ.
|
||||
- **`deliveryZip` changes availability.** Pass it when stock matters; omit for spec-only scrapes.
|
||||
- **Shopify `limit` defaults to 1** — always set 250 for catalog crawls.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Jobs APIs — Indeed & Glassdoor
|
||||
|
||||
| Endpoint | Returns |
|
||||
|---|---|
|
||||
| `/scrape/indeed/listing` | Indeed search results |
|
||||
| `/scrape/indeed/job` | Single Indeed job detail |
|
||||
| `/scrape/glassdoor/listing` | Glassdoor search results |
|
||||
| `/scrape/glassdoor/job` | Single Glassdoor job (incl. salary band, company snippet) |
|
||||
|
||||
All synchronous `GET`.
|
||||
|
||||
## Indeed Listing
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/indeed/listing",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={
|
||||
"keyword": "software engineer",
|
||||
"location": "New York, NY",
|
||||
"sort": "date",
|
||||
"domain": "www.indeed.com",
|
||||
"start": 0,
|
||||
},
|
||||
timeout=300,
|
||||
)
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `keyword` | **Required.** |
|
||||
| `location` | **Required.** |
|
||||
| `sort` | `date`, `relevance` (default). |
|
||||
| `domain` | Country site — `www.indeed.com`, `uk.indeed.com`, `de.indeed.com`. |
|
||||
| `start` | Offset, **steps of 10**. |
|
||||
|
||||
Response: `jobs` array with `title`, `company`, `location`, `salary`, `description`, `postedAt`, `link`, `jobKey`. Salary is free-form string — parse with regex.
|
||||
|
||||
## Indeed Job
|
||||
|
||||
Pass `jobKey` from listing → returns full description, requirements, benefits, company URL.
|
||||
|
||||
## Glassdoor Listing & Job
|
||||
|
||||
```python
|
||||
params = {"keyword": "software engineer", "location": "New York, NY", "sort": "recent"}
|
||||
# pagination: pass back nextPageToken
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `keyword`, `location` | **Required.** |
|
||||
| `sort` | `recent` (default), `relevant`. |
|
||||
| `domain` | Country site. |
|
||||
| `nextPageToken` | Cursor pagination. |
|
||||
|
||||
## Patterns
|
||||
|
||||
### Salary band
|
||||
|
||||
```python
|
||||
import re, statistics
|
||||
|
||||
def salary_band(role, location):
|
||||
page = requests.get(
|
||||
"https://api.hasdata.com/scrape/indeed/listing",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"keyword": role, "location": location}, timeout=300,
|
||||
).json()
|
||||
nums = [int(m.replace(",", ""))
|
||||
for j in page.get("jobs", [])
|
||||
for m in re.findall(r"\$([\d,]+)", j.get("salary") or "")]
|
||||
if not nums: return None
|
||||
return {"n": len(nums), "median": statistics.median(nums)}
|
||||
```
|
||||
|
||||
### Hiring velocity by company
|
||||
|
||||
```python
|
||||
from collections import Counter
|
||||
|
||||
page = indeed_listing(role, loc, sort="date")
|
||||
Counter(j.get("company") for j in page.get("jobs", []))
|
||||
```
|
||||
|
||||
Run weekly; sustained increases often precede earnings/PR signals.
|
||||
|
||||
### Pagination differs
|
||||
|
||||
```python
|
||||
# Indeed: numeric start
|
||||
for p in range(10):
|
||||
page = indeed_listing(kw, loc, start=p * 10)
|
||||
|
||||
# Glassdoor: cursor token
|
||||
out, token = [], None
|
||||
while True:
|
||||
page = glassdoor_listing(kw, loc, next_token=token)
|
||||
out.extend(page.get("jobs", []))
|
||||
token = page.get("nextPageToken")
|
||||
if not token: break
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Salary is free-form string.** Always regex-parse.
|
||||
- **Indeed = numeric start (10), Glassdoor = token.** Don't mix.
|
||||
- **`domain` matters for non-US.** `uk.indeed.com`, `ca.indeed.com`, etc.
|
||||
- **Prefer the API + pagination for bulk.** Reach for the matching Scraper Job only when you want webhook-driven fan-out across many keyword × location pairs without managing the polling loop yourself.
|
||||
@@ -0,0 +1,145 @@
|
||||
# Local Business APIs — Google Maps, Yelp, YellowPages
|
||||
|
||||
| Endpoint | Returns |
|
||||
|---|---|
|
||||
| `/scrape/google-maps/search` | Search results in a viewport |
|
||||
| `/scrape/google-maps/place` | Single place details |
|
||||
| `/scrape/google-maps/reviews` | Reviews for a place, paginated |
|
||||
| `/scrape/google-maps/photos` | Photo gallery |
|
||||
| `/scrape/google-maps/posts` | Owner-published posts (offers, events, announcements) |
|
||||
| `/scrape/google-maps/contributor-reviews` | All reviews by a Google reviewer |
|
||||
| `/scrape/yelp/search` | Yelp search |
|
||||
| `/scrape/yelp/place` | Yelp business detail |
|
||||
| `/scrape/yellowpages/search` | YellowPages search |
|
||||
| `/scrape/yellowpages/place` | YellowPages business detail |
|
||||
|
||||
All synchronous `GET`.
|
||||
|
||||
## Google Maps Search
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/google-maps/search",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": "Pizza", "ll": "@40.7455,-74.0083,14z"},
|
||||
timeout=300,
|
||||
)
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `q` | **Required.** Free-form query. |
|
||||
| `ll` | `@LAT,LNG,ZOOMz` viewport — **lat/lng + zoom, not a city name**. Required for tight pagination. |
|
||||
| `domain`, `gl`, `hl` | Standard. |
|
||||
| `start` | Pagination offset, **steps of 20**. |
|
||||
|
||||
Response: `localResults` — each entry has `position`, `title`, `placeId`, `dataId`, `kgmid`, `thumbnail`, `phone`, `address`, `website`, `description`, `workingHours` (object with `timezone` + `days[]`), `openState`, `rating`, `reviews`, `type` + `types[]` (categories), `price`, `priceDescription`, `gpsCoordinates`, `serviceOptions[]`, `extensions` (offerings, accessibility, payments, …), `menu`. Feed `placeId`/`dataId` into `/place` and `/reviews`.
|
||||
|
||||
## Google Maps Place
|
||||
|
||||
```python
|
||||
params = {"placeId": "ChIJFU2bda4SM4cRKSCRyb6pOB8"}
|
||||
```
|
||||
|
||||
Returns full place detail — coordinates, hours by day, phone, website, popular times, attributes (delivery, dine-in), photo summary.
|
||||
|
||||
## Google Maps Reviews
|
||||
|
||||
```python
|
||||
def reviews(place_id=None, data_id=None, sort_by="newestFirst", token=None):
|
||||
params = {}
|
||||
if place_id: params["placeId"] = place_id
|
||||
if data_id: params["dataId"] = data_id
|
||||
if sort_by: params["sortBy"] = sort_by
|
||||
if token: params["nextPageToken"] = token
|
||||
return requests.get(
|
||||
"https://api.hasdata.com/scrape/google-maps/reviews",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params=params, timeout=300,
|
||||
).json()
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `placeId` / `dataId` | Pass one. `dataId` is the hex pair from Maps results. |
|
||||
| `sortBy` | `newestFirst`, `highestRating`, `lowestRating`, `mostRelevant`. |
|
||||
| `topicId` | Filter by review topic. |
|
||||
| `nextPageToken` | Cursor pagination. |
|
||||
|
||||
## Google Maps Posts
|
||||
|
||||
```python
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/google-maps/posts",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"placeId": "ChIJ..."}, # or dataId="0x...:0x..."
|
||||
timeout=300,
|
||||
)
|
||||
for p in resp.json().get("posts", []):
|
||||
print(p["postedAt"], p["description"][:120], p.get("cta", {}).get("url"))
|
||||
```
|
||||
|
||||
Either `placeId` **or** `dataId` is required. Optional: `hl` (UI language), `nextPageToken` (cursor pagination). 10 credits/call.
|
||||
|
||||
Per-post fields (verified live): `postId`, `locationId`, `title`, `description`, `image`, `cta` (`label` + `url`), `createdAt` (ISO), `postedAt` (human-readable), `shareUrl`, `postUrl`. Response top-level: `posts`, `pagination`, `source`, `requestMetadata`.
|
||||
|
||||
Posts surface current offers, holiday hours, events, and product launches the business is actively promoting. Cheaper signal than the homepage scrape, and `cta.url` is the canonical landing page.
|
||||
|
||||
## Yelp & YellowPages
|
||||
|
||||
```python
|
||||
# Yelp
|
||||
params = {"keyword": "McDonald's", "location": "New York, NY", "start": 0} # steps of 10
|
||||
# YellowPages
|
||||
params = {"keyword": "Plumbers", "location": "New York, NY", "page": 1}
|
||||
```
|
||||
|
||||
YellowPages is US-only — EU/APAC searches return nothing useful.
|
||||
|
||||
## Patterns
|
||||
|
||||
### Lead-gen with emails (Maps + Web Scraping)
|
||||
|
||||
Maps results have website + phone but **not email**. Combine with the Web Scraping API's `extractEmails` only for public business contact pages, legitimate outreach, and workflows that honor opt-out, privacy-law, rate, and terms-of-service constraints:
|
||||
|
||||
```python
|
||||
leads = []
|
||||
for biz in maps_results.get("localResults", []):
|
||||
site = biz.get("website")
|
||||
if not site: continue
|
||||
page = requests.post(
|
||||
"https://api.hasdata.com/scrape/web",
|
||||
headers={"x-api-key": API_KEY},
|
||||
json={"url": site, "extractEmails": True},
|
||||
timeout=300,
|
||||
).json()
|
||||
leads.append({
|
||||
"name": biz["title"],
|
||||
"phone": biz.get("phone"),
|
||||
"website": site,
|
||||
"emails": page.get("extractedEmails") or [],
|
||||
})
|
||||
```
|
||||
|
||||
For higher volume, switch to the `contacts` Scraper Job (see `scraper-jobs.md`) only when you have a legitimate purpose, a compliant outreach process, and rate/opt-out controls.
|
||||
|
||||
### New-business discovery
|
||||
|
||||
Filter Maps by review count `< 5` — usually means recently opened.
|
||||
|
||||
```python
|
||||
new = [b for b in localResults if (b.get("reviews") or 0) < 5]
|
||||
```
|
||||
|
||||
### Multi-location chain mapping
|
||||
|
||||
Search the brand name; every `localResults` entry is a branch.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **`ll` is a viewport, not a city.** `@lat,lng,zoom`. Pasting "Brooklyn" fails.
|
||||
- **Pagination steps differ.** Maps `start` = +20, Yelp `start` = +10, Maps Reviews uses `nextPageToken`.
|
||||
- **`placeId` vs `dataId`** — Place prefers `placeId`; Reviews accepts either.
|
||||
- **YellowPages is US-only.**
|
||||
@@ -0,0 +1,84 @@
|
||||
# Real Estate APIs — Zillow, Redfin
|
||||
|
||||
| Endpoint | Returns |
|
||||
|---|---|
|
||||
| `/scrape/zillow/listing` | Search results by area + filters |
|
||||
| `/scrape/zillow/property` | Single home (history, agent, schools, taxes) |
|
||||
| `/scrape/redfin/listing` | Redfin search results |
|
||||
| `/scrape/redfin/property` | Single Redfin home |
|
||||
|
||||
All synchronous `GET`. 5 credits each.
|
||||
|
||||
For short-term rentals (Airbnb), hotels (Booking), and flights, see `travel.md`.
|
||||
|
||||
## Zillow Listing
|
||||
|
||||
Filter params use **bracketed** keys (`price[min]`, `beds[max]`).
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
def zillow_search(keyword, listing_type="forSale", **filters):
|
||||
r = requests.get(
|
||||
"https://api.hasdata.com/scrape/zillow/listing",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"keyword": keyword, "type": listing_type, **filters},
|
||||
timeout=300,
|
||||
)
|
||||
return r.json()
|
||||
|
||||
zillow_search("Brooklyn, NY", price={"min": 800000, "max": 2000000})
|
||||
zillow_search("33321", "sold", daysOnZillow="6m") # recent comps
|
||||
```
|
||||
|
||||
`requests` + `axios` serialize nested dicts as `price[min]=…&price[max]=…` automatically. With raw `URLSearchParams`, build the bracketed keys yourself.
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `keyword` | **Required.** Area string ("New York, NY", zip, neighborhood). |
|
||||
| `type` | **Required.** `forSale`, `forRent`, `sold`. |
|
||||
| `price[min/max]`, `beds[min/max]`, `baths[min/max]`, `sqft[min/max]` | Range filters. |
|
||||
| `daysOnZillow` | `24h`, `7d`, `14d`, `30d`, `90d`, `6m`, `12m`. |
|
||||
| `page` | Pagination. |
|
||||
|
||||
Response: `requestMetadata`, `searchInformation`, **`properties`** (the listings array — not `listings`), `pagination`.
|
||||
|
||||
## Zillow Property
|
||||
|
||||
```python
|
||||
requests.get(
|
||||
"https://api.hasdata.com/scrape/zillow/property",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"url": url, "extractAgentEmails": "true"},
|
||||
timeout=300,
|
||||
)
|
||||
```
|
||||
|
||||
Takes a full Zillow URL (not zpid). Returns address, lot/sqft/beds/baths, price + tax history, schools, agent block, photos. Agent emails are best-effort.
|
||||
|
||||
## Redfin
|
||||
|
||||
```python
|
||||
# Listing
|
||||
params = {"keyword": "33321", "type": "forSale", "page": 1}
|
||||
# Property
|
||||
params = {"url": "https://www.redfin.com/FL/Tamarac/9...html"}
|
||||
```
|
||||
|
||||
Same bracketed `price[min]`, `beds[min]`, etc. as Zillow. Zip codes work best for `keyword`.
|
||||
|
||||
## Patterns
|
||||
|
||||
### Sold comps for ROI
|
||||
|
||||
```python
|
||||
sold = zillow_search(zip_code, "sold", daysOnZillow="6m").get("properties", [])
|
||||
ppsf = [(l["price"] / l["livingArea"]) for l in sold if l.get("livingArea")]
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Bracketed query keys** — work with `requests`/`axios`, not raw `URLSearchParams`.
|
||||
- **`type=sold` + `daysOnZillow` = comps recipe.** Without `daysOnZillow`, history is unbounded.
|
||||
- **Property endpoints take URLs**, not IDs.
|
||||
- **Agent emails are best-effort.**
|
||||
@@ -0,0 +1,252 @@
|
||||
# Scraper Jobs — async, bulk
|
||||
|
||||
Use only when there's no Scraper-API equivalent (`crawler`, `contacts`, `sec-edgar`, `amazon-bestsellers`, `amazon-product-reviews`) or when you want webhook-driven fan-out without managing your own polling loop. Otherwise the matching Scraper API + paginated client loop is simpler.
|
||||
|
||||
| Slug | Notes |
|
||||
|---|---|
|
||||
| `crawler` | Recursive site crawl. Accepts every Web Scraping API parameter. |
|
||||
| `contacts` | URL list → emails / phones / social profiles. |
|
||||
| `sec-edgar` | Bulk SEC filings by CIK / ticker / company name. |
|
||||
| `google-serp`, `google-maps`, `google-maps-reviews`, `google-trends` | Bulk Google. |
|
||||
| `amazon-search`, `amazon-product`, `amazon-product-reviews`, `amazon-seller-products`, `amazon-bestsellers` | Bulk Amazon. |
|
||||
| `shopify` | Multi-store crawl. |
|
||||
| `zillow`, `redfin`, `airbnb` | Bulk real estate. |
|
||||
| `yelp`, `yellow-pages` | Bulk local. |
|
||||
| `indeed`, `glassdoor` | Bulk jobs. |
|
||||
|
||||
## Lifecycle
|
||||
|
||||
1. `POST /scrapers/<slug>/jobs` → returns the full job record. **The handle is `body.id` (numeric integer), not `jobId`** despite older doc snippets — store this. Status starts as `pending`.
|
||||
2. `GET /scrapers/jobs/<id>` — poll status.
|
||||
3. `GET /scrapers/jobs/<id>/results?page=…&limit=100` — once `status === "finished"`.
|
||||
4. `DELETE /scrapers/jobs/<id>` — stop early (rows produced before stop are kept).
|
||||
|
||||
Status values: `pending` → `in_progress` → `finished` (or `stopped` if cancelled).
|
||||
|
||||
**Shortcut for finished jobs:** the status response on a `finished` job carries a `data` object with direct download URLs:
|
||||
|
||||
```json
|
||||
"data": {
|
||||
"csv": "https://f005.backblazeb2.com/file/.../{uuid}.csv",
|
||||
"json": "https://f005.backblazeb2.com/file/.../{uuid}.json",
|
||||
"xlsx": "https://f005.backblazeb2.com/file/.../{uuid}.xlsx"
|
||||
}
|
||||
```
|
||||
|
||||
For one-shot ingestion, fetch `data.json` directly instead of paging `/results`. **These URLs are short-lived** — download immediately on `finished`.
|
||||
|
||||
## End-to-end (Python)
|
||||
|
||||
```python
|
||||
import os, time, requests
|
||||
|
||||
API_KEY = os.environ["HASDATA_API_KEY"]
|
||||
H = {"x-api-key": API_KEY, "Content-Type": "application/json"}
|
||||
BASE = "https://api.hasdata.com"
|
||||
|
||||
def submit(slug, body):
|
||||
r = requests.post(f"{BASE}/scrapers/{slug}/jobs", headers=H, json=body, timeout=60)
|
||||
r.raise_for_status()
|
||||
return r.json()["id"] # numeric job id — not "jobId"
|
||||
|
||||
def wait(job_id, poll=10, cap=60, timeout=3600):
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
s = requests.get(f"{BASE}/scrapers/jobs/{job_id}", headers=H, timeout=60).json()
|
||||
if s["status"] in ("finished", "stopped"):
|
||||
return s
|
||||
time.sleep(poll)
|
||||
poll = min(poll * 1.5, cap)
|
||||
raise TimeoutError(job_id)
|
||||
|
||||
def results(job_id):
|
||||
page = 1
|
||||
while True:
|
||||
body = requests.get(
|
||||
f"{BASE}/scrapers/jobs/{job_id}/results",
|
||||
headers=H, params={"page": page, "limit": 100}, timeout=120,
|
||||
).json()
|
||||
for row in body["data"]:
|
||||
yield row["data"] # double-wrapped — see below
|
||||
if body["meta"]["currentPage"] >= body["meta"]["lastPage"]:
|
||||
return
|
||||
page += 1
|
||||
```
|
||||
|
||||
### Response shapes
|
||||
|
||||
Submit (live):
|
||||
```json
|
||||
{
|
||||
"id": 416349, // ← the job handle, integer
|
||||
"scraperId": 26,
|
||||
"status": "pending",
|
||||
"creditsSpent": 0,
|
||||
"dataRowsCount": 0,
|
||||
"input": { ... },
|
||||
"createdAt": "...", "updatedAt": "...",
|
||||
"scraper": { "slug": "contacts", ... },
|
||||
"columns": [ ... ]
|
||||
}
|
||||
```
|
||||
|
||||
Status (live; numeric fields arrive as **strings** when populated):
|
||||
```json
|
||||
{
|
||||
"id": 416349,
|
||||
"status": "finished",
|
||||
"creditsSpent": "5", // string!
|
||||
"dataRowsCount": "1", // string!
|
||||
"input": { ... },
|
||||
"data": {
|
||||
"csv": "https://f005.backblazeb2.com/.../{uuid}.csv",
|
||||
"json": "https://f005.backblazeb2.com/.../{uuid}.json",
|
||||
"xlsx": "https://f005.backblazeb2.com/.../{uuid}.xlsx"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Results page:
|
||||
```json
|
||||
{
|
||||
"meta": {
|
||||
"total": 1, "perPage": 100,
|
||||
"currentPage": 1, "lastPage": 1,
|
||||
"firstPage": 1, "firstPageUrl": "/?page=1",
|
||||
"lastPageUrl": "/?page=1",
|
||||
"nextPageUrl": null, "previousPageUrl": null
|
||||
},
|
||||
"data": [
|
||||
{
|
||||
"id": "...", "jobId": 416349, "dataId": "...",
|
||||
"data": { /* the actual scraped row */ },
|
||||
"createdAt": "...", "updatedAt": "..."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Double `data`** — the row is `body["data"][i]["data"]`; the outer wraps with `id`, `jobId`, `dataId`, `createdAt`, `updatedAt`.
|
||||
|
||||
## Common body fields
|
||||
|
||||
- `limit` (int) — max rows. `0` = no cap.
|
||||
- `webhook.url` (string, https), `webhook.events` (any subset of `scraper.job.started`, `scraper.data.scraped`, `scraper.job.finished`), `webhook.headers` (sent on every callback — pin a shared secret here).
|
||||
|
||||
## Webhooks
|
||||
|
||||
```python
|
||||
# Submit with webhook
|
||||
submit("indeed", {
|
||||
"keywords": ["software engineer", "data scientist"],
|
||||
"locations": ["New York, NY", "Remote"],
|
||||
"limit": 500,
|
||||
"webhook": {
|
||||
"url": "https://your.app/hasdata-hook",
|
||||
"events": ["scraper.data.scraped", "scraper.job.finished"],
|
||||
"headers": {"x-shared-secret": SHARED_SECRET},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
```python
|
||||
from flask import Flask, request, abort
|
||||
app = Flask(__name__)
|
||||
|
||||
@app.post("/hasdata-hook")
|
||||
def hook():
|
||||
if request.headers.get("x-shared-secret") != SHARED_SECRET:
|
||||
abort(401)
|
||||
e = request.json
|
||||
if e["event"] == "scraper.data.scraped":
|
||||
save_row(e["jobId"], e["data"])
|
||||
elif e["event"] == "scraper.job.finished":
|
||||
finalize(e["jobId"])
|
||||
return "", 200 # 2xx prevents retry
|
||||
```
|
||||
|
||||
- Async with **3 retries** on non-2xx. **Order not guaranteed** — payload is the source of truth.
|
||||
- **No documented HMAC.** Pin a shared secret via `webhook.headers`, or just fetch results via the API on `scraper.job.finished` and ignore per-row deliveries.
|
||||
- **Always pair webhooks with polling.** A long quiet period probably means missed callbacks.
|
||||
|
||||
## Per-scraper bodies
|
||||
|
||||
### `crawler` — recursive site crawl
|
||||
|
||||
Accepts every Web Scraping API parameter applied to **every page**.
|
||||
|
||||
| Field | Notes |
|
||||
|---|---|
|
||||
| `urls` | **Required.** Seed URLs. |
|
||||
| `maxDepth` | Hops from seed. |
|
||||
| `includePaths` / `excludePaths` | Regex. **Case-sensitive.** |
|
||||
| `limit` | Cap on pages. `0` = unlimited. |
|
||||
|
||||
```python
|
||||
job = submit("crawler", {
|
||||
"urls": ["https://docs.example.com"],
|
||||
"maxDepth": 5,
|
||||
"includePaths": "/docs/.+",
|
||||
"outputFormat": ["markdown"],
|
||||
"excludeTags": ["script", "style", "nav", "footer"],
|
||||
"limit": 2000,
|
||||
})
|
||||
```
|
||||
|
||||
### `contacts` — URLs → contact info
|
||||
|
||||
```python
|
||||
submit("contacts", {"urls": ["https://example.com/about", "https://example.com/team"]})
|
||||
```
|
||||
|
||||
Verified row schema (one row per input URL):
|
||||
|
||||
```json
|
||||
{
|
||||
"url": "https://example.com/about",
|
||||
"emails": ["..."],
|
||||
"phoneNumbers": ["..."],
|
||||
"linkedin": ["..."],
|
||||
"xcom": ["..."], // X / Twitter — note key is "xcom"
|
||||
"facebook": ["..."],
|
||||
"instagram": ["..."],
|
||||
"dribbble": ["..."],
|
||||
"clutch": ["..."]
|
||||
}
|
||||
```
|
||||
|
||||
Empty arrays for missing categories — never null. If you only have a domain, discover URLs first via SERP `site:example.com`.
|
||||
|
||||
### `sec-edgar` — bulk SEC filings
|
||||
|
||||
```python
|
||||
submit("sec-edgar", {
|
||||
"limit": 100,
|
||||
"ciks": ["AAPL", "789019", "Alphabet Inc."],
|
||||
"filingTypes": "10-K, 10-Q, 8-K",
|
||||
"startDate": "2024-01-01",
|
||||
"endDate": "2025-12-31",
|
||||
})
|
||||
```
|
||||
|
||||
`ciks` accepts CIKs, tickers, or company names mixed.
|
||||
|
||||
### Bulk-API equivalents
|
||||
|
||||
`google-serp`, `google-maps`, `amazon-search`, `indeed`, `glassdoor`, etc. Jobs accept arrays of inputs (`keywords[]`, `locations[]`, etc.). Use them when you want webhook fan-out; otherwise the synchronous Scraper API + paginated client loop is simpler.
|
||||
|
||||
### Crawler vs Contacts vs Web Scraping batch
|
||||
|
||||
- **crawler** — unknown URL set, recursive discovery.
|
||||
- **contacts** — known URL list, want extracted contact fields.
|
||||
- **`/scrape/batch/web`** — known URL list, want full HTML/markdown/AI extraction at >1k scale.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Persist the job `id` immediately** (the integer from the submit response — *not* `jobId`). Only handle to status, results, stop.
|
||||
- **Result file retention is short.** Download right after `finished`.
|
||||
- **Webhooks are best-effort.** Always poll as a backup.
|
||||
- **`includePaths` regex is case-sensitive.**
|
||||
- **Status `stopped` is terminal.** Rows already produced remain available.
|
||||
- **Don't poll faster than every 10 s** — wastes concurrency cap.
|
||||
- **Double-wrapped results** — `body["data"][i]["data"]`, not `body["data"][i]`.
|
||||
@@ -0,0 +1,154 @@
|
||||
# Search & SERP APIs
|
||||
|
||||
Pre-parsed JSON for Google, AI Mode, Bing, and the specialized Google panels. Synchronous `GET` under `https://api.hasdata.com`.
|
||||
|
||||
| Endpoint | Returns |
|
||||
|---|---|
|
||||
| `/scrape/google/serp` | Full SERP — organic + every rich-snippet block |
|
||||
| `/scrape/google-light/serp` | Organic only |
|
||||
| `/scrape/google/ai-mode` | Gemini answer + references |
|
||||
| `/scrape/google/ai-overview` | AI Overview block |
|
||||
| `/scrape/google/news` | News articles |
|
||||
| `/scrape/google/shopping` | Shopping carousel |
|
||||
| `/scrape/google/images` | Image search |
|
||||
| `/scrape/google/events` | Local events |
|
||||
| `/scrape/google/short-videos` | Short-video panel |
|
||||
| `/scrape/google/immersive-product` | Expanded product pop-up |
|
||||
| `/scrape/google-trends/search` | Trends + related queries |
|
||||
| `/scrape/bing/serp` | Bing SERP |
|
||||
|
||||
For `/scrape/google/flights`, see `travel.md`.
|
||||
|
||||
## Google SERP
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/google/serp",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": "coffee beans", "gl": "us", "hl": "en", "num": 100},
|
||||
timeout=300,
|
||||
)
|
||||
for hit in resp.json().get("organicResults", []):
|
||||
print(hit["position"], hit["title"], hit["link"])
|
||||
```
|
||||
|
||||
### Query parameters
|
||||
|
||||
| Param | Default | Notes |
|
||||
|---|---|---|
|
||||
| `q` | — | **Required.** |
|
||||
| `location` | — | Canonical, e.g. `"Austin,Texas,United States"`. Hyper-local. |
|
||||
| `uule` | — | Pre-encoded location (mutually exclusive with `location`). |
|
||||
| `domain` | `google.com` | `google.co.uk`, `google.de`, … |
|
||||
| `gl` | — | 2-letter country (`us`, `de`, `jp`). |
|
||||
| `hl` | — | 2-letter UI language. |
|
||||
| `lr` | — | Content-language filter (`lang_en`). |
|
||||
| `tbs` | — | Filters — `qdr:d|w|m|y` for time, `li:1` verbatim, sort, image type. |
|
||||
| `safe` | — | `active` / `off`. |
|
||||
| `start` | `0` | Pagination offset. |
|
||||
| `num` | `10` | Results/page. **Max 100** |
|
||||
| `tbm` | — | `isch` images, `vid`, `nws`, `shop`, `lcl`. |
|
||||
| `deviceType` | — | `desktop`, `mobile`, `tablet`. |
|
||||
|
||||
### Response keys
|
||||
|
||||
```
|
||||
requestMetadata, searchInformation, organicResults, knowledgeGraph, answerBox,
|
||||
aiOverview, topStories, newsResults, localResults, inlineShoppingResults,
|
||||
inlineVideos, inlineImages, recipesResults, perspectives, discussionsAndForums,
|
||||
relatedQuestions, relatedSearches, adResults, pagination
|
||||
```
|
||||
|
||||
Rich-snippet keys appear **only when the SERP shows that block** — always `data.get(key, default)`.
|
||||
|
||||
### Tips
|
||||
|
||||
- `gl`/`hl` change ranking, not just localization. Run the same `q` with different `gl` to study geo-bias.
|
||||
- `location="Austin,Texas,United States"` produces hyperlocal results that differ from `gl=us` alone.
|
||||
|
||||
## Google Light SERP
|
||||
|
||||
Same params as full SERP, but the response is trimmed to a few keys — typically `requestMetadata`, `searchInformation`, `organicResults`, `relatedSearches`, and `pagination` when present. Use for crawler seeding and link discovery when you don't need the heavier rich-snippet blocks.
|
||||
|
||||
## Google AI Mode
|
||||
|
||||
```python
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/google/ai-mode",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": "is coffee good for health?", "location": "Austin,Texas,United States"},
|
||||
timeout=300,
|
||||
)
|
||||
```
|
||||
|
||||
Params: `q` (required), `location`, `uule`, `gl`. Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"requestMetadata": {...},
|
||||
"textBlocks": [
|
||||
{"type":"heading","snippet":"..."},
|
||||
{"type":"paragraph","snippet":"...","snippetHighlightedWords":["..."]},
|
||||
{"type":"list","list":[{"snippet":"..."}]},
|
||||
{"type":"table","table":{...}},
|
||||
{"type":"code","code":"..."}
|
||||
],
|
||||
"references": [{"index":1,"link":"...","title":"...","snippet":"...","source":"..."}]
|
||||
}
|
||||
```
|
||||
|
||||
Block types observed in practice: `heading`, `paragraph`, `list`, `table`, `code`. Always switch on `type` rather than assuming a fixed set.
|
||||
|
||||
Pattern: AI Mode for the answer → `/scrape/web` (markdown) on each `references[].link` → cited RAG context.
|
||||
|
||||
## Google News / Shopping / Bing
|
||||
|
||||
Same shape: `q` + `gl`/`hl`/`location`. News supports `tbs=qdr:d|w|m|y` for time windows. Bing returns the same key set as Google SERP — useful for cross-engine consensus (disagreement = contested topic).
|
||||
|
||||
## Patterns
|
||||
|
||||
### Pagination
|
||||
|
||||
```python
|
||||
def all_organic(q, target=300):
|
||||
out, start = [], 0
|
||||
while len(out) < target:
|
||||
page = requests.get(
|
||||
"https://api.hasdata.com/scrape/google-light/serp",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": q, "num": 100, "start": start},
|
||||
timeout=300,
|
||||
).json().get("organicResults", [])
|
||||
if not page:
|
||||
break
|
||||
out.extend(page)
|
||||
start += 100
|
||||
return out[:target]
|
||||
```
|
||||
|
||||
### Reverse lookup (email / phone / domain → identity)
|
||||
|
||||
```python
|
||||
requests.get(
|
||||
"https://api.hasdata.com/scrape/google/serp",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": f'"{literal}"', "num": 20},
|
||||
timeout=300,
|
||||
).json().get("organicResults", [])
|
||||
```
|
||||
|
||||
Quoted literals (emails, phones, error strings) usually surface the canonical mention.
|
||||
|
||||
### Indexation check
|
||||
|
||||
```python
|
||||
def is_indexed(url):
|
||||
r = requests.get(
|
||||
"https://api.hasdata.com/scrape/google-light/serp",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": f"site:{url}", "num": 1}, timeout=300,
|
||||
)
|
||||
return bool(r.json().get("organicResults"))
|
||||
```
|
||||
@@ -0,0 +1,202 @@
|
||||
# Travel APIs — Airbnb, Booking, Google Flights
|
||||
|
||||
| Endpoint | Returns |
|
||||
|---|---|
|
||||
| `/scrape/airbnb/listing` | Airbnb search results |
|
||||
| `/scrape/airbnb/property` | Single Airbnb listing |
|
||||
| `/scrape/booking/search` | Booking.com search results (hotels, apartments) |
|
||||
| `/scrape/booking/place` | Single Booking.com property with room/rate list |
|
||||
| `/scrape/google/flights` | Google Flights prices and itineraries |
|
||||
|
||||
All synchronous `GET`. Airbnb is 5 credits; Booking is 10; Google Flights is 15.
|
||||
|
||||
For activities at the destination see `/scrape/google/events` (in `search.md`); for ground transport, scrape the operator's site with `POST /scrape/web`.
|
||||
|
||||
## Airbnb
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
def airbnb_search(location, check_in, check_out, **kwargs):
|
||||
return requests.get(
|
||||
"https://api.hasdata.com/scrape/airbnb/listing",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"location": location, "checkIn": check_in, "checkOut": check_out, **kwargs},
|
||||
timeout=300,
|
||||
).json()
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `location` | **Required.** Free-form. |
|
||||
| `checkIn` | **Required.** `YYYY-MM-DD`. |
|
||||
| `checkOut`, `adults`, `children`, `infants`, `pets` | Optional. |
|
||||
| `nextPageToken` | Pagination cursor. |
|
||||
|
||||
### Token pagination
|
||||
|
||||
```python
|
||||
def airbnb_all(location, check_in, check_out):
|
||||
out, token = [], None
|
||||
while True:
|
||||
page = airbnb_search(location, check_in, check_out,
|
||||
**({"nextPageToken": token} if token else {}))
|
||||
out.extend(page.get("listings", []))
|
||||
token = page.get("nextPageToken")
|
||||
if not token:
|
||||
return out
|
||||
```
|
||||
|
||||
### Airbnb Property
|
||||
|
||||
```python
|
||||
requests.get(
|
||||
"https://api.hasdata.com/scrape/airbnb/property",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"url": "https://www.airbnb.com/rooms/12345678"},
|
||||
timeout=300,
|
||||
)
|
||||
```
|
||||
|
||||
## Booking Search
|
||||
|
||||
```python
|
||||
import json, requests
|
||||
|
||||
def booking_search(keyword, check_in, check_out, *, adults=2, children=0,
|
||||
children_ages=None, rooms=1, **filters):
|
||||
params = {
|
||||
"keyword": keyword,
|
||||
"checkInDate": check_in,
|
||||
"checkOutDate": check_out,
|
||||
"adults": adults,
|
||||
"children": children,
|
||||
"rooms": rooms,
|
||||
**filters,
|
||||
}
|
||||
if children and children_ages:
|
||||
params["childrenAgesJson"] = json.dumps(children_ages)
|
||||
return requests.get(
|
||||
"https://api.hasdata.com/scrape/booking/search",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params=params, timeout=300,
|
||||
).json()
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `keyword` | **Required.** City, neighborhood, or property name. |
|
||||
| `checkInDate` / `checkOutDate` | **Required.** `YYYY-MM-DD`. |
|
||||
| `adults`, `children`, `rooms` | **Required.** Pass `children=0` explicitly when none. |
|
||||
| `childrenAgesJson` | Required iff `children > 0` — JSON array of ages (0–17), one per child. |
|
||||
| `price[min]` / `price[max]` | `>= 10` / `>= 20`. Bracketed — `requests`/`axios` serialize nested dicts as `price[min]=…`. |
|
||||
| `rating[]`, `reviewScore[]`, `propertyType[]`, `facilities[]`, `meals[]`, `bedPreference[]`, `roomFacilities[]`, `propertyAccessibility[]`, `roomAccessibility[]`, `distanceFromCenter[]`, `travelGroup[]`, `onlinePayment[]`, `reservationPolicy[]` | Multi-value filters (OR). |
|
||||
| `bedrooms`, `bathrooms` | Minimum count. |
|
||||
| `sort` | `ourTopPicks`, `homesAndApartmentsFirst`, `priceLowestFirst`, `priceHighestFirst`, `bestReviewedAndLowestPrice`, `ratingHighToLow`, `ratingLowToHigh`, `ratingAndPrice`, `distanceFromDowntown`, `topReviewed`. |
|
||||
| `page` | 1-indexed, 25 results per page. |
|
||||
| `currency` | ISO code or `hotelCurrency` to keep native. |
|
||||
| `language` | UI locale. |
|
||||
|
||||
Top-level response (verified live): `requestMetadata`, `searchInformation`, `pagination`, `results`. Per-result keys: `hotelId`, `roomId`, `title`, `url`, `location`, `rating`, `reviews`, `price` (object with `total` / `nightly` / `currency`), `room`, `beds`, `bedTypes`, `policies`, `photo`.
|
||||
|
||||
## Booking Place
|
||||
|
||||
```python
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/booking/place",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={
|
||||
"url": "https://www.booking.com/hotel/fr/le-bristol-paris.html",
|
||||
"checkInDate": "2026-07-10",
|
||||
"checkOutDate": "2026-07-13",
|
||||
"adults": 2,
|
||||
"children": 0,
|
||||
"rooms": 1,
|
||||
},
|
||||
timeout=300,
|
||||
).json()
|
||||
```
|
||||
|
||||
`url` must be `booking.com` / `www.booking.com`. The remaining stay/guest parameters share the same rules as `booking-search` (including `childrenAgesJson` when `children > 0`).
|
||||
|
||||
Response top-level keys: `requestMetadata`, `overview`, `bookingDetails`, `rooms`, `facilities`, `houseRules`, `ratings`, `reviews`, `restaurants`, `breadcrumbs`, `questionsAndAnswers`.
|
||||
|
||||
- `overview` → `id`, `title`, `address`, `description`, `propertyType`, `photos`, `highlights`, `mostPopularFacilities`.
|
||||
- `rooms[i]` → `roomId`, `name`, `bedTypes`, `beds`, `facilities`, `otherFacilities`, `variants[]` (per-rate price/availability). Variants are the actual buyable units; `rooms[i]` is the floor-plan.
|
||||
|
||||
## Google Flights
|
||||
|
||||
```python
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/google/flights",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={
|
||||
"departureId": "JFK",
|
||||
"arrivalId": "LAX",
|
||||
"outboundDate": "2026-06-15",
|
||||
"returnDate": "2026-06-22", # omit for one-way
|
||||
"currency": "USD",
|
||||
},
|
||||
timeout=300,
|
||||
).json()
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `departureId` / `arrivalId` | **Required.** IATA airport codes (`JFK`, `LAX`). |
|
||||
| `outboundDate` | **Required.** `YYYY-MM-DD`. |
|
||||
| `returnDate` | Optional — omit for one-way. |
|
||||
| `currency` | ISO code. |
|
||||
| `gl`, `hl` | Country / language. |
|
||||
| `travelClass` | `1` economy, `2` premium economy, `3` business, `4` first. |
|
||||
| `stops` | `0` any, `1` non-stop, `2` ≤1 stop, `3` ≤2 stops. |
|
||||
| `adults`, `children`, `infantsInSeat`, `infantsOnLap` | Passenger counts. |
|
||||
|
||||
## Patterns
|
||||
|
||||
### STR yield estimate
|
||||
|
||||
```python
|
||||
rentals = airbnb_search(area, ci, co).get("listings", []) # Airbnb → "listings"
|
||||
# pair with /scrape/zillow/listing (see real-estate.md) for purchase price
|
||||
night = sum(r.get("price", 0) for r in rentals) / max(len(rentals), 1)
|
||||
```
|
||||
|
||||
### Hotel-vs-rental price diff
|
||||
|
||||
```python
|
||||
b = booking_search(city, ci, co, adults=2, children=0, rooms=1, sort="priceLowestFirst")
|
||||
a = airbnb_search(city, ci, co, adults=2)
|
||||
def median(xs): xs = sorted(xs); return xs[len(xs)//2] if xs else None
|
||||
median_hotel = median([r["price"]["nightly"] for r in b.get("results", []) if r.get("price")])
|
||||
median_str = median([r["price"] for r in a.get("listings", []) if r.get("price")])
|
||||
```
|
||||
|
||||
### Full trip cost
|
||||
|
||||
```python
|
||||
flight = requests.get(
|
||||
"https://api.hasdata.com/scrape/google/flights",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"departureId": origin, "arrivalId": dest_iata,
|
||||
"outboundDate": dep, "returnDate": ret, "currency": "USD"},
|
||||
timeout=300,
|
||||
).json()
|
||||
cheapest_flight = min((f["price"] for f in flight.get("best_flights", [])), default=None)
|
||||
|
||||
stay = booking_search(city, dep, ret, adults=2, children=0, rooms=1, sort="priceLowestFirst")
|
||||
cheapest_stay = stay.get("results", [{}])[0].get("price", {}).get("total")
|
||||
|
||||
total = (cheapest_flight or 0) + (cheapest_stay or 0)
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Airbnb requires `checkIn`** and uses **token** pagination — store `nextPageToken`, not page numbers.
|
||||
- **Airbnb property endpoints take URLs**, not IDs.
|
||||
- **Booking requires `children` even when zero.** Pass `children=0`. When `children > 0`, also pass `childrenAgesJson` with exactly that many ages.
|
||||
- **Booking `price[min]` / `price[max]`** are bracketed — use a nested dict with `requests`/`axios`.
|
||||
- **Booking `rooms[i].variants[]` is where prices live** — the parent `rooms[i]` describes the floor-plan, variants are the buyable rates with `priceBreakdown` / `cancellationPolicy` / `mealPlan`.
|
||||
- **`bookingDetails` carries the resolved stay context** the response was priced for — echo it back when persisting results so future comparisons use the same dates / occupancy.
|
||||
- **Google Flights uses IATA codes**, not city names. `JFK` not `New York`.
|
||||
- **Round-trip vs one-way** is determined by `returnDate` presence — pass it for round-trip, omit for one-way.
|
||||
@@ -0,0 +1,159 @@
|
||||
# Web Scraping API — `POST /scrape/web`
|
||||
|
||||
One endpoint to fetch any URL, optionally with JS rendering, proxies, AI extraction, and screenshots. Synchronous.
|
||||
|
||||
> Reach for this only when the user gave you a specific URL, or when no Scraper API covers the field. Otherwise the platform-specific APIs return pre-extracted JSON without direct page access. Use only for public pages or content the user is authorized to access.
|
||||
|
||||
## Minimal request
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
# Multiple outputs (or include "json") → response is a JSON object
|
||||
resp = requests.post(
|
||||
"https://api.hasdata.com/scrape/web",
|
||||
headers={"x-api-key": API_KEY},
|
||||
json={"url": "https://example.com", "outputFormat": ["markdown", "json"]},
|
||||
timeout=300,
|
||||
)
|
||||
data = resp.json()
|
||||
assert data["requestMetadata"]["status"] == "ok"
|
||||
print(data["markdown"])
|
||||
|
||||
# Single non-JSON output → response IS the raw content (markdown/html/text bytes)
|
||||
resp = requests.post(
|
||||
"https://api.hasdata.com/scrape/web",
|
||||
headers={"x-api-key": API_KEY},
|
||||
json={"url": "https://example.com", "outputFormat": ["markdown"]},
|
||||
timeout=300,
|
||||
)
|
||||
print(resp.text) # raw markdown — no JSON parsing
|
||||
```
|
||||
|
||||
## Body parameters
|
||||
|
||||
| Parameter | Type | Notes |
|
||||
|---|---|---|
|
||||
| `url` | string | **Required.** Absolute URL. |
|
||||
| `outputFormat` | string[] | `html`, `text`, `markdown`, `json`. **Single non-JSON format → raw content as the body** (not JSON-wrapped); multiple formats → JSON object with one key per format. Always include `"json"` (or another format) when you also need `requestMetadata`. |
|
||||
| `proxyType` | enum | `datacenter` (default) or `residential` — use residential only for authorized geo/availability testing where terms and access controls permit it. |
|
||||
| `proxyCountry` | string | ISO 3166-1 alpha-2 — `US`, `UK`, `DE`, `FR`, `IT`, `SE`, `BR`, `CA`, `JP`, `SG`, `IN`, `ID`, `IE`. |
|
||||
| `jsRendering` | bool | Headless browser — required for SPAs and dynamically-injected content. |
|
||||
| `wait` / `waitFor` | int (ms) / CSS string | Fixed delay vs. wait-until-selector. Prefer `waitFor`. |
|
||||
| `jsScenario` | array | Sequence of click/fill/wait/scroll/evaluate. Requires `jsRendering`. |
|
||||
| `headers` | object | Custom headers. **Cookies go here too — no separate `cookies` parameter.** |
|
||||
| `screenshot` | bool | Returns a CDN URL in the response. |
|
||||
| `extractRules` | object | CSS selectors → field text. `@attr` for attributes. **First match only**, missing → `null`. |
|
||||
| `aiExtractRules` | object | Typed LLM extraction. Types: `string`, `number`, `boolean`, `list`, `item`. |
|
||||
| `extractEmails` / `extractLinks` | bool | Quick helpers. |
|
||||
| `blockResources` / `blockAds` | bool | Skip images/CSS/ads — speeds text-only scrapes. |
|
||||
| `blockUrls` | string[] | Glob patterns to block subresources. |
|
||||
| `removeBase64Images` | bool | Strip inline base64 from response. |
|
||||
| `includeOnlyTags` / `excludeTags` | string[] | Trim DOM before serialization. |
|
||||
|
||||
## CSS extraction (`extractRules`)
|
||||
|
||||
```python
|
||||
"extractRules": {
|
||||
"title": "h1",
|
||||
"links": "a @href", # @attr extracts attribute
|
||||
"price": ".price-now",
|
||||
}
|
||||
```
|
||||
|
||||
First match per selector. For lists of records, use `aiExtractRules` with `type: "list"`.
|
||||
|
||||
## AI extraction (`aiExtractRules`)
|
||||
|
||||
```python
|
||||
"aiExtractRules": {
|
||||
"title": {"type": "string"},
|
||||
"price": {"type": "number"},
|
||||
"in_stock": {"type": "boolean"},
|
||||
"tags": {"type": "list", "description": "category tags"},
|
||||
"author": {"type": "item", "output": {
|
||||
"name": {"type": "string"},
|
||||
"verified": {"type": "boolean"},
|
||||
}},
|
||||
"reviews": {"type": "list", "output": {
|
||||
"rating": {"type": "number"},
|
||||
"text": {"type": "string"},
|
||||
}},
|
||||
}
|
||||
```
|
||||
|
||||
Use when layout varies across pages; otherwise prefer `extractRules` for determinism and predictability.
|
||||
|
||||
## JS scenarios
|
||||
|
||||
```python
|
||||
"jsScenario": [
|
||||
{"fill": ["#email", "user@example.com"]},
|
||||
{"fill": ["#password", PASSWORD]},
|
||||
{"click": "#login"},
|
||||
{"waitFor": ".dashboard"},
|
||||
{"scrollY": 2000},
|
||||
{"waitForAndClick": ".load-more"},
|
||||
{"evaluate": "window.__APP_STATE__"},
|
||||
]
|
||||
```
|
||||
|
||||
Actions: `click`, `fill: [sel, val]`, `wait: ms`, `waitFor: sel`, `waitForAndClick: sel`, `scrollX/scrollY: px`, `evaluate: "JS"`. Sequential. Missing element on `click`/`fill` fails the request — wrap with `waitFor` first.
|
||||
|
||||
## Auth via cookies
|
||||
|
||||
```python
|
||||
"headers": {
|
||||
"User-Agent": "Mozilla/5.0 ...",
|
||||
"Cookie": "session=abc; csrf=xyz",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
}
|
||||
```
|
||||
|
||||
Capture cookies once in a real browser (devtools → Storage → Cookies), forward via the `Cookie` header. Only with explicit user permission and authority to access that account/content; never use cookies to bypass someone else's access controls.
|
||||
|
||||
## Slim response & speed
|
||||
|
||||
```python
|
||||
{
|
||||
"blockResources": True, # skip images/CSS/fonts
|
||||
"blockAds": True, # skip ad/tracking
|
||||
"blockUrls": ["**.googletagmanager.com/**", "**.doubleclick.net/**"],
|
||||
"removeBase64Images": True,
|
||||
"excludeTags": ["script", "style", "nav", "footer"],
|
||||
}
|
||||
```
|
||||
|
||||
Reduces response size 60–90% on noisy pages.
|
||||
|
||||
## Response shape
|
||||
|
||||
The wrapper is JSON **only when** the response is JSON-wrapped — i.e. multiple `outputFormat` values, or a single value that includes `"json"`. With a single non-JSON format the response body is the raw content (`text/markdown`, `text/html`, `text/plain`).
|
||||
|
||||
```json
|
||||
{
|
||||
"requestMetadata": { "id": "uuid", "status": "ok", "url": "..." },
|
||||
"headers": { "content-type": "text/html" },
|
||||
"screenshot": "https://...jpeg",
|
||||
"content": "<!DOCTYPE html>...", // outputFormat: html
|
||||
"markdown": "# Title\n...", // outputFormat: markdown
|
||||
"text": "Title\n...",
|
||||
"extractRules": { ... }, // present iff sent
|
||||
"aiExtractRules": { ... }, // present iff sent
|
||||
"extractedEmails": [ ... ], // iff extractEmails: true
|
||||
"extractedLinks": [ ... ] // iff extractLinks: true
|
||||
}
|
||||
```
|
||||
|
||||
## Batch (`POST /scrape/batch/web`)
|
||||
|
||||
Async wrapper for >1k URLs running the same extraction. Returns `jobId`; poll status, page `/results`. Per-batch cap **10,000 URLs**. For small workloads loop the sync endpoint at concurrency = plan limit.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Disable `jsRendering` first**, enable only when the page needs it — most static pages parse fine without a headless browser.
|
||||
- **`waitFor` > `wait`.** Selector-based waits adapt to network speed.
|
||||
- **Cookies via `headers["Cookie"]` only.**
|
||||
- **`extractRules` returns first match** — for arrays use `aiExtractRules` `type: "list"`.
|
||||
- **Set client timeout ≥ 300 s** to match the server deadline.
|
||||
- **`requestMetadata.status === "ok"` is the only success signal.**
|
||||
@@ -0,0 +1,186 @@
|
||||
# YouTube APIs
|
||||
|
||||
| Endpoint | Returns |
|
||||
|---|---|
|
||||
| `/scrape/youtube/search` | Search results — videos, shorts, channels, playlists |
|
||||
| `/scrape/youtube/video` | Single video metadata (stats, captions, related) |
|
||||
| `/scrape/youtube/channel` | Channel home / videos / shorts / playlists / community |
|
||||
| `/scrape/youtube/transcript` | Full transcript with millisecond offsets |
|
||||
|
||||
All synchronous `GET`. 10 credits each.
|
||||
|
||||
## YouTube Search
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/youtube/search",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": "anthropic claude", "sortBy": "views", "date": "month"},
|
||||
timeout=300,
|
||||
)
|
||||
for v in resp.json().get("videoResults", []):
|
||||
print(v["title"], v.get("extractedViews"), v["link"])
|
||||
```
|
||||
|
||||
### Query parameters
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `q` | **Required.** Free-text query. |
|
||||
| `sortBy` | `relevance` (default), `date`, `views`, `rating`, `popularity`. |
|
||||
| `date` | Upload window: `hour`, `today`, `week`, `month`, `year`. |
|
||||
| `length` | Duration bucket: `under4`, `between420`, `plus20`. |
|
||||
| `videoType` | `video`, `shorts`, `channel`, `playlist`, `movie`. |
|
||||
| `filters[]` | Feature flags ANDed: `hd`, `k4`, `hdr`, `subtitles`, `cc`, `d3`, `d360`, `vr180`, `live`, `bought`, `location`. |
|
||||
| `gl` / `hl` | Two-letter country / language codes. |
|
||||
| `deviceType` | `desktop`, `mobile`. |
|
||||
| `paginationToken` | Opaque cursor from the previous `pagination.nextPageToken`. |
|
||||
| `sp` | Raw YouTube `sp=` token (overrides `sortBy`, `date`, `videoType`, `length`, `filters[]`). |
|
||||
|
||||
Response: `videoResults`, `shortsResults`, `channelResults`, `playlistResults`, `adsResults`, `sponsoredResults`, `searchInformation`, `pagination`.
|
||||
|
||||
Per-video result keys (verified live): `videoId`, `title`, `link`, `channel`, `description`, `length`, `views`, `viewsOriginal`, `publishedDate`, `thumbnail`, `positionOnPage`. `channel` is an object — read `.channel.name` and `.channel.link`.
|
||||
|
||||
## YouTube Video
|
||||
|
||||
```python
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/youtube/video",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"v": "dQw4w9WgXcQ"},
|
||||
timeout=300,
|
||||
)
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `v` | **Required.** 11-character YouTube video ID — the `v=` query value. |
|
||||
| `gl` / `hl` | Country / language. |
|
||||
| `deviceType` | `desktop` / `mobile`. |
|
||||
|
||||
Top-level keys: `videoId`, `title`, `description`, `channel`, `views`, `extractedViews`, `likes`, `extractedLikes`, `lengthSeconds`, `publishedDate`, `keywords`, `captions`, `socialLinks`, `music`, `category`, `thumbnail`, `isFamilySafe`, `isUnlisted`, `relatedVideos`, `relatedShorts`, `endScreenVideos`, `requestMetadata`.
|
||||
|
||||
Use `extractedViews` / `extractedLikes` (integers) for math; `views` / `likes` are the formatted strings.
|
||||
|
||||
## YouTube Channel
|
||||
|
||||
```python
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/youtube/channel",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"channelId": "@MrBeast", "tab": "videos"},
|
||||
timeout=300,
|
||||
)
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `channelId` | **Required.** `@handle`, canonical `UC…` ID, or legacy `/c/<custom>` / `/user/<name>` slug. |
|
||||
| `tab` | `featured` (default), `videos`, `shorts`, `streams`, `playlists`, `posts` / `community`, `podcasts`, `releases`, `about`, `store`. |
|
||||
| `paginationToken` | Cursor for tabs that paginate. |
|
||||
| `gl` / `hl` / `deviceType` | Standard. |
|
||||
|
||||
Response: `channelInfo`, `featuredVideo`, `sections[]`.
|
||||
|
||||
`channelInfo` (verified live): `name`, `handle`, `channelId`, `channelUrl`, `avatar`, `banner`, `description`, `subscribers`, `extractedSubscribers`, `videosCount`, `extractedVideosCount`, `keywords[]`, `availableTabs[]`, `verified`, `websiteUrl`, `rssUrl`, `isFamilySafe`.
|
||||
|
||||
`sections[]` each have `title` + `items[]` — shape depends on the tab. Iterate generically:
|
||||
|
||||
```python
|
||||
for sec in resp.json().get("sections", []):
|
||||
for item in sec.get("items", []):
|
||||
...
|
||||
```
|
||||
|
||||
## YouTube Transcript
|
||||
|
||||
```python
|
||||
resp = requests.get(
|
||||
"https://api.hasdata.com/scrape/youtube/transcript",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"v": "dQw4w9WgXcQ", "languageCode": "en"},
|
||||
timeout=300,
|
||||
)
|
||||
text = " ".join(seg["snippet"] for seg in resp.json().get("transcript", []))
|
||||
```
|
||||
|
||||
| Param | Notes |
|
||||
|---|---|
|
||||
| `v` | **Required.** 11-character video ID. |
|
||||
| `languageCode` | BCP-47 / YouTube code (`en`, `de`, `en-US`, `pt-BR`). Must exist on the video. |
|
||||
| `type` | `asr` to fetch the auto-generated speech-recognition track when no human captions exist. |
|
||||
|
||||
Response: `transcript[]` and `availableTranscripts[]`.
|
||||
|
||||
Each `transcript[]` entry: `startMs`, `endMs`, `snippet`, `startTimeText` (e.g. `"0:18"`).
|
||||
|
||||
## Patterns
|
||||
|
||||
### Search → video → transcript fan-out
|
||||
|
||||
```python
|
||||
def topic_corpus(query, k=5):
|
||||
search = requests.get(
|
||||
"https://api.hasdata.com/scrape/youtube/search",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"q": query, "sortBy": "views"}, timeout=300,
|
||||
).json()
|
||||
docs = []
|
||||
for v in search.get("videoResults", [])[:k]:
|
||||
tr = requests.get(
|
||||
"https://api.hasdata.com/scrape/youtube/transcript",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"v": v["videoId"]}, timeout=300,
|
||||
).json()
|
||||
docs.append({
|
||||
"videoId": v["videoId"],
|
||||
"title": v["title"],
|
||||
"url": v["link"],
|
||||
"text": " ".join(s["snippet"] for s in tr.get("transcript", [])),
|
||||
})
|
||||
return docs
|
||||
```
|
||||
|
||||
### Channel velocity
|
||||
|
||||
```python
|
||||
def channel_velocity(handle):
|
||||
page = requests.get(
|
||||
"https://api.hasdata.com/scrape/youtube/channel",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"channelId": handle, "tab": "videos"}, timeout=300,
|
||||
).json()
|
||||
return [
|
||||
{"date": it.get("publishedDate"),
|
||||
"views": it.get("extractedViews"),
|
||||
"title": it.get("title")}
|
||||
for sec in page.get("sections", [])
|
||||
for it in sec.get("items", [])
|
||||
]
|
||||
```
|
||||
|
||||
### Timestamp search inside a transcript
|
||||
|
||||
```python
|
||||
def mentions(video_id, needle):
|
||||
tr = requests.get(
|
||||
"https://api.hasdata.com/scrape/youtube/transcript",
|
||||
headers={"x-api-key": API_KEY},
|
||||
params={"v": video_id}, timeout=300,
|
||||
).json()
|
||||
return [(s["startTimeText"], s["snippet"])
|
||||
for s in tr.get("transcript", [])
|
||||
if needle.lower() in s["snippet"].lower()]
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **`v` is the 11-char ID, not the URL.** Strip the `v=` value first.
|
||||
- **`languageCode` must exist on the video.** Inspect `availableTranscripts[]` if a fetch fails, then retry.
|
||||
- **`type=asr` is required when no human-authored caption track exists.** Otherwise the API errors on videos with auto-only captions.
|
||||
- **`@handle` resolves to the same channel as the canonical `UC…` ID.** Prefer handles for readability.
|
||||
- **Pagination tokens are opaque** — pass them back verbatim via `paginationToken`.
|
||||
- **`extractedViews` / `extractedLikes`** are integers; `views` / `likes` are formatted strings. Use the integer fields for arithmetic.
|
||||
- **`channelInfo.rssUrl`** is the canonical RSS feed for the channel — use it to subscribe in podcast clients without scraping.
|
||||
Reference in New Issue
Block a user