📦 deps(thirdparty): update snapshots
This commit is contained in:
ci[bot]
@@ -0,0 +1,169 @@
|
||||
---
|
||||
name: hasdata-cli
|
||||
description: Command-line access to search, scraping, and structured web data.
|
||||
risk: safe
|
||||
source: official
|
||||
source_type: official
|
||||
source_repo: HasData/hasdata-cli
|
||||
license: MIT
|
||||
license_source: "https://github.com/HasData/hasdata-cli/blob/main/LICENSE"
|
||||
date_added: "2026-06-04"
|
||||
---
|
||||
|
||||
# hasdata
|
||||
|
||||
Use the `hasdata` CLI for real-time web data. One subcommand per API — flags, enums, defaults are derived from the live schema at `api.hasdata.com/apis`.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use this skill when:
|
||||
|
||||
- The user wants to use the HasData CLI.
|
||||
- The user needs current web data from the command line.
|
||||
- The user wants to automate data collection in scripts.
|
||||
- The user wants to retrieve search, ecommerce, travel, or local business data.
|
||||
- The user needs web-page scraping through the CLI.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `command -v hasdata` — if missing, download the installer from `https://raw.githubusercontent.com/HasData/hasdata-cli/main/install.sh`, inspect it, then run it locally with `sh install.sh`.
|
||||
- One-time setup: the user runs `hasdata configure`, pastes their API key, and it's saved to `~/.hasdata/config.yaml` (mode 0600). Every future call picks it up automatically.
|
||||
- If a call fails with `no API key configured`, the user hasn't run `hasdata configure` yet — tell them to. **Never invent a key.**
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
hasdata <api> --flag value [--flag value ...] --raw | jq .
|
||||
```
|
||||
|
||||
Always pass `--raw` when piping to `jq` (skips pretty-print and TTY detection). Use `--pretty` only for human-readable terminal output.
|
||||
|
||||
## Picking the right subcommand
|
||||
|
||||
| User intent | Subcommand |
|
||||
| --- | --- |
|
||||
| Web search ("what does Google say about…") | `google-serp` (full features) or `google-serp-light` (cheap, single page) |
|
||||
| Latest news | `google-news` |
|
||||
| AI Mode SERP | `google-ai-mode` |
|
||||
| Shopping / product prices | `google-shopping` (broad), `amazon-search` / `amazon-product` (Amazon), `shopify-products` (Shopify) |
|
||||
| Immersive product page | `google-immersive-product` |
|
||||
| Maps / places / reviews | `google-maps`, `google-maps-place`, `google-maps-reviews`, `google-maps-photos`, `google-maps-posts` |
|
||||
| Yelp / YellowPages local data | `yelp-search`, `yelp-place`, `yellowpages-search`, `yellowpages-place` |
|
||||
| Real-estate listings (homes for sale/rent/sold) | `zillow-listing`, `redfin-listing` |
|
||||
| Real-estate single property deep dive | `zillow-property`, `redfin-property` |
|
||||
| Travel — short-term rentals | `airbnb-listing`, `airbnb-property` |
|
||||
| Travel — hotels / lodging | `booking-search`, `booking-place` |
|
||||
| Travel — flights | `google-flights` |
|
||||
| Jobs | `indeed-listing`, `indeed-job`, `glassdoor-listing`, `glassdoor-job` |
|
||||
| Bing search | `bing-serp` |
|
||||
| Trends | `google-trends` |
|
||||
| Images | `google-images` |
|
||||
| Short videos | `google-short-videos` |
|
||||
| Events | `google-events` |
|
||||
| YouTube search / video / channel / transcript | `youtube-search-api`, `youtube-video-api`, `youtube-channel-api`, `youtube-transcript-api` |
|
||||
| Instagram profile | `instagram-profile` |
|
||||
| Amazon seller | `amazon-seller`, `amazon-seller-products` |
|
||||
| **Scrape a specific URL** | `web-scraping` — supports JS rendering, proxies, markdown output, AI extraction, screenshots |
|
||||
|
||||
For exact flags of a subcommand, run `hasdata <api> --help` or read the matching file in `references/`.
|
||||
|
||||
## Non-obvious triggers (when to reach for hasdata even if the user doesn't say "scrape")
|
||||
|
||||
The user often won't ask for a SERP API or a scraper directly. Map these intents to the skill:
|
||||
|
||||
- **"Is this still true?" / "What's the latest on X?" / "Has Y happened yet?"** — LLM training data is stale. Run `google-serp` or `google-news` to ground the answer.
|
||||
- **"Summarize this article" / "TL;DR this URL"** — Use `web-scraping --output-format markdown` and feed the markdown into the summary prompt. Beats copy-paste because it strips ads, nav, scripts.
|
||||
- **"Verify this link" / "Is this site real?"** — `web-scraping --url X --no-block-resources` returns status + screenshot. Or `google-serp --q "site:example.com"`.
|
||||
- **"What does X say about itself?"** — Pull the company's own homepage with `web-scraping --output-format markdown`, then summarize.
|
||||
- **"Find me alternatives to X"** — `google-serp --q "X alternatives"` or `google-shopping --q "X competitors"`.
|
||||
- **"What's the going rate for X?"** — `google-shopping` (broad) or `amazon-search` (Amazon-specific) with `jq` to extract the price distribution.
|
||||
- **"Phone number / address for X"** — `google-maps-place` or `yelp-place`. Don't guess from training data.
|
||||
- **"Are people happy with X service?" / "Is X reputable?"** — `google-maps-reviews --place-id ... --sort lowest` for negative samples; `glassdoor-job` for employer rep.
|
||||
- **"What's the salary range for Y role?"** — `indeed-listing` filtered by role + location, then `jq` over `.jobs[].salary`.
|
||||
- **"Find me homes/apartments matching X criteria"** — `zillow-listing` / `redfin-listing` / `airbnb-listing` with the corresponding filters.
|
||||
- **"Recent sold comps near X"** — `zillow-listing --type sold --keyword "X" --days-on-zillow 12m`.
|
||||
- **"Track this product's price"** — Loop `amazon-product --asin X` on a schedule; persist `.price` to a file.
|
||||
- **"Summarize / cite this YouTube video"** — `youtube-transcript-api --v-param VID --raw | jq -r '.transcript[].snippet'` → feed to the summary prompt. Beats title/thumbnail-based guesses.
|
||||
- **"Find a hotel in $CITY for $DATES under $BUDGET"** — `booking-search --keyword $CITY --check-in-date X --check-out-date Y --adults 2 --children 0 --rooms 1 --price-max $BUDGET --sort priceLowestFirst`. For one specific property, `booking-place --url ...` returns the full room/rate matrix.
|
||||
- **"What's this channel pushing lately?"** — `youtube-channel-api --channel-id @handle --tab videos --raw | jq '.sections[].items[] | {title, publishedDate, views: .extractedViews}'`.
|
||||
- **"Does this business have an active offer / event?"** — `google-maps-posts --place-id X --raw | jq '.posts[] | {postedAt, description, cta}'`. Surfaces current promotions Google indexed.
|
||||
- **"What's trending around X?"** — `google-trends --q "X"` for relative interest; `google-news --q "X"` for headlines.
|
||||
- **"Find businesses near me that do X"** — `google-maps --q "X" --ll "@LAT,LNG,12z"` then fan out `google-maps-place` for contacts.
|
||||
- **"How does this look in country Y?"** — `--gl Y` on SERP commands, `--proxy-country Y` on `web-scraping`. Useful for geo-targeted SEO checks, geo-blocked content.
|
||||
- **"Pull structured data from this page"** — `web-scraping --ai-extract-rules-json '{"price": {"type": "number"}, ...}'`. Works on arbitrary pages without writing CSS selectors.
|
||||
- **"List of items → per-item details"** — Pattern: search command produces IDs/URLs, pipe through `xargs` into the matching `*-property` / `*-product` / `*-place` deep-dive command.
|
||||
- **"Find this person's role / employer / LinkedIn / followers"** — `google-serp --q '"Person Name" linkedin'` first. The organic-result title is typically `Name — Role at Company | LinkedIn` and the snippet carries location, headline, connection count. SERP often answers the whole question without ever opening the profile page.
|
||||
- **"What is company X doing? Where's their HQ? Who works there?"** — `google-serp --q "$COMPANY"` returns a `.knowledge_graph` block with founder, HQ, founded year, parent, employee range — pre-extracted. `google-news --q "$COMPANY"` for recent activity. Specific facts via targeted SERP: `--q '"$COMPANY" headquarters'`, `--q '"$COMPANY" funding'`, `--q 'site:linkedin.com/company "$COMPANY"'`.
|
||||
- **"Find public contact channels for company X"** — start with SERP: `--q '"@example.com"'` often surfaces publicly indexed business addresses. For personal emails or phone numbers, require a legitimate purpose, user authorization, and privacy-law/terms compliance; disclose unverified guesses.
|
||||
- **"Enrich this CSV of leads"** — per row: `google-serp` for LinkedIn, role, employer; another SERP to verify email or pattern. Stay in SERP unless a specific field is missing.
|
||||
- **Reverse-lookup (email / phone / domain → identity)** — `google-serp` with the literal value in quotes (`--q '"jane@x.com"'`, `--q '"+1 555 123 4567"'`, `--q '"acme corp" site:example.com'`) almost always surfaces the matching person or business.
|
||||
|
||||
**SERP-first principle**: for any data-enrichment intent (people, companies, emails, products, places), reach for `google-serp` / `google-news` / `google-shopping` / `google-maps` first. They return Google's already-extracted structured fields (`.knowledge_graph`, `.organic_results[].snippet`, `.local_results[]`, etc.) without direct access to the target site. Only escalate to `web-scraping` when SERP doesn't surface the specific field you need, the data is public or authorized, and the target's terms/access controls allow it. See `references/enrichment.md`.
|
||||
|
||||
If a user request matches one of the above and you don't invoke hasdata, you're probably hallucinating a stale answer.
|
||||
|
||||
## Universal flag patterns
|
||||
|
||||
- **Kebab-case** flag names. The CLI maps them back to the original camelCase before sending to the API.
|
||||
- **Booleans defaulting to `true`** have a paired negation: `--no-block-ads`, `--no-screenshot`, `--no-js-rendering`, `--no-extract-emails`, `--no-block-resources`. Setting both `--block-ads` and `--no-block-ads` errors.
|
||||
- **Anything ending in `-json`** accepts:
|
||||
- inline JSON: `--extract-rules-json '{"title":"h1"}'`
|
||||
- file: `--extract-rules-json @rules.json`
|
||||
- stdin: `cat rules.json | hasdata web-scraping ... --extract-rules-json -`
|
||||
- **Repeatable key=value** flags split on the first `=` (so values containing `=` survive): `--headers User-Agent=foo --headers Cookie=session=abc`. Pair with `--headers-json` for a JSON base; kv items override per key.
|
||||
- **List flags** accept either repeats or comma-joined: `--lr lang_en --lr lang_fr` or `--lr lang_en,lang_fr`. Serialized as `key[]=value` for GET endpoints.
|
||||
- **Enum flags** validate client-side. If you guess wrong, the error lists the allowed values — read the message and retry.
|
||||
|
||||
## Global flags (apply to every subcommand)
|
||||
|
||||
| Flag | Effect |
|
||||
| --- | --- |
|
||||
| `--raw` | Write response bytes as-is (use this when piping to `jq`) |
|
||||
| `--pretty` | Pretty-print JSON (default when stdout is a TTY) |
|
||||
| `-o, --output FILE` | Write response to file instead of stdout (works for binary like screenshots) |
|
||||
| `--verbose` | Log outgoing URL and `X-RateLimit-*` headers to stderr |
|
||||
| `--api-key KEY` | Override env var (rarely needed) |
|
||||
| `--timeout DURATION` | Per-request timeout (default 2m) |
|
||||
| `--retries N` | Max retries on 429/5xx (default 2) |
|
||||
|
||||
## Output contract
|
||||
|
||||
Responses are JSON. Pipe through `jq` for extraction:
|
||||
|
||||
```bash
|
||||
hasdata google-serp --q "espresso machine" --num 10 --raw \
|
||||
| jq -c '.organic_results[] | {title, link, snippet}'
|
||||
```
|
||||
|
||||
For real-estate / e-commerce results, the array shape is API-specific — read a single response with `--pretty` first to learn the schema, then write the `jq` filter.
|
||||
|
||||
## Exit codes (script-safe)
|
||||
|
||||
| Code | Meaning |
|
||||
| --- | --- |
|
||||
| 0 | success |
|
||||
| 1 | user / CLI-input error (missing required flag, bad enum value, missing API key) |
|
||||
| 2 | network error |
|
||||
| 3 | API returned 4xx (auth, quota, validation) |
|
||||
| 4 | API returned 5xx |
|
||||
|
||||
## References
|
||||
|
||||
- [`references/enrichment.md`](references/enrichment.md) — **person and company enrichment** (LinkedIn lookup, emails, HQ/funding/news, CSV-row enrichment, reverse-lookup) — the highest-leverage cross-API workflows
|
||||
- [`references/search.md`](references/search.md) — Google SERP / Bing / News / Trends flag catalog
|
||||
- [`references/web-scraping.md`](references/web-scraping.md) — `web-scraping` flags, JS scenarios, AI extraction
|
||||
- [`references/real-estate.md`](references/real-estate.md) — Zillow / Redfin filters and bracketed params
|
||||
- [`references/travel.md`](references/travel.md) — Airbnb / Booking / Google Flights (lodging + transport)
|
||||
- [`references/ecommerce.md`](references/ecommerce.md) — Amazon / Shopify
|
||||
- [`references/local-business.md`](references/local-business.md) — Maps (search/place/reviews/photos/posts) / Yelp / YellowPages
|
||||
- [`references/jobs.md`](references/jobs.md) — Indeed / Glassdoor
|
||||
- [`references/youtube.md`](references/youtube.md) — search / video / channel / transcript
|
||||
- [`references/all-commands.md`](references/all-commands.md) — full subcommand index with credit costs
|
||||
|
||||
|
||||
## Limitations
|
||||
|
||||
* Requires access to HasData services and valid credentials.
|
||||
* Data quality and available fields depend on the target website and extraction method used.
|
||||
* Website changes can impact extraction results and may require adjustments to extraction logic.
|
||||
* Rate limits, quotas, and account restrictions may apply depending on the endpoint and subscription plan.
|
||||
@@ -0,0 +1,107 @@
|
||||
# All commands
|
||||
|
||||
Authoritative source: `hasdata --help`. This file is a snapshot — when in doubt, run `hasdata <api> --help` directly.
|
||||
|
||||
## Search
|
||||
|
||||
| Command | Cost | Notes |
|
||||
| --- | --- | --- |
|
||||
| `google-serp` | 10 | Full Google SERP — organic, ads, knowledge graph, PAA, AI overview |
|
||||
| `google-serp-light` | 5 | Cheap single-page SERP |
|
||||
| `google-ai-mode` | 5 | Google AI Mode answer |
|
||||
| `google-news` | 10 | Google News results |
|
||||
| `google-shopping` | 10 | Google Shopping results |
|
||||
| `google-immersive-product` | 5 | Immersive product page details |
|
||||
| `google-events` | 5 | Google Events |
|
||||
| `google-short-videos` | 10 | Short videos panel |
|
||||
| `google-trends` | 5 | Search trends data |
|
||||
| `google-images` | 5 | Image search |
|
||||
| `bing-serp` | 10 | Bing SERP |
|
||||
|
||||
## Maps & local
|
||||
|
||||
| Command | Cost | Notes |
|
||||
| --- | --- | --- |
|
||||
| `google-maps` | 5 | Maps search |
|
||||
| `google-maps-place` | 5 | Single place by place_id |
|
||||
| `google-maps-reviews` | 5 | Place reviews |
|
||||
| `google-maps-contributor-reviews` | 5 | Reviews by contributor |
|
||||
| `google-maps-photos` | 5 | Place photos |
|
||||
| `google-maps-posts` | 10 | Business-owner posts (offers, events, announcements) |
|
||||
| `yelp-search` | 5 | Yelp business search |
|
||||
| `yelp-place` | 5 | Single Yelp business |
|
||||
| `yellowpages-search` | 5 | YellowPages search |
|
||||
| `yellowpages-place` | 5 | Single YellowPages listing |
|
||||
|
||||
## E-commerce
|
||||
|
||||
| Command | Cost | Notes |
|
||||
| --- | --- | --- |
|
||||
| `amazon-search` | 5 | Amazon search results |
|
||||
| `amazon-product` | 5 | Amazon product by ASIN |
|
||||
| `amazon-seller` | 5 | Amazon seller profile |
|
||||
| `amazon-seller-products` | 5 | Seller's product catalog |
|
||||
| `shopify-products` | 5 | Any Shopify store's products |
|
||||
| `shopify-collections` | 5 | Shopify collections |
|
||||
|
||||
## Real estate
|
||||
|
||||
| Command | Cost | Notes |
|
||||
| --- | --- | --- |
|
||||
| `zillow-listing` | 5 | Zillow filtered search |
|
||||
| `zillow-property` | 5 | Zillow single property |
|
||||
| `redfin-listing` | 5 | Redfin filtered search |
|
||||
| `redfin-property` | 5 | Redfin single property |
|
||||
|
||||
## Travel
|
||||
|
||||
| Command | Cost | Notes |
|
||||
| --- | --- | --- |
|
||||
| `airbnb-listing` | 5 | Airbnb filtered search |
|
||||
| `airbnb-property` | 5 | Airbnb single listing |
|
||||
| `booking-search` | 10 | Booking.com search (hotels, apartments) |
|
||||
| `booking-place` | 10 | Booking.com property + room/rate list |
|
||||
| `google-flights` | 15 | Flight search via Google Flights |
|
||||
|
||||
## Jobs
|
||||
|
||||
| Command | Cost | Notes |
|
||||
| --- | --- | --- |
|
||||
| `indeed-listing` | 5 | Indeed search |
|
||||
| `indeed-job` | 5 | Single Indeed posting |
|
||||
| `glassdoor-listing` | 10 | Glassdoor search (with ratings) |
|
||||
| `glassdoor-job` | 10 | Single Glassdoor posting |
|
||||
|
||||
## Web
|
||||
|
||||
| Command | Cost | Notes |
|
||||
| --- | --- | --- |
|
||||
| `web-scraping` | 10 | Arbitrary URL — JS rendering, AI extraction, markdown output, screenshots |
|
||||
|
||||
## Social
|
||||
|
||||
| Command | Cost | Notes |
|
||||
| --- | --- | --- |
|
||||
| `instagram-profile` | 5 | Instagram profile by handle |
|
||||
|
||||
## YouTube
|
||||
|
||||
| Command | Cost | Notes |
|
||||
| --- | --- | --- |
|
||||
| `youtube-search-api` | 10 | YouTube search (videos, shorts, channels, playlists) |
|
||||
| `youtube-video-api` | 10 | Single video metadata + stats + related |
|
||||
| `youtube-channel-api` | 10 | Channel home / videos / shorts / playlists / community |
|
||||
| `youtube-transcript-api` | 10 | Full transcript with millisecond offsets |
|
||||
|
||||
## Utility (no cost)
|
||||
|
||||
| Command | Notes |
|
||||
| --- | --- |
|
||||
| `configure` | Interactive setup; writes ~/.hasdata/config.yaml |
|
||||
| `version` | Print version |
|
||||
| `update` | Self-update from GitHub Releases |
|
||||
| `completion {bash\|zsh\|fish\|powershell}` | Generate shell completion |
|
||||
|
||||
## Deprecated
|
||||
|
||||
`amazon-reviews` is deprecated — Amazon now requires login to access reviews. The subcommand still exists for backwards compatibility but currently returns no data; don't suggest it.
|
||||
@@ -0,0 +1,106 @@
|
||||
# E-commerce reference
|
||||
|
||||
Subcommands: `amazon-search`, `amazon-product`, `amazon-seller`, `amazon-seller-products`, `shopify-products`, `shopify-collections`. 5 credits each.
|
||||
|
||||
---
|
||||
|
||||
## amazon-search
|
||||
|
||||
```bash
|
||||
hasdata amazon-search --q "wireless earbuds" [--domain amazon.com] --raw | jq '.results[]'
|
||||
```
|
||||
|
||||
- `--q TEXT` (required)
|
||||
- `--domain amazon.com|amazon.co.uk|amazon.de|amazon.in|amazon.co.jp|amazon.fr|amazon.it|amazon.es|amazon.com.br|amazon.com.mx|amazon.ca|amazon.com.au`
|
||||
- `--page N` — pagination
|
||||
- `--sort featured|price-asc-rank|price-desc-rank|review-rank|date-desc-rank`
|
||||
- `--node ID` — restrict to category node
|
||||
- `--customer-reviews` — only items with reviews
|
||||
|
||||
Per-result fields: `asin`, `title`, `price`, `link`, `image`, `rating`, `reviews_count`, `prime`.
|
||||
|
||||
## amazon-product
|
||||
|
||||
```bash
|
||||
hasdata amazon-product --asin B08N5WRWNW [--domain amazon.com] --raw | jq .
|
||||
```
|
||||
|
||||
- `--asin ASIN` (required)
|
||||
- `--domain` as above
|
||||
- `--include-reviews` — include first page of reviews
|
||||
- `--include-html` — return raw HTML alongside parsed data
|
||||
|
||||
Returns: title, price, availability, features, descriptions, variants, images, ratings, top reviews, A+ content.
|
||||
|
||||
## amazon-seller
|
||||
|
||||
```bash
|
||||
hasdata amazon-seller --seller-id A1234567890ABC [--domain amazon.com] --raw
|
||||
```
|
||||
|
||||
Seller profile: name, ratings, review count, returns policy, shipping policy, "About" content.
|
||||
|
||||
## amazon-seller-products
|
||||
|
||||
```bash
|
||||
hasdata amazon-seller-products --seller-id A1234567890ABC [--domain amazon.com] [--page 1] --raw
|
||||
```
|
||||
|
||||
List of products from a specific seller — useful for competitor analysis or storefront crawling.
|
||||
|
||||
---
|
||||
|
||||
## shopify-products
|
||||
|
||||
```bash
|
||||
hasdata shopify-products --url "https://store.example.com" [--page 1] --raw | jq '.products[]'
|
||||
```
|
||||
|
||||
Works on any Shopify store (it queries the public `/products.json`-style endpoint). Returns title, vendor, product_type, variants[] (with prices, SKUs, stock), images, tags, handle.
|
||||
|
||||
## shopify-collections
|
||||
|
||||
```bash
|
||||
hasdata shopify-collections --url "https://store.example.com" --raw | jq '.collections[]'
|
||||
```
|
||||
|
||||
For drilling into a specific collection, append `/collections/SLUG` to the URL or use the collection handle returned here.
|
||||
|
||||
---
|
||||
|
||||
## Non-obvious use cases
|
||||
|
||||
- **Cross-marketplace price arbitrage** — same `--asin` across `--domain amazon.com|amazon.co.uk|amazon.de` shows currency-normalized regional differences; useful for grey-market resellers and buyers shipping internationally.
|
||||
- **"Is this product still available?"** — `amazon-product --asin X --raw | jq '.availability'`. Avoids hallucinating an answer based on stale training data.
|
||||
- **Variant matrix dump** — `amazon-product --asin X --raw | jq '.variants[] | {asin, color, size, price}'` returns the full color/size/etc. lattice with current prices.
|
||||
- **Counterfeit-listing detection** — `amazon-search --q "BRAND PRODUCT" --raw` then check `.results[].seller_name` for non-authorized sellers; pivot to `amazon-seller` to inspect their other listings.
|
||||
- **Storefront catalog** — `amazon-seller-products --seller-id X --page 1..N` paginates a seller's full catalog; useful for competitor analysis or due diligence on a vendor.
|
||||
- **"What were the bestsellers in category X?"** — `amazon-search --q "CATEGORY KEYWORD" --sort featured` returns Amazon's own ranking (`--sort review-rank` for review-weighted).
|
||||
- **Lead enrichment for a Shopify store** — `shopify-products --url store.example.com --raw` exposes vendor names, tags, product types, SKUs — useful for competitor product-line audits.
|
||||
- **Stock check** — `shopify-products` returns each variant's `available` boolean; can power "notify me when X is back" without scraping the cart UI.
|
||||
- **Price-drop monitoring** — schedule `amazon-product --asin X` daily; persist `.price` to a file; alert when delta > N%.
|
||||
- **A/B-test detection** — same `--asin` from two different `--proxy-country` settings (via web-scraping fallback if amazon-product doesn't support it) sometimes shows different price/title due to A/B tests.
|
||||
- **Gift-card / coupon discovery** — `google-shopping --q "PRODUCT"` often surfaces resellers offering rebates Amazon doesn't show.
|
||||
- **Product-image extraction for a moodboard** — `amazon-product --asin X --raw | jq -r '.images[]'` returns CDN URLs you can download separately.
|
||||
- **Compare reviews summary** — `amazon-product --asin X --include-reviews --raw | jq '.reviews[] | {rating, title, body}'` gives a quick sentiment sample without scraping the review page.
|
||||
- **Build a Shopify product feed for ads** — `shopify-products --url X --raw | jq -c '.products[] | {id, title, price: .variants[0].price, url: ("https://" + $store + "/products/" + .handle)}'`.
|
||||
|
||||
## Common patterns
|
||||
|
||||
```bash
|
||||
# Price tracking — pull current price for a known ASIN
|
||||
hasdata amazon-product --asin "$ASIN" --raw | jq '.price'
|
||||
|
||||
# Product discovery → details fan-out
|
||||
hasdata amazon-search --q "$Q" --raw \
|
||||
| jq -r '.results[].asin' \
|
||||
| head -5 \
|
||||
| xargs -I{} hasdata amazon-product --asin {} --raw
|
||||
|
||||
# Compare across marketplaces
|
||||
for d in amazon.com amazon.co.uk amazon.de; do
|
||||
echo "=== $d ==="
|
||||
hasdata amazon-product --asin "$ASIN" --domain "$d" --raw \
|
||||
| jq '{currency: .currency, price: .price}'
|
||||
done
|
||||
```
|
||||
@@ -0,0 +1,227 @@
|
||||
# Data enrichment
|
||||
|
||||
Enriching a company, domain, or authorized contact list with public data. Use these workflows only for permitted business research or user-authorized contact enrichment, and respect site terms, robots/access controls, privacy law, opt-out obligations, and rate limits.
|
||||
|
||||
## SERP first. Web-scraping is the last resort.
|
||||
|
||||
`google-serp` is your primary enrichment tool. Reasons:
|
||||
|
||||
- **Google has already extracted the structured fields you want.** `.knowledge_graph` carries HQ, founder, founded year, parent company, employees, industry. `.organic_results[]` titles and snippets carry person → role → employer mappings (LinkedIn titles are literally `Name — Role at Company`). `.local_results[]` carry phone/address/hours.
|
||||
- **It avoids unnecessary direct access.** Many target sites gate, rate-limit, or restrict scraping. Google's snippets can answer public, high-level questions without rendering the target page.
|
||||
- **It's a broad public index.** Quoted queries (`--q '"info@example.com"'`, `--q '"+1 555 123 4567"'`, `--q '"Acme Corp"'`) can find publicly indexed business contact or company references.
|
||||
|
||||
Use `google-serp` (or `google-news` for recency, `google-maps` for places, `google-shopping` for products) **first**. Only fall through to `web-scraping` when:
|
||||
- A specific field you need isn't in any SERP snippet, AND
|
||||
- The target page renders that field server-side or via JS that the scraper can handle, AND
|
||||
- The user explicitly needs it and has authority to access it (don't fan out to N web-scraping calls when SERP would have answered N - 0 of them).
|
||||
|
||||
The patterns below show the full chain so you understand when to escalate. Most rows in a real CSV stop after step 1 or 2.
|
||||
|
||||
---
|
||||
|
||||
## Person enrichment
|
||||
|
||||
### Step 1 — SERP for role, employer, LinkedIn URL
|
||||
|
||||
```bash
|
||||
hasdata google-serp --q '"Jane Doe" linkedin' --num 5 --raw \
|
||||
| jq -c '.organic_results[] | select(.link | contains("linkedin.com/in/")) |
|
||||
{title, snippet, link}'
|
||||
```
|
||||
|
||||
The result usually looks like:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Jane Doe — Senior Engineer at Acme Corp | LinkedIn",
|
||||
"snippet": "San Francisco, CA · 500+ connections · Engineering @ Acme. Previously...",
|
||||
"link": "https://www.linkedin.com/in/janedoe"
|
||||
}
|
||||
```
|
||||
|
||||
You now have role, employer, location, LinkedIn URL, and a connection-count hint — without scraping anything. **Stop here unless a specific extra field is required.**
|
||||
|
||||
### Step 2 — Refine with targeted SERP queries
|
||||
|
||||
If step 1 didn't carry what you need, ask Google more specifically:
|
||||
|
||||
```bash
|
||||
# Disambiguate by company
|
||||
hasdata google-serp --q '"Jane Doe" "Acme Corp"' --num 10 --raw \
|
||||
| jq -c '.organic_results[] | {title, snippet, link}'
|
||||
|
||||
# Other social profiles
|
||||
hasdata google-serp --q '"Jane Doe" site:twitter.com OR site:x.com' --num 3 --raw
|
||||
hasdata google-serp --q '"Jane Doe" site:github.com' --num 3 --raw
|
||||
|
||||
# Past employers / bio paragraphs
|
||||
hasdata google-serp --q '"Jane Doe" bio OR background OR experience' --num 5 --raw \
|
||||
| jq -r '.organic_results[].snippet'
|
||||
```
|
||||
|
||||
### Step 3 — Web-scraping (only if SERP came up short)
|
||||
|
||||
When SERP snippets truncated the field you need, or the user explicitly wants full profile content, first confirm the profile is public or the user has authorization to access it:
|
||||
|
||||
```bash
|
||||
hasdata web-scraping --url "https://www.linkedin.com/in/janedoe" \
|
||||
--output-format markdown --no-screenshot --no-block-resources \
|
||||
--raw | jq -r .markdown
|
||||
```
|
||||
|
||||
Or for structured fields, AI extraction:
|
||||
|
||||
```bash
|
||||
hasdata web-scraping --url "https://www.linkedin.com/in/janedoe" \
|
||||
--ai-extract-rules-json '{
|
||||
"headline": {"type": "string"},
|
||||
"location": {"type": "string"},
|
||||
"company": {"type": "string"},
|
||||
"role": {"type": "string"},
|
||||
"followers": {"type": "number"},
|
||||
"experience": {"type": "list", "output": {
|
||||
"company": {"type": "string"},
|
||||
"role": {"type": "string"},
|
||||
"duration": {"type": "string"}
|
||||
}}
|
||||
}' --raw | jq .
|
||||
```
|
||||
|
||||
LinkedIn sometimes blocks the public preview; if it does, fall back to step 2 (combining SERP snippets) — it's almost always enough.
|
||||
|
||||
### Email lookup
|
||||
|
||||
Triangulate, don't promise. Use this only for business contact discovery, user-authorized enrichment, or another legitimate purpose. SERP first, scraping last; never present a guessed personal email as verified.
|
||||
|
||||
```bash
|
||||
# 1. Has Google already indexed the email anywhere?
|
||||
hasdata google-serp --q '"jane.doe@acme.com"' --num 10 --raw \
|
||||
| jq -c '.organic_results[] | {title, snippet, link}'
|
||||
|
||||
# 2. What email format does the company use? Look for any indexed @company.com address.
|
||||
hasdata google-serp --q 'site:acme.com "@acme.com"' --num 10 --raw \
|
||||
| jq -r '.organic_results[].snippet' \
|
||||
| grep -oE '[A-Za-z0-9._-]+@acme\.com' | sort -u
|
||||
|
||||
# 3. Pattern-guess + SERP-verify
|
||||
for guess in "jane.doe" "jdoe" "jane" "j.doe" "janed"; do
|
||||
count=$(hasdata google-serp --q "\"$guess@acme.com\"" --num 1 --raw \
|
||||
| jq -r '.organic_results | length')
|
||||
[ "$count" -gt 0 ] && echo "$guess@acme.com (appears in SERP)"
|
||||
done
|
||||
|
||||
# 4. Last resort — scrape the company's public contact / about / team pages for emails
|
||||
hasdata web-scraping --url "https://acme.com/about" --extract-emails --raw \
|
||||
| jq -r '.emails // [] | .[]'
|
||||
```
|
||||
|
||||
Always tell the user when an email is a pattern-guess vs. confirmed via SERP/scrape, and avoid collecting personal contact data when the user lacks authorization.
|
||||
|
||||
---
|
||||
|
||||
## Company enrichment
|
||||
|
||||
### Step 1 — SERP knowledge_graph
|
||||
|
||||
```bash
|
||||
hasdata google-serp --q "Acme Corp" --num 5 --raw | jq '.knowledge_graph // {}'
|
||||
```
|
||||
|
||||
`.knowledge_graph` typically contains: founder, founded (year), headquarters, parent_organization, ceo, employees (range), revenue, stock_price, industry, products. **For the majority of company enrichment requests, this single call is the entire answer.**
|
||||
|
||||
### Step 2 — Targeted SERP for specific fields
|
||||
|
||||
```bash
|
||||
# Headquarters
|
||||
hasdata google-serp --q '"Acme Corp" headquarters' --num 5 --raw \
|
||||
| jq -r '.organic_results[].snippet'
|
||||
|
||||
# Funding / acquisition signals
|
||||
hasdata google-serp --q '"Acme Corp" raises OR acquires OR acquired OR ipo OR funding' --num 10 --raw \
|
||||
| jq -c '.organic_results[] | {title, snippet, link}'
|
||||
|
||||
# Recent news
|
||||
hasdata google-news --q "Acme Corp" --gl us --raw \
|
||||
| jq -c '.news_results[] | {title, source: .source.name, date, link}'
|
||||
|
||||
# LinkedIn company page
|
||||
hasdata google-serp --q '"Acme Corp" site:linkedin.com/company' --num 3 --raw \
|
||||
| jq -c '.organic_results[] | {title, snippet, link}'
|
||||
|
||||
# Employee profiles in a specific function/region
|
||||
hasdata google-serp \
|
||||
--q 'site:linkedin.com/in "Acme Corp" engineer' --gl us --num 25 --raw \
|
||||
| jq -r '.organic_results[] | "\(.title)\t\(.link)"'
|
||||
```
|
||||
|
||||
### Step 3 — Web-scraping (only when SERP can't fill a specific field)
|
||||
|
||||
```bash
|
||||
# Company About page → AI-extract structured fields
|
||||
hasdata web-scraping --url "https://acme.com/about" \
|
||||
--ai-extract-rules-json '{
|
||||
"name": {"type": "string"},
|
||||
"founded": {"type": "number"},
|
||||
"headquarters": {"type": "string"},
|
||||
"employees": {"type": "string"},
|
||||
"industry": {"type": "string"},
|
||||
"description": {"type": "string"},
|
||||
"products": {"type": "list"}
|
||||
}' --raw | jq .
|
||||
```
|
||||
|
||||
Reach for this only when the user wants something SERP can't provide (e.g. mission statement verbatim, full product taxonomy, leadership team page parsed into rows).
|
||||
|
||||
---
|
||||
|
||||
## CSV row enrichment
|
||||
|
||||
For a list of N rows, fan out one or two SERP calls per row. Keep web-scraping out of the loop unless a specific row needs it.
|
||||
|
||||
```bash
|
||||
# Input: people.csv with one column "name"
|
||||
while IFS=, read -r name; do
|
||||
result=$(hasdata google-serp --q "\"$name\" linkedin" --num 1 --raw)
|
||||
linkedin=$(echo "$result" | jq -r '.organic_results[0].link // ""')
|
||||
title=$(echo "$result" | jq -r '.organic_results[0].title // ""')
|
||||
snippet=$(echo "$result" | jq -r '.organic_results[0].snippet // ""')
|
||||
printf '%s\t%s\t%s\t%s\n' "$name" "$title" "$snippet" "$linkedin"
|
||||
done < people.csv > enriched.tsv
|
||||
```
|
||||
|
||||
That's it. One SERP call per row, role/employer/LinkedIn extracted from the title and snippet. Add a second SERP call only if a row's first result didn't match (`select(.title | test("\(name)"; "i"))` filtering for confidence).
|
||||
|
||||
---
|
||||
|
||||
## Reverse-lookup
|
||||
|
||||
Always SERP-first with the literal value quoted. Use reverse lookup only for user-authorized investigation, business contact verification, or another legitimate purpose; do not use it for doxxing, stalking, harassment, or collecting private personal data.
|
||||
|
||||
```bash
|
||||
# Business email → public identity signal
|
||||
hasdata google-serp --q '"jane@example.com"' --num 10 --raw \
|
||||
| jq -c '.organic_results[] | {title, snippet, link}'
|
||||
|
||||
# Business phone → owner / business
|
||||
hasdata google-serp --q '"+1 555 123 4567"' --num 10 --raw
|
||||
# Combine with yelp-search / yellowpages-search if it's a business number.
|
||||
|
||||
# Domain → company
|
||||
hasdata google-serp --q "site:example.com" --num 5 --raw \
|
||||
| jq '.organic_results[0].title'
|
||||
hasdata google-serp --q "Acme Corp" --num 5 --raw | jq '.knowledge_graph // {}'
|
||||
```
|
||||
|
||||
Only scrape the domain (`web-scraping --url "https://example.com"`) when you specifically need the homepage's body text.
|
||||
|
||||
---
|
||||
|
||||
## Tips for reliable enrichment
|
||||
|
||||
- **Always quote names and other multi-token strings** — `"Jane Doe"` matches the exact person; `Jane Doe` matches noise.
|
||||
- **Use `site:` aggressively** — `site:linkedin.com/in/`, `site:linkedin.com/company/`, `site:github.com`, `site:crunchbase.com`. Google's `site:` is the cheapest way to scope an enrichment search.
|
||||
- **Read the `.knowledge_graph`** before doing anything else for a company. If it's populated, you're often done.
|
||||
- **AI-extract over CSS selectors** when you do need to scrape — LinkedIn / Crunchbase / About-page markup changes constantly; AI extraction with field names + descriptions survives layout churn.
|
||||
- **Cross-source verify** — never enrich from a single source. If LinkedIn's title says "Acme Corp" and a `--q '"Jane Doe" "Acme Corp"'` SERP corroborates with multiple results, confidence is high.
|
||||
- **Mark guesses** — pattern-guessed emails, inferred locations, single-source roles should be flagged to the user as unverified.
|
||||
- **Respect privacy and authorization** — do not collect or infer personal contact details without a legitimate purpose and user authority.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Jobs reference
|
||||
|
||||
Subcommands: `indeed-listing`, `indeed-job`, `glassdoor-listing`, `glassdoor-job`. Indeed: 5 credits. Glassdoor: 10 credits.
|
||||
|
||||
`*-listing` searches by query/location; `*-job` deep-dives a specific posting by URL or ID.
|
||||
|
||||
---
|
||||
|
||||
## indeed-listing
|
||||
|
||||
```bash
|
||||
hasdata indeed-listing \
|
||||
--query "backend engineer" --location "Remote" \
|
||||
[--days-since-posted 7] [--job-type fulltime|parttime|contract|temporary|internship] \
|
||||
[--remote] [--page 1] \
|
||||
--raw | jq '.jobs[] | {title, company, location, posted, salary, url}'
|
||||
```
|
||||
|
||||
Other flags worth knowing:
|
||||
- `--salary-min N` / `--salary-max N`
|
||||
- `--experience-level entry|mid|senior`
|
||||
- `--sort relevance|date`
|
||||
- `--country us|gb|ca|de|fr|...`
|
||||
|
||||
## indeed-job
|
||||
|
||||
```bash
|
||||
hasdata indeed-job --url "https://www.indeed.com/viewjob?jk=..." --raw | jq .
|
||||
```
|
||||
|
||||
Or `--job-key JK`. Returns full description, company info, benefits, hiring insights, similar jobs.
|
||||
|
||||
## glassdoor-listing
|
||||
|
||||
```bash
|
||||
hasdata glassdoor-listing --keyword "data scientist" --location "Boston" --raw \
|
||||
| jq '.jobs[] | {title, employer, salary_estimate, rating}'
|
||||
```
|
||||
|
||||
Glassdoor includes employer ratings and salary estimates per result. Use this when the user cares about employer reputation alongside the role.
|
||||
|
||||
## glassdoor-job
|
||||
|
||||
```bash
|
||||
hasdata glassdoor-job --url "https://www.glassdoor.com/Job/jobs.htm?...JV=..." --raw | jq .
|
||||
```
|
||||
|
||||
Returns full posting plus employer rating breakdown, recent reviews, salary estimate range, interview difficulty.
|
||||
|
||||
---
|
||||
|
||||
## Non-obvious use cases
|
||||
|
||||
- **Salary-negotiation research** — `indeed-listing --query ROLE --location CITY` then `jq '[.jobs[].salary | select(.)] | sort_by(.min)'` to build a defensible range before a comp conversation.
|
||||
- **"Should I move for this job?"** — same role across 3–5 cities; compare median salaries to local cost of living.
|
||||
- **Hiring-pattern intel on a competitor** — `indeed-listing --query "company:NAME"` (or use `glassdoor-listing` filtered by employer) returns recent postings; surfaces what teams are growing, what stack they're on, what locations they're in.
|
||||
- **"Are they really hiring or is this a ghost job?"** — `indeed-job --url X --raw | jq '.posted_at'`. Postings older than 60 days that haven't been refreshed are often ghosts.
|
||||
- **Stack popularity by region** — `indeed-listing --query "Rust" --location "Berlin"` vs `--location "San Francisco"` to compare absolute postings and salary deltas for a specific tech.
|
||||
- **Career-pivot research** — `indeed-listing --query "TARGET ROLE" --raw | jq -r '.jobs[].description'` then summarize the most-required skills with an LLM. Reveals the actual gap, not what bootcamps claim.
|
||||
- **Employer reputation deep-dive** — `glassdoor-listing` returns ratings inline; `glassdoor-job` returns recent reviews + interview difficulty. Use before applying or accepting.
|
||||
- **Remote-job filter** — `--remote` on `indeed-listing` cuts to fully-remote postings; use `--country gb` etc. to find role markets that are remote-friendly outside the US.
|
||||
- **Visa-friendly employer detection** — search `indeed-listing --query "ROLE H1B sponsorship"` or `--query "ROLE relocation"` — listings that mention these are more likely to support visa transfers.
|
||||
- **Internship-only filter** — `--job-type internship` on `indeed-listing` for early-career searches.
|
||||
- **Salary-band reverse engineering** — Glassdoor's `salary_estimate` is an estimate; cross-check by sampling 5–10 indeed postings for the same role/title in the same city and computing your own median.
|
||||
- **"Who's leaving Company X?"** — `glassdoor-listing --keyword "previously at: COMPANY"` is a stretch query but sometimes surfaces postings where ex-employees describe their transition.
|
||||
- **Detect layoffs before the news** — sudden surge in `indeed-listing --query "ex-COMPANY"` postings in a region often precedes the announcement.
|
||||
- **Negotiation prep — interview difficulty** — `glassdoor-job --url X --raw | jq '.interview_difficulty, .interview_experiences[]'` shows how previous candidates rated the process.
|
||||
|
||||
## Common patterns
|
||||
|
||||
```bash
|
||||
# Salary distribution for a role
|
||||
hasdata indeed-listing --query "senior python developer" --location "New York, NY" \
|
||||
--raw | jq '[.jobs[].salary | select(. != null)] | unique'
|
||||
|
||||
# Compare same role on both platforms
|
||||
for src in indeed-listing glassdoor-listing; do
|
||||
echo "=== $src ==="
|
||||
hasdata "$src" --query "platform engineer" --location "SF" --raw \
|
||||
| jq '.jobs[:5][] | {title, company: (.company // .employer), location}'
|
||||
done
|
||||
```
|
||||
|
||||
Start with Indeed for breadth; escalate to Glassdoor when employer ratings or interview-difficulty data are needed.
|
||||
@@ -0,0 +1,123 @@
|
||||
# Local-business / maps reference
|
||||
|
||||
Subcommands: `google-maps`, `google-maps-place`, `google-maps-reviews`, `google-maps-contributor-reviews`, `google-maps-photos`, `google-maps-posts`, `yelp-search`, `yelp-place`, `yellowpages-search`, `yellowpages-place`. 5 credits each (10 for `google-maps-posts` and YellowPages).
|
||||
|
||||
---
|
||||
|
||||
## google-maps
|
||||
|
||||
```bash
|
||||
hasdata google-maps --q "coffee" --ll "@30.2672,-97.7431,14z" --raw | jq '.local_results[]'
|
||||
```
|
||||
|
||||
Required: `--q TEXT`. Other useful flags:
|
||||
- `--ll "@LAT,LNG,ZOOMz"` — center point + zoom (Google's `ll` parameter)
|
||||
- `--gl us|gb|...` — country
|
||||
- `--hl en|es|...` — language
|
||||
- `--type search|place` — search vs. specific place lookup
|
||||
- `--data` / `--cid` / `--fid` — Google identifiers (advanced)
|
||||
|
||||
Per-result: `title`, `place_id`, `rating`, `reviews`, `address`, `phone`, `website`, `types[]`, `gps_coordinates`.
|
||||
|
||||
## google-maps-place
|
||||
|
||||
```bash
|
||||
hasdata google-maps-place --place-id "ChIJ..." --raw | jq .
|
||||
```
|
||||
|
||||
Single place: full details, hours, popular times, attributes.
|
||||
|
||||
## google-maps-reviews
|
||||
|
||||
```bash
|
||||
hasdata google-maps-reviews --place-id "ChIJ..." [--sort newest|highest|lowest] --raw \
|
||||
| jq '.reviews[] | {author, rating, date, snippet}'
|
||||
```
|
||||
|
||||
For pagination, the response includes a `next_page_token` — pass via `--next-page-token`.
|
||||
|
||||
## google-maps-contributor-reviews
|
||||
|
||||
Reviews authored by a specific Google contributor (by `--contributor-id`). Useful for local-guide analysis.
|
||||
|
||||
## google-maps-photos
|
||||
|
||||
```bash
|
||||
hasdata google-maps-photos --place-id "ChIJ..." --raw
|
||||
```
|
||||
|
||||
Returns photo URLs by category (interior, exterior, food, etc.).
|
||||
|
||||
## google-maps-posts (10 credits)
|
||||
|
||||
```bash
|
||||
hasdata google-maps-posts --place-id "ChIJ..." [--hl en] [--next-page-token TOKEN] --raw \
|
||||
| jq '.posts[] | {postedAt, description, cta, postUrl}'
|
||||
```
|
||||
|
||||
Posts are the business-owner publications shown on a Maps listing: offers, events, holiday hours, announcements. Either `--place-id` **or** `--data-id` is required.
|
||||
|
||||
Per-post fields (verified live): `postId`, `locationId`, `title`, `description`, `image`, `cta` (object with `label` + `url`), `createdAt` (ISO), `postedAt` (human-readable), `shareUrl`, `postUrl`. Use `pagination.nextPageToken` for older posts.
|
||||
|
||||
---
|
||||
|
||||
## yelp-search
|
||||
|
||||
```bash
|
||||
hasdata yelp-search --query "italian" --location "Brooklyn, NY" [--page 1] --raw \
|
||||
| jq '.businesses[] | {name, rating, review_count, price, categories}'
|
||||
```
|
||||
|
||||
## yelp-place
|
||||
|
||||
```bash
|
||||
hasdata yelp-place --url "https://www.yelp.com/biz/SLUG" --raw | jq .
|
||||
```
|
||||
|
||||
Single business: full details, hours, top reviews, photos, attributes.
|
||||
|
||||
## yellowpages-search
|
||||
|
||||
```bash
|
||||
hasdata yellowpages-search --search-terms "plumber" --geo-location-terms "Atlanta, GA" --raw
|
||||
```
|
||||
|
||||
## yellowpages-place
|
||||
|
||||
```bash
|
||||
hasdata yellowpages-place --url "https://www.yellowpages.com/atlanta-ga/mip/..." --raw
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Non-obvious use cases
|
||||
|
||||
- **Sales-lead research** — `google-maps --q "INDUSTRY" --ll "@LAT,LNG,12z"` to enumerate businesses, then `xargs` into `google-maps-place` for public phone/website details. Use email collection only for legitimate business outreach with opt-out, privacy-law, and rate-limit controls.
|
||||
- **Reputation monitoring** — `google-maps-reviews --place-id X --sort lowest` returns the worst reviews first; great for surfacing crisis signals fast. Run weekly to detect new 1-star drops.
|
||||
- **"Is this business open?"** — `google-maps-place --place-id X --raw | jq '.hours'` for current hours; also surfaces `permanently_closed` status.
|
||||
- **Verify an address user gave you** — `google-maps --q "BUSINESS NAME, CITY" --raw | jq '.local_results[0].address'`. Don't trust user-provided addresses for high-stakes actions (mailing, payments).
|
||||
- **Public business contact lookup** — `google-maps-place` includes phone and website; `web-scraping --url WEBSITE --extract-emails` returns emails parsed from the homepage. Use only for public business contact channels and disclose uncertainty.
|
||||
- **Competitive-density mapping** — `google-maps --q "coffee shop" --ll "@LAT,LNG,Zz"` at varying zoom levels; aggregate `.local_results[]` into a CSV with addresses + ratings to find under-served zones.
|
||||
- **Service-area validation** — chain `yelp-search --location "CITY"` for a few cities to confirm a business with the same name covers them.
|
||||
- **Find recently opened businesses** — `google-maps-place` returns `description.years_in_business` and posts/updates timestamps; sort.
|
||||
- **Negative-review sample for product analysis** — `yelp-place --url X --raw | jq '.reviews[] | select(.rating <= 2)'`; useful as input to a "what do customers complain about" summary.
|
||||
- **Photo-mining for visual docs** — `google-maps-photos --place-id X` returns categorized photo URLs (interior, food, exterior). Useful when the user needs an image and doesn't want to rely on `google-images`.
|
||||
- **Local-guide credibility** — `google-maps-contributor-reviews --contributor-id X` shows everything a specific reviewer wrote, useful for filtering out shilling/fake reviewers when their pattern is suspicious.
|
||||
- **YellowPages for B2B niches** — service-business categories (plumbers, electricians, lawyers) are often better indexed by YellowPages than Yelp; try both when one comes up empty.
|
||||
- **Cross-platform reputation diff** — same business name + city via `yelp-search` and `google-maps` to compare ratings across platforms (gap often signals fake reviews on one).
|
||||
- **Promo / event surveillance** — `google-maps-posts --place-id X` surfaces current offers, holiday hours, and limited-time events the business is actively pushing. Cheaper signal than scraping the website, and the `cta.url` typically links to the canonical landing page.
|
||||
- **Detect a business about to relaunch / rebrand** — sudden burst of new `google-maps-posts` after months of silence usually precedes a re-grand-opening or ownership change.
|
||||
|
||||
## Common patterns
|
||||
|
||||
```bash
|
||||
# Build a directory: search → fan out to per-place details
|
||||
hasdata google-maps --q "yoga studios" --ll "@30.27,-97.74,12z" --raw \
|
||||
| jq -r '.local_results[].place_id' \
|
||||
| head -10 \
|
||||
| xargs -I{} hasdata google-maps-place --place-id {} --raw
|
||||
|
||||
# Sentiment / review analysis
|
||||
hasdata google-maps-reviews --place-id "$PID" --sort lowest --raw \
|
||||
| jq '.reviews[] | {rating, snippet}'
|
||||
```
|
||||
@@ -0,0 +1,126 @@
|
||||
# Real-estate reference
|
||||
|
||||
Subcommands: `zillow-listing`, `zillow-property`, `redfin-listing`, `redfin-property` — 5 credits each.
|
||||
|
||||
For short-term rentals (Airbnb), hotels (Booking) and flights, see `travel.md`.
|
||||
|
||||
`*-listing` is for filtered searches; `*-property` is a single-property deep dive.
|
||||
|
||||
---
|
||||
|
||||
## zillow-listing
|
||||
|
||||
```bash
|
||||
hasdata zillow-listing --keyword "Austin, TX" --type forSale [filters] --raw | jq '.results[]'
|
||||
```
|
||||
|
||||
Required:
|
||||
- `--keyword "City, ST"` (default: `New York, NY`)
|
||||
- `--type forSale|forRent|sold` (default: `forSale`)
|
||||
|
||||
Price / size (bracketed pairs, kept as floats):
|
||||
- `--price-min N --price-max N`
|
||||
- `--beds-min N --beds-max N`
|
||||
- `--baths-min N --baths-max N`
|
||||
- `--square-feet-min N --square-feet-max N`
|
||||
- `--lot-size-min N --lot-size-max N`
|
||||
- `--year-built-min N --year-built-max N`
|
||||
- `--hoa N` — max HOA fee
|
||||
- `--parking-spots-min N`
|
||||
|
||||
Array filters (enum-validated, lowercase camelCase values):
|
||||
- `--home-types house|townhome|multiFamily|condo|lot|apartment|manufactured` (repeatable)
|
||||
- `--pets allowsLargeDogs|allowsSmallDogs|allowsCats` (repeatable)
|
||||
- `--other-amenities ac|pool|waterfront|onsiteParking|inUnitLaundry|acceptZillowApplications|incomeRestricted|apartmentCommunity` (repeatable)
|
||||
- `--views city|mountain|park|water` (repeatable)
|
||||
- `--basement finished|unfinished` (repeatable)
|
||||
- `--property-status comingSoon|acceptingBackupOffers|pendingAndUnderContract` (repeatable)
|
||||
- `--listing-publish-options ownerPosted|agentListed|newConstruction|foreclosures|auctions|foreclosed|preForeclosures` (repeatable)
|
||||
- `--tours open|3d` (repeatable)
|
||||
|
||||
Booleans:
|
||||
- `--must-have-garage` — only listings with a garage
|
||||
- `--single-story-only`
|
||||
- `--hide55plus-communities`
|
||||
|
||||
Other:
|
||||
- `--listing-type byAgent|byOwner`
|
||||
- `--days-on-zillow 1|7|14|30|90|6m|12m|24m|36m`
|
||||
- `--keywords "open floor plan"` — refinement keywords (matches in description)
|
||||
- `--move-in-date 2026-06-01`
|
||||
- `--page N` — pagination
|
||||
- `--sort verifiedSource|homesForYou|priceHighToLow|priceLowToHigh|paymentHighToLow|paymentLowToHigh|newest|bedrooms|bathrooms|squareFeet|lotSize`
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Family home, mid-market, sorted cheapest first
|
||||
hasdata zillow-listing \
|
||||
--keyword "Austin, TX" --type forSale \
|
||||
--price-min 400000 --price-max 900000 \
|
||||
--beds-min 3 --beds-max 5 --baths-min 2 \
|
||||
--home-types house --home-types townhome \
|
||||
--sort priceLowToHigh --raw | jq '.results[] | {address, price, beds, baths}'
|
||||
|
||||
# Pet-friendly rental
|
||||
hasdata zillow-listing \
|
||||
--keyword "Seattle, WA" --type forRent \
|
||||
--price-max 4000 \
|
||||
--pets allowsSmallDogs --pets allowsCats \
|
||||
--parking-spots-min 1 --must-have-garage \
|
||||
--raw
|
||||
|
||||
# Recently sold comps
|
||||
hasdata zillow-listing \
|
||||
--keyword "Miami, FL" --type sold \
|
||||
--square-feet-min 1500 --square-feet-max 4000 \
|
||||
--year-built-min 2000 --year-built-max 2020 \
|
||||
--days-on-zillow 12m --sort newest --raw
|
||||
```
|
||||
|
||||
Bracketed query params (`price[max]`, `homeTypes[]`, `yearBuilt[min]`) are handled by the CLI — pass the kebab-case flags shown above, not the raw API names.
|
||||
|
||||
## zillow-property
|
||||
|
||||
```bash
|
||||
hasdata zillow-property --url "https://www.zillow.com/homedetails/.../123_zpid/" --raw | jq .
|
||||
```
|
||||
|
||||
Or with `--zpid <ID>`. Returns full property details (photos, history, schools, taxes, walk-score, etc.).
|
||||
|
||||
## redfin-listing
|
||||
|
||||
Similar shape to `zillow-listing` but Redfin's enums differ. Run `hasdata redfin-listing --help` for the exact list. Common pattern:
|
||||
|
||||
```bash
|
||||
hasdata redfin-listing --location "San Francisco, CA" --status forSale \
|
||||
--min-price 800000 --max-price 1500000 \
|
||||
--min-beds 2 --raw
|
||||
```
|
||||
|
||||
## redfin-property
|
||||
|
||||
```bash
|
||||
hasdata redfin-property --url "https://www.redfin.com/CA/San-Francisco/.../home/12345" --raw
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Non-obvious use cases
|
||||
|
||||
- **Investment screening** — combine `--type sold` + `--days-on-zillow 12m` + `--year-built-min` + `--lot-size-min` to surface flip / value-add candidates. Then `xargs` into `zillow-property` for ARV analysis.
|
||||
- **Tax-appeal comps** — `--type sold --keyword "ZIP CODE" --days-on-zillow 12m` filtered to your home's beds/baths/sqft band gives recent sales the assessor used; export to CSV with `jq -r '.results[] | [.address, .price, .beds, .baths, .squareFootage, .soldDate] | @csv'`.
|
||||
- **Appraiser comp pull** — same trick, narrower square-footage and same year-built band.
|
||||
- **Motivated-seller signal** — `--type forSale --days-on-zillow 90` returns listings that have lingered. Often willing to negotiate.
|
||||
- **Pre-relocation neighborhood scan** — run the same `--type forRent` filter across 5–10 neighborhoods, dump rent distributions with `jq '.results[].price'`, eyeball cost differences before booking visits.
|
||||
- **STR-vs-LTR feasibility** — pair `airbnb-listing` (see `travel.md`) for nightly rates with `zillow-listing --type forSale` for purchase price in the same area; compute gross yield client-side.
|
||||
- **HOA filter** — `--hoa N` caps fee; useful for buyers who want max payment ceilings.
|
||||
- **School-driven house hunt** — `zillow-property` returns school ratings; filter `zillow-listing` results down by walking each property and keeping those with rating ≥ X.
|
||||
- **Open-houses this weekend** — Zillow tags open-house listings; check `.results[].openHouseTimes` for upcoming slots.
|
||||
- **3D-tour / virtual-tour-only filter** — `--tours 3d` → only listings with virtual tours. Useful for remote / international buyers.
|
||||
- **Pet-friendly rentals at scale** — `--pets allowsLargeDogs --pets allowsCats` for multi-pet households. Pairs well with `--keyword` for specific neighborhoods.
|
||||
- **Foreclosure and pre-foreclosure leads** — `--listing-publish-options foreclosures --listing-publish-options preForeclosures`.
|
||||
- **Non-traditional listing types** — `--listing-type byOwner` for FSBO; `--listing-type byAgent` for agent-listed (default mix).
|
||||
- **Move-in date constraint** — `--move-in-date YYYY-MM-DD` for rental searches with a hard timing requirement.
|
||||
- **Bulk address verification** — pipe a list of property URLs through `zillow-property` to confirm they resolve and pull the canonical address Zillow uses.
|
||||
- **Verify a Redfin/Zillow listing is real** — `redfin-property --url X --raw | jq .status` to confirm it hasn't been pulled.
|
||||
@@ -0,0 +1,122 @@
|
||||
# Search references
|
||||
|
||||
Subcommands: `google-serp`, `google-serp-light`, `google-ai-mode`, `google-news`, `google-shopping`, `bing-serp`, `google-trends`, `google-images`, `google-events`, `google-short-videos`, `google-immersive-product`.
|
||||
|
||||
For `google-flights`, see `travel.md`.
|
||||
|
||||
Run `hasdata <api> --help` for the live, authoritative flag set. Below are the commonly-used flags and example invocations.
|
||||
|
||||
---
|
||||
|
||||
## google-serp (10 credits)
|
||||
|
||||
```bash
|
||||
hasdata google-serp --q "QUERY" [--gl COUNTRY] [--hl LANG] [--num 10] [--start 0] --raw | jq .
|
||||
```
|
||||
|
||||
Common flags:
|
||||
- `--q TEXT` (required) — search query
|
||||
- `--gl us|gb|ca|de|fr|...` — country code; affects results
|
||||
- `--hl en|es|fr|de|...` — UI language
|
||||
- `--num 10..100` — results per page
|
||||
- `--start 0|10|20...` — pagination offset (multiples of 10)
|
||||
- `--location "Austin,Texas,United States"` — Google canonical location
|
||||
- `--device-type desktop|mobile|tablet`
|
||||
- `--tbm isch|vid|nws|shop|lcl` — search type
|
||||
- `--safe active|off`
|
||||
- `--lr lang_en --lr lang_fr` — restrict to language(s)
|
||||
- `--domain google.com|google.co.uk|...`
|
||||
- `--tbs cdr:1,cd_min:10/17/2018,cd_max:3/8/2021` — advanced search filters
|
||||
|
||||
Useful response fields (via `jq`):
|
||||
- `.organic_results[] | {title, link, snippet}` — main results
|
||||
- `.ai_overview` — AI Overview block (when present)
|
||||
- `.answer_box`, `.knowledge_graph`, `.related_searches`, `.people_also_ask`
|
||||
- `.local_results`, `.shopping_results`, `.news_results`
|
||||
|
||||
Example — top-10 organic for prompt grounding:
|
||||
```bash
|
||||
hasdata google-serp --q "$Q" --num 10 --raw \
|
||||
| jq -r '.organic_results[] | "- \(.title): \(.snippet)"'
|
||||
```
|
||||
|
||||
## google-serp-light (5 credits)
|
||||
|
||||
Same flags as `google-serp` but cheaper and returns a single page. Use when the user wants quick results and doesn't need PAA/AI Overview/local sections.
|
||||
|
||||
## google-ai-mode (5 credits)
|
||||
|
||||
Returns Google's AI Mode answer for a query. Same `--q` / `--gl` / `--hl` semantics.
|
||||
|
||||
## google-news (10 credits)
|
||||
|
||||
```bash
|
||||
hasdata google-news --q "QUERY" [--gl us] [--hl en] --raw | jq '.news_results[]'
|
||||
```
|
||||
|
||||
Per-article fields: `title`, `link`, `source.name`, `date`, `snippet`, `thumbnail`.
|
||||
|
||||
## google-shopping (10 credits)
|
||||
|
||||
```bash
|
||||
hasdata google-shopping --q "PRODUCT" [--gl us] --raw | jq '.shopping_results[]'
|
||||
```
|
||||
|
||||
Per-result: `title`, `link`, `price`, `extracted_price`, `source`, `rating`, `reviews`, `delivery`.
|
||||
|
||||
## bing-serp (10 credits)
|
||||
|
||||
```bash
|
||||
hasdata bing-serp --q "QUERY" [--cc us] [--setlang en] --raw | jq '.organic_results[]'
|
||||
```
|
||||
|
||||
Use when the user explicitly asks for Bing or wants a non-Google second opinion.
|
||||
|
||||
## google-trends (5 credits)
|
||||
|
||||
```bash
|
||||
hasdata google-trends --q "TERM" [--geo US] [--cat 0] [--time "today 12-m"] --raw | jq .
|
||||
```
|
||||
|
||||
Multiple terms for comparison: `--q "term1,term2,term3"` (comma-separated, max 5).
|
||||
|
||||
## google-images (5 credits)
|
||||
|
||||
```bash
|
||||
hasdata google-images --q "QUERY" --raw | jq '.images_results[] | {title, original, source}'
|
||||
```
|
||||
|
||||
## google-events (5 credits)
|
||||
|
||||
```bash
|
||||
hasdata google-events --q "concerts in austin" [--gl us] --raw | jq '.events_results[]'
|
||||
```
|
||||
|
||||
## google-short-videos (10 credits) / google-immersive-product (5 credits)
|
||||
|
||||
Less common — run `--help` to see flags when needed.
|
||||
|
||||
---
|
||||
|
||||
## Non-obvious use cases
|
||||
|
||||
- **Fact-check a claim before answering** — instead of relying on training data, run `google-serp --q "EXACT CLAIM"` and check whether top results corroborate or contradict.
|
||||
- **Resolve "what is the URL for X?"** — agents often hallucinate URLs. `google-serp --q "X official site" --num 3` and pick the result with the matching domain.
|
||||
- **Source-find a quote** — `google-serp --q "\"the exact quoted text here\""` (escape the inner quotes). Returns the page that originated it.
|
||||
- **Compare same query across regions** — same `--q`, different `--gl us|gb|de|fr` to see how SERP differs by geo. Useful for international SEO and "what do users in country Y see".
|
||||
- **Time-bound search** — `--tbs qdr:d` (past day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year). Combine with `google-news` for fresh-only news. Or `--tbs cdr:1,cd_min:M/D/YYYY,cd_max:M/D/YYYY` for an explicit window.
|
||||
- **Site-restricted search** — `--q "site:example.com TOPIC"` to search within one domain (better than scraping the site's own search box).
|
||||
- **"What's been written about X recently?"** — `google-news --q "X" --gl us` then `jq` over `.news_results[] | select(.date | test("hours? ago|day ago"))`.
|
||||
- **Discover competitors** — `google-serp --q "best alternatives to PRODUCT"` followed by extracting brand names from the top results' titles.
|
||||
- **Find documentation links** — `google-serp --q "LIBRARY official docs"` instead of guessing a URL pattern.
|
||||
- **Translate search intent** — `--hl de --gl de --q "the English query"` shows German-language ranking; useful for multi-locale SEO checks.
|
||||
- **Trends over time** — `google-trends --q "term1,term2"` (comma-separated, ≤5 terms) for relative interest curves. Better than guessing "is X popular now" from training data.
|
||||
- **Image-driven research** — `google-images --q "X"` to grab source URLs; pipe top results into `web-scraping` to read context.
|
||||
- **Map-distance trick** — for "things near X", use `google-maps --ll "@LAT,LNG,Zz"` not a SERP query — much higher signal for proximity intent.
|
||||
- **Fact-check pricing** — when training data has stale prices for SaaS or subscription services, search and read the pricing page rather than answering from memory.
|
||||
|
||||
## Picking the right SERP variant
|
||||
|
||||
- Use `google-serp-light` when only top organic results are needed (skips PAA / AI overview / local sections).
|
||||
- Use `google-serp` when you need the full SERP feature set (PAA, AI overview, knowledge graph, local pack).
|
||||
- Cache results client-side when running the same query repeatedly — the CLI does not cache.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Travel reference
|
||||
|
||||
Subcommands:
|
||||
- `airbnb-listing`, `airbnb-property` (5 credits each) — short-term rentals.
|
||||
- `booking-search`, `booking-place` (10 credits each) — hotels and other lodging on Booking.com.
|
||||
- `google-flights` (15 credits) — flight prices and itineraries via Google Flights.
|
||||
|
||||
`*-listing` / `*-search` is the filtered search; `*-property` / `*-place` is a single-property deep dive.
|
||||
|
||||
For activities at the destination see `google-events` (in `search.md`); for ground transport scrape the operator's site with `web-scraping`.
|
||||
|
||||
---
|
||||
|
||||
## airbnb-listing
|
||||
|
||||
```bash
|
||||
hasdata airbnb-listing --location "Lisbon, Portugal" \
|
||||
--check-in 2026-06-15 --check-out 2026-06-22 \
|
||||
--adults 2 --price-max 200 --raw
|
||||
```
|
||||
|
||||
Run `--help` for the full filter set (room type, amenities, instant book, etc.).
|
||||
|
||||
## airbnb-property
|
||||
|
||||
```bash
|
||||
hasdata airbnb-property --url "https://www.airbnb.com/rooms/12345678" --raw
|
||||
```
|
||||
|
||||
## booking-search (10 credits)
|
||||
|
||||
```bash
|
||||
hasdata booking-search \
|
||||
--keyword "Lisbon" \
|
||||
--check-in-date 2026-07-10 --check-out-date 2026-07-13 \
|
||||
--adults 2 --children 0 --rooms 1 \
|
||||
[--price-min 50 --price-max 250] [--rating 4 --rating 5] \
|
||||
[--review-score reviewScoreVeryGood --review-score reviewScoreSuperb] \
|
||||
[--property-type hotels --property-type apartments] \
|
||||
[--meals breakfastIncluded] [--facilities freeParking --facilities pool] \
|
||||
[--sort priceLowestFirst|ratingHighToLow|topReviewed|...] \
|
||||
[--page 2] [--currency USD] [--language en-us] \
|
||||
--raw | jq '.results[]'
|
||||
```
|
||||
|
||||
Required (no defaults work in production, even though `--help` shows them): `--keyword`, `--check-in-date`, `--check-out-date`, `--adults`, `--children`, `--rooms`. Pass `--children 0` explicitly when none.
|
||||
|
||||
When `--children > 0`, **also pass `--children-ages-json '[5,7]'`** with one age per child (0–17). Booking rejects the request otherwise.
|
||||
|
||||
Bracketed price filters (`--price-min` / `--price-max`) require `>= 10` / `>= 20` respectively; one of the two is required when filtering on price.
|
||||
|
||||
Top-level response: `results`, `searchInformation`, `pagination`, `requestMetadata`. Per-result keys (verified live): `hotelId`, `roomId`, `title`, `url`, `location`, `rating`, `reviews`, `price`, `room`, `beds`, `bedTypes`, `policies`, `photo`.
|
||||
|
||||
```bash
|
||||
# Cheap-first filtered search
|
||||
hasdata booking-search --keyword "Paris" \
|
||||
--check-in-date 2026-08-01 --check-out-date 2026-08-04 \
|
||||
--adults 2 --children 0 --rooms 1 \
|
||||
--review-score reviewScoreVeryGood \
|
||||
--sort priceLowestFirst --raw \
|
||||
| jq -c '.results[] | {title, price: .price.total, rating, url}'
|
||||
```
|
||||
|
||||
## booking-place (10 credits)
|
||||
|
||||
```bash
|
||||
hasdata booking-place \
|
||||
--url "https://www.booking.com/hotel/fr/le-bristol-paris.html" \
|
||||
--check-in-date 2026-07-10 --check-out-date 2026-07-13 \
|
||||
--adults 2 --children 0 --rooms 1 \
|
||||
[--currency USD] [--language en-us] \
|
||||
--raw | jq .
|
||||
```
|
||||
|
||||
Required: `--url` (must be on `booking.com` / `www.booking.com`), stay dates, `--adults`, `--children`, `--rooms`.
|
||||
|
||||
Response: `overview`, `bookingDetails`, `rooms[]`, `facilities`, `houseRules`, `ratings`, `reviews`, `restaurants`, `breadcrumbs`, `questionsAndAnswers`. `overview` carries `id`, `title`, `address`, `description`, `propertyType`, `photos`, `highlights`, `mostPopularFacilities`. Each `rooms[i]` has `roomId`, `name`, `bedTypes`, `beds`, `facilities`, `otherFacilities`, `variants[]` (pricing/availability per package).
|
||||
|
||||
## google-flights (15 credits)
|
||||
|
||||
```bash
|
||||
hasdata google-flights \
|
||||
--departure-id "JFK" --arrival-id "LAX" \
|
||||
--outbound-date 2026-06-15 --return-date 2026-06-22 \
|
||||
--currency USD --raw | jq .
|
||||
```
|
||||
|
||||
Round-trip vs. one-way controlled by presence/absence of `--return-date`. Run `--help` for the full flag set (cabin class, max stops, preferred airlines, etc.).
|
||||
|
||||
---
|
||||
|
||||
## Non-obvious use cases
|
||||
|
||||
- **Hotel-vs-rental arbitrage** — same dates and party size via `booking-search` and `airbnb-listing`; compare nightly cost percentile-for-percentile. The cheaper platform isn't always the same one across cities.
|
||||
- **Conference-room pricing audit** — `booking-search --keyword "$CITY" --check-in-date $START --check-out-date $END --sort priceHighestFirst` during a known conference window vs an idle week; the delta is the conference premium.
|
||||
- **Family-friendly filter** — `--children-ages-json '[5,9]' --children 2 --rooms 1 --travel-group family` and Booking returns only properties that accept the party size with appropriate beds.
|
||||
- **Loyalty-program portfolio** — `booking-search --keyword "$CITY" --raw | jq '.results[] | select(.title | test("Marriott|Hilton|Hyatt"))'` filters to chains you collect points with.
|
||||
- **Airbnb price-arbitrage check** — same dates, same area, two `airbnb-listing` calls with different `--adults` counts to surface listings that don't scale per-person price. Sometimes the difference is the deal.
|
||||
- **STR-vs-LTR feasibility** — pair `airbnb-listing` for nightly rates with `zillow-listing --type forSale` (see `real-estate.md`) for purchase price in the same area; compute gross yield client-side.
|
||||
- **Flights without a travel API** — `google-flights` with `--departure-id`, `--arrival-id`, dates, and `--currency` for ad-hoc fare checks against a vendor like Skyscanner.
|
||||
- **Multi-leg cost planning** — chain `google-flights` calls for each leg and sum `.best_flights[].price`; cheaper than the round-trip price-bot SaaS for one-off itineraries.
|
||||
- **Trip-cost preview** — combine `google-flights` (transport) + `booking-search` / `airbnb-listing` (lodging) + `google-events` (activities, see `search.md`) into one cost estimate before pitching a destination.
|
||||
@@ -0,0 +1,181 @@
|
||||
# web-scraping reference
|
||||
|
||||
Subcommand: `web-scraping` (10 credits/call). Single endpoint for arbitrary URL scraping with JS rendering, proxies, AI extraction, screenshots, markdown conversion.
|
||||
|
||||
> **`web-scraping` is the last resort, not the default.** Before reaching for it, ask: would `google-serp` (or `google-news` / `google-shopping` / `google-maps`) already have the field I need? Google's `.knowledge_graph`, `.organic_results[].snippet`, `.local_results[]` carry pre-extracted public facts without direct page access. Only invoke `web-scraping` when:
|
||||
>
|
||||
> - the user gave you a specific URL to read, OR
|
||||
> - SERP came up short for a specific field, OR
|
||||
> - the target page renders content that doesn't show up in any SERP snippet.
|
||||
>
|
||||
> Use it only for public pages or content the user is authorized to access, and respect site terms, robots/access controls, privacy law, and rate limits.
|
||||
>
|
||||
> See `references/enrichment.md` for the SERP-first patterns.
|
||||
|
||||
```bash
|
||||
hasdata web-scraping --url "URL" [flags] --raw | jq .
|
||||
```
|
||||
|
||||
## Required
|
||||
|
||||
- `--url URL` — target page
|
||||
|
||||
## Output formats
|
||||
|
||||
- `--output-format html|text|markdown|json` (repeatable)
|
||||
- One format → that format directly (e.g. `--output-format markdown` returns markdown text under `.markdown`)
|
||||
- Multiple → JSON response with one key per format
|
||||
- `--output-format json` combined with others → wraps everything in JSON
|
||||
|
||||
```bash
|
||||
# LLM-friendly markdown for prompt context
|
||||
hasdata web-scraping --url "$URL" --output-format markdown --raw | jq -r .markdown
|
||||
```
|
||||
|
||||
## Proxy & rendering
|
||||
|
||||
- `--proxy-type datacenter|residential` (default datacenter)
|
||||
- `--proxy-country US|UK|DE|IE|FR|IT|SE|BR|CA|JP|SG|IN|ID` (default US)
|
||||
- `--js-rendering` / `--no-js-rendering` (default on) — full headless browser
|
||||
- `--block-ads` / `--no-block-ads` (default on)
|
||||
- `--block-resources` / `--no-block-resources` (default on) — blocks images/CSS for speed
|
||||
- `--screenshot` / `--no-screenshot` (default on; the result includes a screenshot URL)
|
||||
- `--remove-base64-images` — strip inline base64 images from response
|
||||
- `--extract-emails` / `--no-extract-emails` (default on)
|
||||
- `--extract-links` (default off)
|
||||
|
||||
## Wait controls
|
||||
|
||||
- `--wait MS` — fixed wait after page load
|
||||
- `--wait-for "CSS_SELECTOR"` — wait until selector appears
|
||||
|
||||
## Custom JS scenario (complex array — JSON only)
|
||||
|
||||
```bash
|
||||
hasdata web-scraping --url "$URL" \
|
||||
--js-scenario-json '[
|
||||
{"wait": 2000},
|
||||
{"click": ".load-more"},
|
||||
{"waitFor": ".item"},
|
||||
{"scrollY": 1000},
|
||||
{"fill": ["input#q", "espresso"]}
|
||||
]' --raw
|
||||
```
|
||||
|
||||
Supported actions: `evaluate`, `click`, `wait`, `waitFor`, `waitForAndClick`, `scrollX`, `scrollY`, `fill`. Executed sequentially.
|
||||
|
||||
Accepts raw JSON, `@file.json`, or `-` (stdin).
|
||||
|
||||
## Headers (kvSlice + JSON escape)
|
||||
|
||||
```bash
|
||||
# Repeatable kv form (splits on first `=`)
|
||||
hasdata web-scraping --url "$URL" \
|
||||
--headers "User-Agent=hasdata-cli" \
|
||||
--headers "Accept-Language=en-US,en;q=0.9" \
|
||||
--headers "Cookie=session=abc=def" \
|
||||
--raw
|
||||
|
||||
# JSON base + kv overrides
|
||||
hasdata web-scraping --url "$URL" \
|
||||
--headers-json '{"User-Agent":"base","X-Common":"shared"}' \
|
||||
--headers "User-Agent=override" \
|
||||
--raw
|
||||
```
|
||||
|
||||
## CSS-selector data extraction (kvSlice or JSON)
|
||||
|
||||
```bash
|
||||
# Lightweight kv form: --extract-rules KEY=SELECTOR
|
||||
hasdata web-scraping --url "https://quotes.toscrape.com" \
|
||||
--extract-rules "quote=.quote .text" \
|
||||
--extract-rules "author=.quote .author" \
|
||||
--raw | jq .
|
||||
|
||||
# JSON form for complex selectors / attributes
|
||||
hasdata web-scraping --url "$URL" \
|
||||
--extract-rules-json '{"title":"h1","links":"a @href","price":".price-now"}' \
|
||||
--raw
|
||||
```
|
||||
|
||||
`@href`, `@src`, etc. extract attributes. Without `@`, extracts text content.
|
||||
|
||||
## AI extraction (LLM-driven)
|
||||
|
||||
```bash
|
||||
hasdata web-scraping --url "$URL" \
|
||||
--ai-extract-rules-json '{
|
||||
"headline": {"type": "string", "description": "the main story headline"},
|
||||
"comments_count": {"type": "number"},
|
||||
"is_paid_content": {"type": "boolean"},
|
||||
"tags": {"type": "list", "description": "topic tags"},
|
||||
"author": {"type": "item", "output": {
|
||||
"name": {"type": "string"},
|
||||
"verified": {"type": "boolean"}
|
||||
}}
|
||||
}' --raw | jq .
|
||||
```
|
||||
|
||||
Supported types: `string`, `number`, `boolean`, `list`, `item` (nested object — defines its shape under `output`).
|
||||
|
||||
## Tag filtering
|
||||
|
||||
- `--include-only-tags "main,article"` (comma-joined CSS selectors) — keep only matching elements
|
||||
- `--exclude-tags script --exclude-tags style` (repeatable) — remove elements
|
||||
|
||||
```bash
|
||||
hasdata web-scraping --url "$URL" \
|
||||
--output-format markdown \
|
||||
--include-only-tags "article,main" \
|
||||
--exclude-tags script --exclude-tags style --exclude-tags nav \
|
||||
--raw | jq -r .markdown
|
||||
```
|
||||
|
||||
## URL blocklist
|
||||
|
||||
```bash
|
||||
--block-urls-json '["**.googletagmanager.com/**","**.doubleclick.net/**"]'
|
||||
```
|
||||
|
||||
Glob patterns block specific subresource URLs from loading.
|
||||
|
||||
## Saving binary output
|
||||
|
||||
The `web-scraping` response is JSON, but if `--output-format` is set to a single non-JSON format, the wrapped result is still JSON. Use `jq -r .markdown > file.md` to extract text. For screenshots specifically, the response contains a screenshot URL — fetch it separately with `curl`.
|
||||
|
||||
## Non-obvious use cases
|
||||
|
||||
- **Page-to-prompt grounding** — `--output-format markdown` produces clean LLM-ready text from any URL. Strip nav/ads with `--exclude-tags script --exclude-tags style --exclude-tags nav`. Beats fetch + regex.
|
||||
- **JavaScript-rendered SPAs that `curl` can't read** — default `--js-rendering` uses a real browser, so React/Vue/Angular pages return their hydrated DOM, not the empty shell.
|
||||
- **Geo/availability testing where allowed** — `--proxy-type residential` can model residential network availability; use only for authorized tests where the target's terms and access controls permit it.
|
||||
- **Geo-targeted content** — `--proxy-country DE` to see what users in Germany see (different prices, currencies, A/B variants, or geo-blocked content).
|
||||
- **Quick "is this page real" check** — `--screenshot` (default on) returns a screenshot URL in the response; verify visually without manually opening the URL.
|
||||
- **Universal price extractor** — `--ai-extract-rules-json '{"price":{"type":"number"},"currency":{"type":"string"},"in_stock":{"type":"boolean"}}'` works on arbitrary retailer pages without writing a CSS selector. Cheaper than maintaining per-site selectors when the user only needs occasional spot-checks.
|
||||
- **Authenticated content with user authority** — `--headers Cookie=session=...` injects auth cookies if the user has them. Use only with explicit permission and authority to access that account/content; never use cookies to bypass someone else's access controls.
|
||||
- **Convert paginated lists to a clean record set** — combine `--js-scenario-json` (click "Load more" 5×) with `--ai-extract-rules-json` (pull the list shape). Lets you scrape paginated SPAs with one CLI call instead of N.
|
||||
- **Headless screenshot of a layout** — set `--js-rendering`, `--no-block-resources` (so CSS loads), and capture the screenshot URL from the response. Useful for "render this URL and show me what it looks like".
|
||||
- **Markdown for RAG ingestion** — pipe `.markdown` from many URLs into a JSONL corpus; embed and store. The CLI handles JS, ads, images so you don't need a custom pipeline.
|
||||
- **Fallback for any other API** — when no purpose-built API exists for a vertical (e.g. niche directories, government pages, less-popular real-estate sites), `web-scraping` is the catch-all.
|
||||
- **Detect content changes** — schedule `web-scraping --url X --output-format markdown` and diff the output across runs to flag pricing-page or terms-of-service changes.
|
||||
- **Read PDFs / non-HTML resources** — `--output-format text` works on text-extractable PDFs accessible via URL (the underlying renderer handles them).
|
||||
- **AI extraction for forms or tables** — pages with structured data in HTML tables are easy: `--ai-extract-rules-json '{"rows":{"type":"list","output":{"name":{"type":"string"},"value":{"type":"number"}}}}'`. The model fills in nested rows.
|
||||
|
||||
## Common patterns
|
||||
|
||||
```bash
|
||||
# Full-page markdown for RAG
|
||||
hasdata web-scraping --url "$URL" \
|
||||
--output-format markdown --no-screenshot --no-block-resources \
|
||||
--raw | jq -r .markdown >> corpus.md
|
||||
|
||||
# JS-heavy SPA: wait + scroll
|
||||
hasdata web-scraping --url "$URL" \
|
||||
--js-scenario-json '[{"wait":2000},{"scrollY":2000},{"wait":1500}]' \
|
||||
--wait-for ".item" \
|
||||
--output-format html --raw | jq -r .html
|
||||
|
||||
# Extract structured data from an arbitrary page
|
||||
hasdata web-scraping --url "$URL" \
|
||||
--ai-extract-rules-json '{"price":{"type":"number"},"in_stock":{"type":"boolean"}}' \
|
||||
--raw | jq .
|
||||
```
|
||||
@@ -0,0 +1,145 @@
|
||||
# YouTube reference
|
||||
|
||||
Subcommands: `youtube-search-api`, `youtube-video-api`, `youtube-channel-api`, `youtube-transcript-api`. 10 credits each.
|
||||
|
||||
`*-search-api` is keyword-driven; the others target a specific video, channel, or transcript. Most workflows chain search → video → transcript.
|
||||
|
||||
---
|
||||
|
||||
## youtube-search-api
|
||||
|
||||
```bash
|
||||
hasdata youtube-search-api --q "QUERY" [--sort-by relevance|date|views|rating|popularity] \
|
||||
[--length under4|between420|plus20] [--date hour|today|week|month|year] \
|
||||
[--video-type video|shorts|channel|playlist|movie] \
|
||||
[--gl us] [--hl en] [--device-type desktop|mobile] \
|
||||
[--pagination-token TOKEN] --raw | jq .
|
||||
```
|
||||
|
||||
Common flags:
|
||||
- `--q TEXT` (required)
|
||||
- `--sort-by relevance|date|views|rating|popularity`
|
||||
- `--date hour|today|week|month|year` — upload-recency window
|
||||
- `--length under4|between420|plus20` — duration bucket (`<4m`, `4–20m`, `>20m`)
|
||||
- `--video-type video|shorts|channel|playlist|movie`
|
||||
- `--filters hd,k4,hdr,subtitles,cc,d3,d360,vr180,live,bought,location` — feature flags ANDed
|
||||
- `--gl`, `--hl` — country / language
|
||||
- `--pagination-token` — copy from previous response's `pagination.nextPageToken`
|
||||
- `--sp` — raw YouTube `sp=` filter token (overrides `sort-by` / `date` / `video-type` / `length` / `filters`)
|
||||
|
||||
Top-level response keys: `videoResults`, `shortsResults`, `channelResults`, `playlistResults`, `adsResults`, `sponsoredResults`, `searchInformation`, `pagination`.
|
||||
|
||||
Per-video result: `videoId`, `title`, `link`, `channel`, `description`, `length`, `views`, `viewsOriginal`, `publishedDate`, `thumbnail`, `positionOnPage`.
|
||||
|
||||
```bash
|
||||
# Latest videos for a topic
|
||||
hasdata youtube-search-api --q "$Q" --sort-by date --date week --raw \
|
||||
| jq -c '.videoResults[] | {title, channel: .channel.name, views, publishedDate, link}'
|
||||
```
|
||||
|
||||
## youtube-video-api
|
||||
|
||||
```bash
|
||||
hasdata youtube-video-api --v-param VIDEO_ID [--gl us] [--hl en] --raw | jq .
|
||||
```
|
||||
|
||||
- `--v-param` (required) — 11-char video ID (the `v=` part of the watch URL)
|
||||
- `--device-type desktop|mobile`
|
||||
- `--gl`, `--hl`
|
||||
|
||||
Top-level fields: `videoId`, `title`, `description`, `channel`, `views`, `extractedViews`, `likes`, `extractedLikes`, `lengthSeconds`, `publishedDate`, `keywords[]`, `captions`, `socialLinks`, `music`, `category`, `thumbnail`, `isFamilySafe`, `isUnlisted`, `relatedVideos[]`, `relatedShorts[]`, `endScreenVideos[]`.
|
||||
|
||||
```bash
|
||||
# Quick stats
|
||||
hasdata youtube-video-api --v-param "$VID" --raw \
|
||||
| jq '{title, views: .extractedViews, likes: .extractedLikes, length: .lengthSeconds, published: .publishedDate, channel: .channel.name}'
|
||||
```
|
||||
|
||||
## youtube-channel-api
|
||||
|
||||
```bash
|
||||
hasdata youtube-channel-api --channel-id "@HANDLE_OR_UCID" \
|
||||
[--tab featured|videos|shorts|streams|playlists|posts|community|podcasts|releases|about|store] \
|
||||
[--gl us] [--hl en] [--pagination-token TOKEN] --raw | jq .
|
||||
```
|
||||
|
||||
- `--channel-id` (required) — `@handle`, `UC…` canonical ID, or legacy `/c/<custom>` / `/user/<name>` URL slug
|
||||
- `--tab` — which tab to scrape; default `featured` (Home page)
|
||||
- `--pagination-token` — for tabs that paginate (`videos`, `shorts`, etc.)
|
||||
|
||||
Top-level response: `channelInfo`, `featuredVideo`, `sections[]`.
|
||||
|
||||
`channelInfo` carries: `name`, `handle`, `channelId`, `channelUrl`, `avatar`, `banner`, `description`, `subscribers`, `extractedSubscribers`, `videosCount`, `extractedVideosCount`, `keywords[]`, `availableTabs[]`, `verified`, `websiteUrl`, `rssUrl`, `isFamilySafe`.
|
||||
|
||||
```bash
|
||||
# All uploads (paginated)
|
||||
hasdata youtube-channel-api --channel-id "@MrBeast" --tab videos --raw \
|
||||
| jq -c '.sections[].items[]? | {title, videoId, views, publishedDate}'
|
||||
```
|
||||
|
||||
## youtube-transcript-api
|
||||
|
||||
```bash
|
||||
hasdata youtube-transcript-api --v-param VIDEO_ID [--language-code en] [--type asr] --raw | jq .
|
||||
```
|
||||
|
||||
- `--v-param` (required) — 11-char video ID
|
||||
- `--language-code` — BCP-47 / YouTube code (`en`, `de`, `en-US`, `pt-BR`); must match a track the video actually has
|
||||
- `--type asr` — fetch the auto-generated speech-recognition track (omit for human-authored)
|
||||
|
||||
Response: `transcript[]` and `availableTranscripts[]`.
|
||||
|
||||
Each `transcript[]` entry: `startMs`, `endMs`, `snippet`, `startTimeText`. Join `snippet`s to reconstruct the full text.
|
||||
|
||||
```bash
|
||||
# Flatten to plain text
|
||||
hasdata youtube-transcript-api --v-param "$VID" --raw \
|
||||
| jq -r '.transcript[].snippet' | tr '\n' ' '
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Non-obvious use cases
|
||||
|
||||
- **"What is this video actually about?"** — `youtube-transcript-api --v-param X --raw | jq -r '.transcript[].snippet'` → feed to an LLM for a real summary, not a thumbnail / title guess.
|
||||
- **Cite YouTube content in a brief** — transcript + timestamps lets you quote with `(02:14)` accuracy. `--type asr` fills the gap when the creator never uploaded captions.
|
||||
- **Channel growth audit** — `youtube-channel-api --channel-id @X --tab videos` paginated; `jq '.sections[].items[] | {publishedDate, views: .extractedViews}'` gives a velocity-over-time table.
|
||||
- **Trend discovery** — `youtube-search-api --q "TOPIC" --sort-by views --date month` returns the highest-viewed videos in the last month — a leading indicator for cultural trends well before they hit Google News.
|
||||
- **Competitor content map** — `youtube-channel-api --channel-id @competitor --tab videos --raw | jq -r '.sections[].items[].title'` enumerates every published title, useful for content-gap analysis.
|
||||
- **Brand-safety scan** — `youtube-search-api --q "BRAND" --sort-by date --date week --raw | jq '.videoResults[] | select(.title | test("BRAND"; "i"))'` catches new mentions before they go viral.
|
||||
- **Influencer due diligence** — combine `youtube-channel-api` for subscriber/video counts with `youtube-video-api` on their top videos (engagement rate ≈ `likes / views`).
|
||||
- **"Find the moment they said X"** — `youtube-transcript-api` then `jq -r '.transcript[] | select(.snippet | test("X"; "i")) | "\(.startTimeText): \(.snippet)"'` returns timestamps of every mention.
|
||||
- **Music-licensing lookup** — `youtube-video-api --v-param X --raw | jq .music` returns identified tracks (artist, title) when YouTube's Content ID matched.
|
||||
- **Translate / re-localize a video** — pull the English transcript with `youtube-transcript-api`, translate via LLM, regenerate subtitles. Cheaper than re-transcribing audio.
|
||||
- **Build a podcast / RSS feed for a channel** — `youtube-channel-api --raw | jq -r .channelInfo.rssUrl` returns the official RSS URL; subscribe in any podcast app.
|
||||
- **Detect deleted videos** — `youtube-video-api --v-param X` returns an error for removed videos; useful for catching takedowns in archival pipelines.
|
||||
- **Bulk research → transcript chain** — `youtube-search-api --q "$Q" --raw | jq -r '.videoResults[].videoId' | xargs -I{} hasdata youtube-transcript-api --v-param {} --raw` builds a research corpus from a topic search.
|
||||
- **Shorts-only / long-form-only feeds** — `--video-type shorts` vs `--length plus20` to bias toward one format.
|
||||
- **Live-stream discovery** — `--filters live` returns only currently live broadcasts; pair with `--sort-by date` for fresh streams.
|
||||
|
||||
## Pipelines
|
||||
|
||||
```bash
|
||||
# Search → top 5 video transcripts as a corpus
|
||||
hasdata youtube-search-api --q "$Q" --sort-by views --raw \
|
||||
| jq -r '.videoResults[:5][].videoId' \
|
||||
| while read -r vid; do
|
||||
echo "=== $vid ==="
|
||||
hasdata youtube-transcript-api --v-param "$vid" --raw \
|
||||
| jq -r '.transcript[].snippet'
|
||||
done > corpus.txt
|
||||
|
||||
# Channel → CSV of all videos
|
||||
hasdata youtube-channel-api --channel-id "@$HANDLE" --tab videos --raw \
|
||||
| jq -r '.sections[].items[]? | [.videoId, .title, (.extractedViews // 0), .publishedDate] | @csv' \
|
||||
> channel_videos.csv
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **`--v-param` is exactly 11 chars** — not the full watch URL. Strip the `v=` value or use `jq -r '.videoResults[0].videoId'` from a prior search.
|
||||
- **`--language-code` must exist on the video.** Pass one of the codes listed in `availableTranscripts[]`, or omit for the default track.
|
||||
- **`--type asr` is needed when no human-authored captions exist.** Without it the API returns the default human track and errors if there isn't one.
|
||||
- **`@handle` is preferred over `UC…`** — easier to read, same result. Legacy `/c/` and `/user/` slugs also resolve.
|
||||
- **`pagination.nextPageToken` is opaque** — pass it back via `--pagination-token` verbatim; don't try to decode.
|
||||
- **`views` vs `extractedViews`** — `views` is the formatted string (`"1.2M views"`), `extractedViews` is the integer. Use the integer for math.
|
||||
Reference in New Issue
Block a user