# Every review, then new ones (/docs/recipes/every-review-and-new-ones) Pull a business's review history once, then collect only the reviews written since, with a bulk job that refreshes itself from the earlier job Send the businesses you care about, and get back their reviews as one file. A week later, send the earlier job's id and get only the reviews written since. The example reads two Yelp businesses, and the same steps work for Tripadvisor places, App Store apps and Google Play apps. Measured on 22/11/2026: a bulk job on 2 Yelp businesses with `since=2026-11-01` read 6 reviews in 9 pages and was charged 3 credits. A refresh of that job right after it found 0 new reviews and was charged 3 credits. ## Start in five minutes Set your key once. Every block on this page reads it. ```bash title="Terminal" export SOCIALCRAWL_API_KEY=sc_your_key_here ``` Paste this prompt into your coding agent: ```text title="Prompt" Read https://www.socialcrawl.dev/docs/recipes/every-review-and-new-ones.md and follow it to collect the Yelp reviews of the businesses la-folie-san-francisco-2 and zj8Lq1T8KIC5zwFief15jg written since 2026-11-01. Use the key in SOCIALCRAWL_API_KEY. Check the price in the roster first and show me what the job reserves. Send an Idempotency-Key, wait for the job with the status route at most once a minute, and never raise max_reviews_per_item without asking me. ``` Or read the newest page of one business yourself, which needs no job: ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/yelp/business/reviews" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode "id=zj8Lq1T8KIC5zwFief15jg" \ --data-urlencode "limit=49" \ --data-urlencode "sort=newest" ``` ## How it works - A single page call returns the newest reviews of one business, newest first. - A bulk job collects the history of many businesses in the background and delivers it as files. - You read the job's status for free until it ends, then download one combined file. - You store each review once by its id. - Next week you start a new job with `refresh_of` set to the earlier job, and it collects only what is newer. ## What you need | Call | What it gives | Credits | | --- | --- | --- | | `GET /v1/yelp/business/reviews` | One page of a business's reviews, newest first | Billed per page, see the [Yelp page](/docs/yelp.md) | | `GET /v1/tripadvisor/reviews` | A place's newest reviews, as many as `depth` asks | See the [Tripadvisor page](/docs/tripadvisor.md) | | `POST /v1/yelp/business/reviews/history` | One business's whole history as a background job | See the [Yelp page](/docs/yelp.md) | | `POST /v1/prism/reviews/bulk` | The history of many businesses in one job | Reserved up front, settled to the pages read, see the [roster](/docs/endpoint-pricing.md) | | The job's status route (Step 4) | The job's progress, then its download links | Free to read | The same bulk job takes Tripadvisor places, App Store apps and Google Play apps through its `platform` field. ## Steps ### Step 1: read the newest page Send `sort=newest` and `limit=49` for the largest page. The reviews come back newest first, each with its id, rating, text, date and the owner's reply when there is one. **cURL** ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/yelp/business/reviews" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode "id=zj8Lq1T8KIC5zwFief15jg" \ --data-urlencode "limit=49" \ --data-urlencode "sort=newest" \ --data-urlencode "since=2026-11-15" ``` **Python** ```python title="Python" import json, os, urllib.parse, urllib.request query = {"id": "zj8Lq1T8KIC5zwFief15jg", "limit": "49", "sort": "newest", "since": "2026-11-15"} url = "https://www.socialcrawl.dev/v1/yelp/business/reviews?" + urllib.parse.urlencode(query) req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}) page = json.load(urllib.request.urlopen(req, timeout=120)) for row in page["data"]["items"]: print(row["review"]["published_at"], row["review"]["rating"]["value"], row["review"]["id"]) ``` - `since` stops the walk at the first older review, because `sort=newest` is in date order. - On 22/11/2026 a page without `since` held 49 reviews from 21/11/2026 back to 04/06/2026 for 1 credit. With `since=2026-11-15` it held 2 reviews, ended with `stopped_at: since` and cost 1 credit. - For Tripadvisor, `GET /v1/tripadvisor/reviews` takes the place's `url_path` and a `depth`. On 22/11/2026 `depth=40` returned 40 reviews of one landmark for 1 credit, and `has_more` was `false`. ### Step 2: start a history job for one business For a single business with a long history, send its id or link to the history route. ```bash title="cURL" curl -X POST "https://www.socialcrawl.dev/v1/yelp/business/reviews/history" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ -H "content-type: application/json" \ -H "Idempotency-Key: reviews-history-1" \ -d '{"id": "zj8Lq1T8KIC5zwFief15jg", "since": "2024-11-01"}' ``` It answers `202` with a `job_id`. Submitting holds the most the request can read and the job settles to the pages it read. Read the reviews 100 at a time with the history route with the job id added to its path, and `offset` and `limit` as query parameters, which costs 0 credits. This page's runs recorded the bulk job in step 3 instead, so this step has no measured run. ### Step 3: collect many businesses in one bulk job ```bash title="cURL" curl -X POST "https://www.socialcrawl.dev/v1/prism/reviews/bulk" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ -H "content-type: application/json" \ -H "Idempotency-Key: reviews-bulk-1" \ -d '{"platform": "yelp", "items": ["la-folie-san-francisco-2", "zj8Lq1T8KIC5zwFief15jg"], "since": "2026-11-01"}' ``` - `items` is a JSON array of business ids, aliases or links. Send a real array, not a string. - `since` and `until` take a UTC day as `YYYY-MM-DD`. Any other field in the body is a free `400`. - The job answers `202` and reserves credits in blocks, never the whole job at once. On 22/11/2026 the reservation was 25 credits. - An `Idempotency-Key` makes a retried submit return the same job without a second reservation. [Retry safely](/docs/idempotency.md) explains the header. ### Step 4: follow the job Read the status route, `GET /v1/prism/reviews/bulk/` followed by the job id, about once a minute until `data.job.status` is no longer `queued` or `running`. Reading is free. ```bash title="Terminal" JOB_ID=job_your_job_id # data.job_id from Step 3 API=https://www.socialcrawl.dev/v1 curl -sS "$API/prism/reviews/bulk/$JOB_ID?limit=10" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" ``` ```json title="Response (trimmed)" { "data": { "job": { "job_id": "job_JLdwzTaMDkww1OKlX9D3E", "status": "succeeded", "items_done": 2, "items_total": 2, "pages_read": 9, "reviews_collected": 6, "credits_charged": 3 }, "combined_url": "https://...", "manifest_url": "https://..." } } ``` - The job ends as `succeeded` and settles to the pages it read. The 25 credits it reserved became 3 charged. - On 22/11/2026 each job took a few minutes, so read the status once a minute and do not poll faster. - Download links are minted fresh on every read and expire after an hour, so read the status again when a link has expired. - `combined_url` is `null` when the job found no reviews. ### Step 5: store each review by its id The combined file is gzipped JSON lines. The first line of each business has `type: business`, and every review line has `type: review`, the `item` it belongs to and a `review` object with its `id`. Insert each review on its id and skip the ones you hold. ### Step 6: refresh next week Start a new job with the same `platform` and `items` and add `refresh_of` with the earlier job's id. ```bash title="Terminal" JOB_ID=job_your_first_job_id # data.job_id of the earlier job curl -X POST "https://www.socialcrawl.dev/v1/prism/reviews/bulk" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ -H "content-type: application/json" \ -H "Idempotency-Key: reviews-bulk-2" \ -d '{"platform": "yelp", "items": ["la-folie-san-francisco-2", "zj8Lq1T8KIC5zwFief15jg"], "refresh_of": "'"$JOB_ID"'"}' ``` It reads the same items again, from the day the earlier job finished. The earlier job must still be kept, which is 7 days from its submit. On 22/11/2026 the refresh ran right after the first job, so it found 0 new reviews and was charged 3 credits. ## Store only what is new The review id is the key, and the newest stored date is your watermark. Save the id of each finished job next to your reviews, because the next refresh needs it. A job older than 7 days cannot be refreshed. Start a fresh job with `since` set to your newest stored date instead. ## What it costs A bulk job reserves credits in blocks and settles to the pages it read. A business that cannot be read costs nothing. The [roster](/docs/endpoint-pricing.md) states the price of each call. What the runs measured on 22/11/2026: | Run | Reviews | Credits | | --- | --- | --- | | Yelp newest page, `limit=49` | 49 | 1 | | Yelp newest page with `since=2026-11-15` | 2 | 1 | | Tripadvisor, `depth=40` | 40 | 1 | | Bulk job, 2 Yelp businesses, `since=2026-11-01` | 6 | 3 charged, 25 reserved | | Refresh of that job with `refresh_of` | 0 | 3 charged, 25 reserved | If you collect for many businesses every week, read [Volume costs](/docs/volume-costs.md) and email socialcrawl@ridio.io with your monthly volume. ## Tips, and what not to use - A job pauses when your balance cannot cover the next reservation. It keeps what it has delivered and resumes by itself once you add credits. - A Tripadvisor place id that does not exist can answer an empty page instead of an error. Read the place from `GET /v1/tripadvisor/search` first. - A business with accented letters in its link is safest sent by its 22-character business id. Read the id from `GET /v1/yelp/search`. - Do not use this for products. For product reviews, use `GET /v1/prism/product-reviews`. - To pay only when something is new, use [Listening](/docs/engines/listening.md) or [Monitors](/docs/engines/monitors.md). ## Full program and agent brief **Full program (Python)** ```python title="Python" # reviews_bulk.py: collect reviews of some businesses, then only the new ones next time. # Run: SOCIALCRAWL_API_KEY=... python3 reviews_bulk.py (Python 3.9+, nothing to install) import gzip import json import os import sqlite3 import time import urllib.request KEY = os.environ["SOCIALCRAWL_API_KEY"] BASE = "https://www.socialcrawl.dev/v1" BODY = { "platform": "yelp", "items": ["la-folie-san-francisco-2", "zj8Lq1T8KIC5zwFief15jg"], "since": "2026-11-01", # the first run only } STATE_FILE = "reviews_job.txt" # the id of the last finished job def call(url, body=None): data = json.dumps(body).encode() if body is not None else None headers = {"x-api-key": KEY} if data: headers["content-type"] = "application/json" req = urllib.request.Request(url, data=data, headers=headers) with urllib.request.urlopen(req, timeout=120) as res: return json.load(res) # 1. Start the job. After the first run, ask only for what is new since the last job. body = dict(BODY) if os.path.exists(STATE_FILE): body.pop("since", None) body["refresh_of"] = open(STATE_FILE).read().strip() job = call(f"{BASE}/prism/reviews/bulk", body)["data"] print("job", job["job_id"], job["status"], "reserved", job["credits_reserved"]) # 2. Wait for it, reading the status once a minute (reading is free). deadline = time.time() + 15 * 60 while True: time.sleep(60) status = call(f"{BASE}/prism/reviews/bulk/{job['job_id']}?limit=10")["data"] state = status["job"] print(state["status"], f"{state['items_done']}/{state['items_total']} businesses", state["reviews_collected"], "reviews") if state["status"] not in ("queued", "running"): break if time.time() > deadline: raise SystemExit("still running after 15 minutes: read the status again later") if state["status"] != "succeeded": raise SystemExit(f"job ended as {state['status']}: {state.get('credits_charged')} credits charged") # 3. Download the combined file and store each review once, keyed on its id. lines = [] if status["combined_url"]: # null when the job found no reviews with urllib.request.urlopen(status["combined_url"], timeout=120) as res: lines = gzip.decompress(res.read()).decode().splitlines() db = sqlite3.connect("reviews.db") db.execute("CREATE TABLE IF NOT EXISTS reviews (id TEXT PRIMARY KEY, business TEXT, published_at TEXT, rating INTEGER, text TEXT)") new = 0 for line in lines: row = json.loads(line) if row["type"] != "review": continue r = row["review"] cur = db.execute("INSERT OR IGNORE INTO reviews VALUES (?, ?, ?, ?, ?)", (r["id"], row["item"], r["published_at"], r["rating"]["value"], r["text"])) new += cur.rowcount db.commit() open(STATE_FILE, "w").write(job["job_id"]) print(f"{new} new reviews stored; job charged {state['credits_charged']} credits") ``` **Agent brief** ```text title="Agent brief" Goal: the Yelp reviews of two businesses since 2026-11-01, each stored once, then only new reviews on later runs. Read https://www.socialcrawl.dev/docs/recipes/every-review-and-new-ones.md first. 1. POST /v1/prism/reviews/bulk with platform, items as a real JSON array, since, and an Idempotency-Key. Show me credits_reserved. 2. Read GET /v1/prism/reviews/bulk/ followed by the job id about once a minute, for at most 15 minutes, until data.job.status is not queued or running. 3. Download combined_url, a gzipped file of JSON lines, and store each line with type review by review.id. 4. Save the job id. Next run, send the same body with refresh_of set to that job id and no since. 5. Never raise max_reviews_per_item without asking me. A paused job resumes when credits are added. ``` ## Next steps - [Yelp](/docs/yelp.md): Every Yelp endpoint, with the history job and its prices. - [Tripadvisor](/docs/tripadvisor.md): Place search and reviews, newest first. - [Monitors](/docs/engines/monitors.md): Get told when something changes, without a job of your own.