SocialCrawl
Cookbooks

Collect every review and watch for new ones

Pull a business's review history once, then collect only the reviews written since, with a bulk job that refreshes itself from the earlier job

Send the businesses you care about, and get back their reviews as one file. A week later, send the earlier job's id and get only the reviews written since. The example reads two Yelp businesses, and the same steps work for Tripadvisor places, App Store apps and Google Play apps.

Measured on 22/11/2026: a bulk job on 2 Yelp businesses with since=2026-11-01 read 6 reviews in 9 pages and was charged 3 credits. A refresh of that job right after it found 0 new reviews and was charged 3 credits.

Start in five minutes

Set your key once. Every block on this page reads it.

Terminal
export SOCIALCRAWL_API_KEY=sc_your_key_here

Paste this prompt into your coding agent:

Prompt
Read https://www.socialcrawl.dev/docs/recipes/every-review-and-new-ones.md and follow it to collect the Yelp reviews of the businesses la-folie-san-francisco-2 and zj8Lq1T8KIC5zwFief15jg written since 2026-11-01. Use the key in SOCIALCRAWL_API_KEY. Check the price in the roster first and show me what the job reserves. Send an Idempotency-Key, wait for the job with the status route at most once a minute, and never raise max_reviews_per_item without asking me.

Or read the newest page of one business yourself, which needs no job:

cURL
curl -G "https://www.socialcrawl.dev/v1/yelp/business/reviews" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode "id=zj8Lq1T8KIC5zwFief15jg" \
  --data-urlencode "limit=49" \
  --data-urlencode "sort=newest"

How it works

  • A single page call returns the newest reviews of one business, newest first.
  • A bulk job collects the history of many businesses in the background and delivers it as files.
  • You read the job's status for free until it ends, then download one combined file.
  • You store each review once by its id.
  • Next week you start a new job with refresh_of set to the earlier job, and it collects only what is newer.

What you need

CallWhat it givesCredits
GET /v1/yelp/business/reviewsOne page of a business's reviews, newest firstBilled per page, see the Yelp page
GET /v1/tripadvisor/reviewsA place's newest reviews, as many as depth asksSee the Tripadvisor page
POST /v1/yelp/business/reviews/historyOne business's whole history as a background jobSee the Yelp page
POST /v1/prism/reviews/bulkThe history of many businesses in one jobReserved up front, settled to the pages read, see the roster
The job's status route (Step 4)The job's progress, then its download linksFree to read

The same bulk job takes Tripadvisor places, App Store apps and Google Play apps through its platform field.

Steps

Step 1: read the newest page

Send sort=newest and limit=49 for the largest page. The reviews come back newest first, each with its id, rating, text, date and the owner's reply when there is one.

cURL
curl -G "https://www.socialcrawl.dev/v1/yelp/business/reviews" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode "id=zj8Lq1T8KIC5zwFief15jg" \
  --data-urlencode "limit=49" \
  --data-urlencode "sort=newest" \
  --data-urlencode "since=2026-11-15"
Python
import json, os, urllib.parse, urllib.request

query = {"id": "zj8Lq1T8KIC5zwFief15jg", "limit": "49", "sort": "newest", "since": "2026-11-15"}
url = "https://www.socialcrawl.dev/v1/yelp/business/reviews?" + urllib.parse.urlencode(query)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
page = json.load(urllib.request.urlopen(req, timeout=120))
for row in page["data"]["items"]:
    print(row["review"]["published_at"], row["review"]["rating"]["value"], row["review"]["id"])
  • since stops the walk at the first older review, because sort=newest is in date order.
  • On 22/11/2026 a page without since held 49 reviews from 21/11/2026 back to 04/06/2026 for 1 credit. With since=2026-11-15 it held 2 reviews, ended with stopped_at: since and cost 1 credit.
  • For Tripadvisor, GET /v1/tripadvisor/reviews takes the place's url_path and a depth. On 22/11/2026 depth=40 returned 40 reviews of one landmark for 1 credit, and has_more was false.

Step 2: start a history job for one business

For a single business with a long history, send its id or link to the history route.

cURL
curl -X POST "https://www.socialcrawl.dev/v1/yelp/business/reviews/history" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  -H "content-type: application/json" \
  -H "Idempotency-Key: reviews-history-1" \
  -d '{"id": "zj8Lq1T8KIC5zwFief15jg", "since": "2024-11-01"}'

It answers 202 with a job_id. Submitting holds the most the request can read and the job settles to the pages it read. Read the reviews 100 at a time with the history route with the job id added to its path, and offset and limit as query parameters, which costs 0 credits. This page's runs recorded the bulk job in step 3 instead, so this step has no measured run.

Step 3: collect many businesses in one bulk job

cURL
curl -X POST "https://www.socialcrawl.dev/v1/prism/reviews/bulk" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  -H "content-type: application/json" \
  -H "Idempotency-Key: reviews-bulk-1" \
  -d '{"platform": "yelp", "items": ["la-folie-san-francisco-2", "zj8Lq1T8KIC5zwFief15jg"], "since": "2026-11-01"}'
  • items is a JSON array of business ids, aliases or links. Send a real array, not a string.
  • since and until take a UTC day as YYYY-MM-DD. Any other field in the body is a free 400.
  • The job answers 202 and reserves credits in blocks, never the whole job at once. On 22/11/2026 the reservation was 25 credits.
  • An Idempotency-Key makes a retried submit return the same job without a second reservation. Retry safely explains the header.

Step 4: follow the job

Read the status route, GET /v1/prism/reviews/bulk/ followed by the job id, about once a minute until data.job.status is no longer queued or running. Reading is free.

Terminal
JOB_ID=job_your_job_id   # data.job_id from Step 3
API=https://www.socialcrawl.dev/v1
curl -sS "$API/prism/reviews/bulk/$JOB_ID?limit=10" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY"
Response (trimmed)
{
  "data": {
    "job": {
      "job_id": "job_JLdwzTaMDkww1OKlX9D3E",
      "status": "succeeded",
      "items_done": 2,
      "items_total": 2,
      "pages_read": 9,
      "reviews_collected": 6,
      "credits_charged": 3
    },
    "combined_url": "https://...",
    "manifest_url": "https://..."
  }
}
  • The job ends as succeeded and settles to the pages it read. The 25 credits it reserved became 3 charged.
  • On 22/11/2026 each job took a few minutes, so read the status once a minute and do not poll faster.
  • Download links are minted fresh on every read and expire after an hour, so read the status again when a link has expired.
  • combined_url is null when the job found no reviews.

Step 5: store each review by its id

The combined file is gzipped JSON lines. The first line of each business has type: business, and every review line has type: review, the item it belongs to and a review object with its id. Insert each review on its id and skip the ones you hold.

Step 6: refresh next week

Start a new job with the same platform and items and add refresh_of with the earlier job's id.

Terminal
JOB_ID=job_your_first_job_id   # data.job_id of the earlier job
curl -X POST "https://www.socialcrawl.dev/v1/prism/reviews/bulk" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  -H "content-type: application/json" \
  -H "Idempotency-Key: reviews-bulk-2" \
  -d '{"platform": "yelp", "items": ["la-folie-san-francisco-2", "zj8Lq1T8KIC5zwFief15jg"], "refresh_of": "'"$JOB_ID"'"}'

It reads the same items again, from the day the earlier job finished. The earlier job must still be kept, which is 7 days from its submit. On 22/11/2026 the refresh ran right after the first job, so it found 0 new reviews and was charged 3 credits.

Store only what is new

The review id is the key, and the newest stored date is your watermark. Save the id of each finished job next to your reviews, because the next refresh needs it. A job older than 7 days cannot be refreshed. Start a fresh job with since set to your newest stored date instead.

What it costs

A bulk job reserves credits in blocks and settles to the pages it read. A business that cannot be read costs nothing. The roster states the price of each call.

What the runs measured on 22/11/2026:

RunReviewsCredits
Yelp newest page, limit=49491
Yelp newest page with since=2026-11-1521
Tripadvisor, depth=40401
Bulk job, 2 Yelp businesses, since=2026-11-0163 charged, 25 reserved
Refresh of that job with refresh_of03 charged, 25 reserved

If you collect for many businesses every week, read Volume costs and email socialcrawl@ridio.io with your monthly volume.

Tips, and what not to use

  • A job pauses when your balance cannot cover the next reservation. It keeps what it has delivered and resumes by itself once you add credits.
  • A Tripadvisor place id that does not exist can answer an empty page instead of an error. Read the place from GET /v1/tripadvisor/search first.
  • A business with accented letters in its link is safest sent by its 22-character business id. Read the id from GET /v1/yelp/search.
  • Do not use this for products. For product reviews, use GET /v1/prism/product-reviews.
  • To pay only when something is new, use Listening or Monitors.

Full program and agent brief

Next steps