SocialCrawl
Cookbooks

Poll an account daily without paying twice

Fetch only the posts an account published since your last run, bounded by a date on the first run and by the newest post you hold on every run after

Send a handle, and get back only the posts published since your last run. The first run stops at a date. Every run after it stops at the newest post you already hold, so a quiet day reads one page and ends there.

Measured on 22/11/2026: a first run bounded by date read 36 videos of one news publisher in 4 pages for 4 credits. The next run, stopped at the newest video it held, read 1 page for 1 credit and found no new video.

Start in five minutes

Set your key once. Every block on this page reads it.

Terminal
export SOCIALCRAWL_API_KEY=sc_your_key_here

Paste this prompt into your coding agent:

Prompt
Read https://www.socialcrawl.dev/docs/recipes/poll-an-account-daily.md and follow it to poll the TikTok handle washingtonpost for new videos. Use the key in SOCIALCRAWL_API_KEY. Price one page with dry_run=true first and show me the estimate. On the first run use since=2026-11-20. On later runs use stop_at_id with the newest video you hold, and never raise the page cap without asking me.

Or price the first page yourself. A dry run quotes it for 0 credits:

cURL
curl -G "https://www.socialcrawl.dev/v1/tiktok/profile/videos" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode "handle=washingtonpost" \
  --data-urlencode "since=2026-11-20" \
  --data-urlencode "dry_run=true"

How it works

  • The first run has no post to stop at, so it sends since with a date and walks back to it.
  • You keep the newest post you received and save its id.
  • Every later run sends that id as stop_at_id. The page stops just before that post.
  • The same boundary goes on every page of one run, and the run ends when has_more is false.
  • Pinned posts sit out of date order, so they never count as the newest post you hold.

What you need

CallWhat it givesCredits
GET /v1/tiktok/profile/videos with since or stop_at_idThe posts newer than your boundary, one page at a timeBilled per page, see the roster
GET /v1/utility/estimate or dry_run=trueThe price of the request0

The same two parameters work on the Facebook, Instagram, Threads, X and YouTube post lists. The Pagination page names each endpoint.

Steps

Step 1: run once, bounded by a date

cURL
curl -G "https://www.socialcrawl.dev/v1/tiktok/profile/videos" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode "handle=washingtonpost" \
  --data-urlencode "since=2026-11-15"
Python
import json, os, urllib.parse, urllib.request

query = {"handle": "washingtonpost", "since": "2026-11-15"}
url = "https://www.socialcrawl.dev/v1/tiktok/profile/videos?" + urllib.parse.urlencode(query)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
page = json.load(urllib.request.urlopen(req, timeout=120))
print(len(page["data"]["items"]), page["pagination"]["has_more"], page["pagination"].get("stopped_at"))
  • since takes a date such as 2026-11-15 or an ISO 8601 timestamp. A date is read as midnight UTC.
  • While has_more is true, send next_cursor back as cursor with the same since.
  • The page that reaches an older post ends the walk with pagination.stopped_at set to since.
  • On 22/11/2026 the four pages read 7, 10, 10 and 9 videos, and the last ended with stopped_at: since and has_more: false.

Step 2: save the newest post you hold

Keep the id of the first row that is not pinned. A profile opens with the posts its owner pinned, and those can be years old.

Response (trimmed)
{
  "data": {
    "items": [
      { "post": { "id": "7710747895028337934", "published_at": "2026-11-21T22:00:06.000Z", "flags": { "pinned": false } } }
    ]
  },
  "credits_used": 1,
  "pagination": { "next_cursor": "sc.eyJ2Ijoy...", "has_more": true, "page_size": 7 }
}

Step 3: poll with stop_at_id

cURL
curl -G "https://www.socialcrawl.dev/v1/tiktok/profile/videos" \
  -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  --data-urlencode "handle=washingtonpost" \
  --data-urlencode "stop_at_id=7710747895028337934"
Python
import json, os, urllib.parse, urllib.request

query = {"handle": "washingtonpost", "stop_at_id": "7710747895028337934"}
url = "https://www.socialcrawl.dev/v1/tiktok/profile/videos?" + urllib.parse.urlencode(query)
req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]})
page = json.load(urllib.request.urlopen(req, timeout=120))
print(len(page["data"]["items"]), page["pagination"].get("stopped_at"))
  • The page stops just before the post you named. That post and every older one are left off.
  • pagination.stopped_at is known_id and next_cursor is null, so the run is over.
  • If the post is not on the page, the page comes back whole with its cursor. Keep walking with the same stop_at_id.
  • stop_at_id takes a post id or a post link. One longer than 512 characters is a free 400.
  • On 22/11/2026 this run came back with 3 pinned videos you already hold and no new video, stopped_at was known_id, and it cost 1 credit.

Step 4: schedule it

Run the program below once a day from cron or any scheduler. It keeps the newest post id and the ids it has seen in a small file. A bad date is a 400 that costs nothing: on 22/11/2026 a since that was not a date answered 400 INVALID_REQUEST for 0 credits.

Store only what is new

Send the same stop_at_id or since on every page of the walk, then keep the newest post you received for the next run.

  • Key each post on post.id.
  • Skip a row whose flags.pinned is true. It is not a new post, and it can come back on every run.
  • Save the newest id only after the whole run finishes. A run that fails halfway then starts again from the old boundary.
  • On YouTube, send stop_at_id rather than a date. YouTube list dates come from labels such as "4 months ago", so a date stop there is an estimate.

What it costs

Each page costs what a page of that endpoint costs. A poll that finds your newest post on page 1 pays for page 1 and stops there. The roster lists the price of every endpoint.

What the runs measured on 22/11/2026:

RunPagesCredits
The dry run00
First run, since=2026-11-15, 36 videos44
Second run, stop_at_id, no new video11
A since that is not a date00
The program below, first run with since=2026-11-20, 7 videos, served from cache10

Tips, and what not to use

  • A page emptied by since is still billed. If a daily check costs you more than the new posts are worth, Listening and Monitors bill when something is new.
  • Do not keep sending a date after the first run. The newest post you hold is a tighter boundary, so each poll reads fewer pages.
  • Do not poll more often than you need. Each poll that finds nothing still reads one page.
  • Start a Monitor when you want no cron job at all. A Monitor runs the check for you on a schedule you choose.

Full program and agent brief

Next steps