# Poll an account daily (/docs/recipes/poll-an-account-daily) Fetch only the posts an account published since your last run, bounded by a date on the first run and by the newest post you hold on every run after Send a handle, and get back only the posts published since your last run. The first run stops at a date. Every run after it stops at the newest post you already hold, so a quiet day reads one page and ends there. Measured on 22/11/2026: a first run bounded by date read 36 videos of one news publisher in 4 pages for 4 credits. The next run, stopped at the newest video it held, read 1 page for 1 credit and found no new video. ## Start in five minutes Set your key once. Every block on this page reads it. ```bash title="Terminal" export SOCIALCRAWL_API_KEY=sc_your_key_here ``` Paste this prompt into your coding agent: ```text title="Prompt" Read https://www.socialcrawl.dev/docs/recipes/poll-an-account-daily.md and follow it to poll the TikTok handle washingtonpost for new videos. Use the key in SOCIALCRAWL_API_KEY. Price one page with dry_run=true first and show me the estimate. On the first run use since=2026-11-20. On later runs use stop_at_id with the newest video you hold, and never raise the page cap without asking me. ``` Or price the first page yourself. A dry run quotes it for 0 credits: ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/tiktok/profile/videos" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode "handle=washingtonpost" \ --data-urlencode "since=2026-11-20" \ --data-urlencode "dry_run=true" ``` ## How it works - The first run has no post to stop at, so it sends `since` with a date and walks back to it. - You keep the newest post you received and save its id. - Every later run sends that id as `stop_at_id`. The page stops just before that post. - The same boundary goes on every page of one run, and the run ends when `has_more` is `false`. - Pinned posts sit out of date order, so they never count as the newest post you hold. ## What you need | Call | What it gives | Credits | | --- | --- | --- | | `GET /v1/tiktok/profile/videos` with `since` or `stop_at_id` | The posts newer than your boundary, one page at a time | Billed per page, see the [roster](/docs/endpoint-pricing.md) | | `GET /v1/utility/estimate` or `dry_run=true` | The price of the request | 0 | The same two parameters work on the Facebook, Instagram, Threads, X and YouTube post lists. The [Pagination](/docs/pagination.md#incremental-sync) page names each endpoint. ## Steps ### Step 1: run once, bounded by a date **cURL** ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/tiktok/profile/videos" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode "handle=washingtonpost" \ --data-urlencode "since=2026-11-15" ``` **Python** ```python title="Python" import json, os, urllib.parse, urllib.request query = {"handle": "washingtonpost", "since": "2026-11-15"} url = "https://www.socialcrawl.dev/v1/tiktok/profile/videos?" + urllib.parse.urlencode(query) req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}) page = json.load(urllib.request.urlopen(req, timeout=120)) print(len(page["data"]["items"]), page["pagination"]["has_more"], page["pagination"].get("stopped_at")) ``` - `since` takes a date such as `2026-11-15` or an ISO 8601 timestamp. A date is read as midnight UTC. - While `has_more` is `true`, send `next_cursor` back as `cursor` with the same `since`. - The page that reaches an older post ends the walk with `pagination.stopped_at` set to `since`. - On 22/11/2026 the four pages read 7, 10, 10 and 9 videos, and the last ended with `stopped_at: since` and `has_more: false`. ### Step 2: save the newest post you hold Keep the id of the first row that is not pinned. A profile opens with the posts its owner pinned, and those can be years old. ```json title="Response (trimmed)" { "data": { "items": [ { "post": { "id": "7710747895028337934", "published_at": "2026-11-21T22:00:06.000Z", "flags": { "pinned": false } } } ] }, "credits_used": 1, "pagination": { "next_cursor": "sc.eyJ2Ijoy...", "has_more": true, "page_size": 7 } } ``` ### Step 3: poll with `stop_at_id` **cURL** ```bash title="cURL" curl -G "https://www.socialcrawl.dev/v1/tiktok/profile/videos" \ -H "x-api-key: $SOCIALCRAWL_API_KEY" \ --data-urlencode "handle=washingtonpost" \ --data-urlencode "stop_at_id=7710747895028337934" ``` **Python** ```python title="Python" import json, os, urllib.parse, urllib.request query = {"handle": "washingtonpost", "stop_at_id": "7710747895028337934"} url = "https://www.socialcrawl.dev/v1/tiktok/profile/videos?" + urllib.parse.urlencode(query) req = urllib.request.Request(url, headers={"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}) page = json.load(urllib.request.urlopen(req, timeout=120)) print(len(page["data"]["items"]), page["pagination"].get("stopped_at")) ``` - The page stops just before the post you named. That post and every older one are left off. - `pagination.stopped_at` is `known_id` and `next_cursor` is `null`, so the run is over. - If the post is not on the page, the page comes back whole with its cursor. Keep walking with the same `stop_at_id`. - `stop_at_id` takes a post id or a post link. One longer than 512 characters is a free `400`. - On 22/11/2026 this run came back with 3 pinned videos you already hold and no new video, `stopped_at` was `known_id`, and it cost 1 credit. ### Step 4: schedule it Run the program below once a day from cron or any scheduler. It keeps the newest post id and the ids it has seen in a small file. A bad date is a `400` that costs nothing: on 22/11/2026 a `since` that was not a date answered `400 INVALID_REQUEST` for 0 credits. ## Store only what is new Send the same `stop_at_id` or `since` on every page of the walk, then keep the newest post you received for the next run. - Key each post on `post.id`. - Skip a row whose `flags.pinned` is `true`. It is not a new post, and it can come back on every run. - Save the newest id only after the whole run finishes. A run that fails halfway then starts again from the old boundary. - On YouTube, send `stop_at_id` rather than a date. YouTube list dates come from labels such as "4 months ago", so a date stop there is an estimate. ## What it costs Each page costs what a page of that endpoint costs. A poll that finds your newest post on page 1 pays for page 1 and stops there. The [roster](/docs/endpoint-pricing.md) lists the price of every endpoint. What the runs measured on 22/11/2026: | Run | Pages | Credits | | --- | --- | --- | | The dry run | 0 | 0 | | First run, `since=2026-11-15`, 36 videos | 4 | 4 | | Second run, `stop_at_id`, no new video | 1 | 1 | | A `since` that is not a date | 0 | 0 | | The program below, first run with `since=2026-11-20`, 7 videos, served from cache | 1 | 0 | ## Tips, and what not to use - A page emptied by `since` is still billed. If a daily check costs you more than the new posts are worth, [Listening](/docs/engines/listening.md) and [Monitors](/docs/engines/monitors.md) bill when something is new. - Do not keep sending a date after the first run. The newest post you hold is a tighter boundary, so each poll reads fewer pages. - Do not poll more often than you need. Each poll that finds nothing still reads one page. - Start a Monitor when you want no cron job at all. A Monitor runs the check for you on a schedule you choose. ## Full program and agent brief **Full program (Python)** ```python title="Python" # poll_account.py: fetch only the videos posted since the last run, each kept once. # Run: SOCIALCRAWL_API_KEY=... python3 poll_account.py (Python 3.9+, nothing to install) import json import os import time import urllib.error import urllib.parse import urllib.request KEY = os.environ["SOCIALCRAWL_API_KEY"] BASE = "https://www.socialcrawl.dev/v1" PATH = "tiktok/profile/videos" HANDLE = "washingtonpost" FIRST_RUN_SINCE = "2026-11-20" # only used when you hold no post yet STATE_FILE = "poll_state.json" MAX_PAGES = 10 # a poll should stop long before this def get(url): for attempt in range(6): req = urllib.request.Request(url, headers={"x-api-key": KEY}) try: with urllib.request.urlopen(req, timeout=120) as res: return json.load(res) except urllib.error.HTTPError as err: if err.code not in (429, 503) or attempt == 5: raise time.sleep(int(err.headers.get("Retry-After", "2"))) # same url again state = {"newest_id": None, "seen": []} if os.path.exists(STATE_FILE): with open(STATE_FILE) as f: state = json.load(f) seen = set(state["seen"]) # The same boundary goes on every page of the walk. boundary = {"stop_at_id": state["newest_id"]} if state["newest_id"] else {"since": FIRST_RUN_SINCE} rows, cursor, credits = [], None, 0 for _ in range(MAX_PAGES): query = {"handle": HANDLE, **boundary} if cursor: query["cursor"] = cursor page = get(f"{BASE}/{PATH}?" + urllib.parse.urlencode(query)) rows.extend(page["data"]["items"]) credits += page["credits_used"] cursor = page["pagination"]["next_cursor"] # null once the walk has stopped if not cursor: break # A pinned post sits out of date order: it is not a new post, and never the newest you hold. fresh = [r for r in rows if r["post"]["id"] not in seen and not r["post"]["flags"]["pinned"]] newest = next((r for r in rows if not r["post"]["flags"]["pinned"]), None) for r in fresh: print(r["post"]["published_at"], r["post"]["id"]) print(f"{len(rows)} rows read, {len(fresh)} new, {credits} credits") state["seen"] = sorted(seen | {r["post"]["id"] for r in rows}) if newest: state["newest_id"] = newest["post"]["id"] with open(STATE_FILE, "w") as f: json.dump(state, f) ``` **Agent brief** ```text title="Agent brief" Goal: a daily check of one TikTok account that reads only the videos posted since the last run. Read https://www.socialcrawl.dev/docs/recipes/poll-an-account-daily.md first. 1. Send GET /v1/tiktok/profile/videos with handle and dry_run=true. Show me the estimate. 2. First run: send since=2026-11-20. Later runs: send stop_at_id with the newest non-pinned post id I hold. 3. Send the same boundary on every page. Follow pagination.next_cursor until it is null. 4. Skip pinned rows. Save the newest non-pinned post id only after the whole run finishes. 5. Report only posts whose id I have not seen. Never raise the page cap without asking me. ``` ## Next steps - [Get a whole account](/docs/recipes/get-a-whole-account.md): Walk every video or comment an account has, with a cap and a safe retry. - [Monitors](/docs/engines/monitors.md): Let a scheduled Monitor do the polling for you. - [Pagination](/docs/pagination.md): The cursor rule, since, until and the stop reasons.