Watch something over time
Which watch fits your job, how its results reach you, pushed to a webhook or read from a feed, and how to verify each delivery
SocialCrawl can repeat a call on a schedule and tell you what it found. Pick the watch by what you want to know, then pick how its results reach you: pushed to your HTTPS endpoint, or read from a feed when you are ready.
Which watch fits your job?
| You want to know | Use | Results reach you through | Cost |
|---|---|---|---|
| What is new about a topic, keyword or brand across platforms, every hour or slower | A Discover schedule: POST /v1/prism/discover with schedule | Event webhooks or the events feed | Each run bills as the request does, capped by max_credits_per_run. Schedule routes are free |
| The new posts or comments of an account or post you already know | A Monitor: POST /v1/monitors with track and webhook_rows: "new" | Monitor webhooks | The recipe's own cost plus 1 credit per run |
| Followers, views or likes of accounts you name, as a time series | A numbers-only Monitor: POST /v1/monitors with track | Monitor webhooks, or no webhook and read /export | The recipe's own cost plus 1 credit per run |
| Whether a web page or a web search result changed | A web monitor: POST /v1/web/monitors | Its check history, and a webhook_url call (see below) | The scrape or search plus 1 credit per check |
| That your balance is running low, a pack was paid for, or auto-recharge ran | Billing alerts, set up on the dashboard | Billing webhooks and a low-credit email | Free |
| What is new since your last check, once, on one endpoint | The endpoint's own since or since_id parameter | The call's answer | The call's own price |
A Monitor runs any endpoint (its recipe) on an hourly, daily, weekly or cron cadence. A Discover schedule re-runs a whole multi-platform search and delivers only rows it has not delivered before. Both are explained, with every field, on their own pages.
How do the results reach you?
| Channel | Push or pull | Carries | Signature | Retries | De-duplicate on |
|---|---|---|---|---|---|
| Monitor webhooks | Push to the Monitor's webhook_url | One body per completed run: the result, fired alerts, deltas | x-socialcrawl-signature | Up to 6 attempts. 10 failed deliveries in a row pause it | run_id |
| Event webhooks | Push to one endpoint registered with POST /v1/webhook-endpoints | Signed batches of account events: discover.* and endpoint.disabled | Standard Webhooks: webhook-id, webhook-timestamp, webhook-signature | 7 retries over about 41 hours. A 410 disables it at once, 5 days of failures disable it | Each inner event id |
| Events feed | Pull with GET /v1/events, or long-poll with wait_s | The same account events, oldest first, from a cursor | None: an authenticated read with your key | Nothing to retry. Events are kept 30 days | The event id |
| Billing webhooks | Push to the URL saved on the dashboard | credits.low, credits.exhausted, payment.succeeded, auto_recharge.* | x-socialcrawl-signature | Up to 6 attempts. 10 failed deliveries in a row pause it | event and created |
Every route on the events feed and on event webhooks is free. Monitor runs do not appear in the events feed and are not sent to event webhooks: a Monitor delivers only to its own webhook_url. Discover runs are the other way round: there is no webhook field on Discover, and one event webhook registration covers every run and schedule on your API key.
Read the feed now. It costs nothing and shows what your account's runs and schedules have produced:
curl -G "https://www.socialcrawl.dev/v1/events" \
-H "x-api-key: YOUR_API_KEY" \
--data-urlencode "types=discover.*" \
--data-urlencode "limit=5"Which verifier do I need?
There are two signature schemes, so one receiver may need two verifiers:
x-socialcrawl-signature, on Monitor webhooks, billing webhooks and the batch job completion webhook. The header readst=<unix seconds>,v1=<hex>, and the hex is the HMAC-SHA256 of<t>.<raw body>keyed with your secret. One routine verifies all three. Node and Python verifiers: Verifying signatures.webhook-signature, on event webhooks, withwebhook-idandwebhook-timestamp. This is the Standard Webhooks scheme, so its published libraries verify it. Verifiers and fixed test vectors: How do I verify a delivery?.
Verify against the raw request body before you parse it, answer 2xx quickly, and do the work afterwards. Every push channel delivers at least once, so the same delivery can arrive twice: keep the id named in the table above and skip one you have already processed.
Web jobs and web monitors
POST /v1/web/crawl, POST /v1/web/batch-scrape and POST /v1/web/monitors take a webhook_url, which is called when a job settles or a check runs. That call carries neither signature above, so treat it as a nudge, not as data: read the result with your key from GET /v1/web/jobs/{job_id} or GET /v1/web/monitors/{monitor_id}/checks before you act on it. See Web.
Next steps
Monitor webhooks
Run any endpoint on a schedule and receive each run, signed.
Event webhooks
One endpoint for every Discover run and schedule, with test sends.
Events feed
Read the same events in order with a cursor or a named consumer.
Discover schedules
Re-run a multi-platform search and get only the new rows.