# Event webhooks (/docs/webhook-endpoints) Register one HTTPS endpoint and receive signed batches of account events, such as a discover run settling, with retries and a test send Register one HTTPS endpoint with `POST /v1/webhook-endpoints` and SocialCrawl sends it every event your API key's work produces: a discover run that completes, pauses or finds new rows on a schedule. Each delivery is a signed JSON batch of events, retried for up to 41 hours until your endpoint answers `2xx`. Every webhook endpoint route is free. ```bash title="cURL" curl -X POST 'https://www.socialcrawl.dev/v1/webhook-endpoints' \ -H 'x-api-key: YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://example.com/hooks/socialcrawl", "types": ["discover.*"], "description": "discover alerts" }' ``` ```json title="Response" { "success": true, "data": { "endpoint": { "id": "we_V1StGXR8Z5jdHi6BmyT", "url": "https://example.com/hooks/socialcrawl", "description": "discover alerts", "types": ["discover.*"], "subject_ids": null, "status": "enabled", "disabled_reason": null, "failing_since": null, "api_version": "2026-10-01", "secret_rotation": null, "created_at": "2026-10-19T10:00:00.000Z", "updated_at": "2026-10-19T10:00:00.000Z", "next": { "self_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT", "test_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/test", "attempts_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/attempts", "rotate_secret_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/rotate-secret", "events_url": "/v1/events" } }, "secret": "whsec_...", "next_step": "Store data.secret now: it is shown once and never again. ..." }, "credits_used": 0, "request_id": "req_..." } ``` `data.secret` is the signing secret. It is shown in this answer only; SocialCrawl stores it encrypted and never returns it again. Leave `types` out to receive every event type, now and as new products start sending events. The endpoint belongs to the API key that created it: it receives the events of that key's runs and schedules and the events about the whole account, and only that key can read, change or delete it. ## Send a test ```bash title="cURL" curl -X POST 'https://www.socialcrawl.dev/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/test' \ -H 'x-api-key: YOUR_API_KEY' \ -H 'Content-Type: application/json' \ -d '{ "type": "discover.completed" }' ``` A test sends one signed sample of the type you name (default: the first type the endpoint subscribes to) and answers with what your receiver did: ```json title="Response" { "success": true, "data": { "type": "discover.completed", "attempt": { "id": "wat_...", "delivery_id": "msg_...", "test": true, "status": 200, "outcome": "delivered", "duration_ms": 142, "response_excerpt": "ok", "error": null, "created_at": "2026-10-19T10:00:01.000Z" }, "next_step": "The receiver answered 2xx. Real events are delivered the same way, in batches, signed per attempt." }, "credits_used": 0, "request_id": "req_..." } ``` The sample event carries `"test": true` beside its `type` and `"cursor": null`. A test is sent once, never retried, never enters the event log and never changes the endpoint's health, so a test answered `410` does not disable it. ## What a delivery looks like Every delivery is an `event.batch`, also when it holds one event. A batch holds at most 100 events or 256 KB. ```json title="JSON" { "id": "bat_V1StGXR8Z5jdHi6BmyT", "type": "event.batch", "data": { "events": [ { "id": "evt_k3m7x2q7v4b6n5c5z6w2a2s4d3", "type": "discover.completed", "api_version": "2026-10-01", "created_at": "2026-10-19T10:14:52.118Z", "cursor": "c_...", "subject": { "kind": "discover_run", "id": "run_8Hq2LmNp4Rt6" }, "data": { "run_id": "run_8Hq2LmNp4Rt6", "status": "completed", "stopped_because": null, "counts": { "legs": { "planned": 12, "done": 11, "empty": 1, "failed": 0, "skipped": 0, "pending": 0 }, "rows_kept": 84 }, "credits": { "spent": 42 }, "version": 7, "schedule_id": null }, "links": { "run": "/v1/prism/discover/runs/run_8Hq2LmNp4Rt6", "results": "/v1/prism/discover/runs/run_8Hq2LmNp4Rt6/results.jsonl", "changes": "/v1/prism/discover/runs/run_8Hq2LmNp4Rt6?changes_since=0" } } ] } } ``` Discover events carry counts and ids only, never your keywords, context or row text. `counts.legs` counts the run's searches by state and `counts.rows_kept` the rows it kept. Fetch the rows from the URLs in `links`. Each event keeps its own `id`. Deliveries are at least once and unordered, so deduplicate on each inner event `id`. Every event also carries `cursor`, its position in your account's [event feed](/docs/events.md): to fill a gap after an outage, read `GET /v1/events?cursor=`. ## Event types | Type | When | `data` | | ---- | ---- | ------ | | `discover.completed`, `discover.partial`, `discover.paused` | A background or scheduled discover run settles in that status | `run_id`, `status`, `stopped_because`, `counts`, `credits`, `version`, `schedule_id` | | `discover.failed` | A background or scheduled discover run settles `failed`; a run you cancel sends nothing | `run_id`, `status`, `stopped_because`, `counts`, `credits`, `version`, `schedule_id` | | `discover.progress` | A background run passes each tenth of its planned searches | `run_id`, `status`, `progress` (`percent`, `legs_finished`, `legs_planned`), `counts`, `credits`, `version` | | `discover.accounts_ready` | A `view=accounts` run has enriched and judged its accounts | `run_id`, `accounts` (`kept`, `enriched`, `deferred`), `credits`, `version` | | `discover.new_rows`, `discover.new_accounts`, `discover.new_items` | A scheduled run settles with new results for its view | `schedule_id`, `run_id`, `count`, `ids` (the first 100), `more_ids`, `version` | | `discover.run_skipped` | A schedule occurrence is skipped | `schedule_id`, `reason`, `previous_run_id`, `scheduled_for` | | `discover.run_failed` | A scheduled run fails, or three skips in a row pause the schedule | `schedule_id`, `run_id`, `reason` | | `endpoint.disabled` | One of your webhook endpoints was disabled (below) | `endpoint_id`, `reason`, `last_error` | `types` takes exact types, a namespace wildcard such as `discover.*`, or `*`. A run that answered inside the original `GET` call sends no event: you already have its result. ## How do I verify a delivery? Every attempt carries three headers, in the [Standard Webhooks](https://www.standardwebhooks.com) format: ```text title="Header" webhook-id: msg_V1StGXR8Z5jdHi6BmyT webhook-timestamp: 1792231200 webhook-signature: v1,n4OBPwOJOeyYgw9sOfwuvt4WnxtW6CX30IHU5WLqf5Y= ``` The signature is the base64 HMAC-SHA256 of `{webhook-id}.{webhook-timestamp}.{raw body}`, keyed with the base64 decoding of your secret after its `whsec_` prefix. `webhook-id` is the same on every retry of a batch; `webhook-timestamp` is fresh on each attempt. Verify over the raw body before you parse it, and reject a timestamp more than 300 seconds from your clock. ```js title="JavaScript" import { createHmac, timingSafeEqual } from "node:crypto"; // True if the Standard Webhooks signature over the raw body is valid and fresh. export function verify(secret, headers, rawBody, nowSeconds = Date.now() / 1000) { const id = headers["webhook-id"], ts = headers["webhook-timestamp"]; if (Math.abs(nowSeconds - Number(ts)) > 300) return false; const key = Buffer.from(secret.replace(/^whsec_/, ""), "base64"); const expected = createHmac("sha256", key).update(`${id}.${ts}.`).update(rawBody).digest(); return headers["webhook-signature"].split(" ").some((entry) => { const [version, sig] = entry.split(","); const got = Buffer.from(sig ?? "", "base64"); return version === "v1" && got.length === expected.length && timingSafeEqual(got, expected); }); } ``` ```python title="Python" import base64, hashlib, hmac, time def verify(secret: str, headers: dict, body: bytes, now: float | None = None) -> bool: """True if the Standard Webhooks signature over the raw body is valid and fresh.""" msg_id, ts, sigs = headers["webhook-id"], headers["webhook-timestamp"], headers["webhook-signature"] if abs((now or time.time()) - int(ts)) > 300: return False key = base64.b64decode(secret.removeprefix("whsec_")) signed = f"{msg_id}.{ts}.".encode() + body expected = base64.b64encode(hmac.new(key, signed, hashlib.sha256).digest()).decode() for entry in sigs.split(" "): version, _, sig = entry.partition(",") if version == "v1" and hmac.compare_digest(sig, expected): return True return False ``` `webhook-signature` can hold several space-separated `v1,` entries: during a secret rotation each delivery is signed with the old secret and the new one, and a match on any entry is valid. ### Test vectors Check your verifier against these before you go live. Both use the made-up secret `whsec_bGlzdGVuaW5nLXRlc3QtdmVjdG9yLWtleS0wMDAwMDE=` and `webhook-timestamp: 1792231200` (17/10/2026 10:00:00 UTC), so pass that time as "now". The body is the exact bytes shown, with no trailing newline. | Vector | `webhook-id` | Body | Expected `webhook-signature` | | ------ | ------------ | ---- | ---------------------------- | | 1, an empty batch | `msg_test_0001` | `{"id":"bat_test_0001","type":"event.batch","data":{"events":[]}}` | `v1,n4OBPwOJOeyYgw9sOfwuvt4WnxtW6CX30IHU5WLqf5Y=` | | 2, a one-event test send | `msg_test_0002` | the line below, 625 bytes | `v1,iZQsG+1XQ/FlbzM4E2LmgzArJz+9XMXyMVfFZbw8q80=` | ```json title="Vector 2 body" {"id":"bat_test_0002","type":"event.batch","data":{"events":[{"id":"evt_test_0001","type":"mention.created","test":true,"api_version":"2026-10-01","created_at":"2026-10-17T10:00:00Z","cursor":"c_test_0001","subject":{"kind":"stream","id":"lst_test","name":"Client B: Globex Mobile"},"data":{"platform":"twitter","mention":{"mention_id":"men_test_0001","evidence":{"id":"tw_000000000000","platform":"twitter","url":"https://x.com/example/status/1","published_at":"2026-10-17","text":"Globex Mobile is down again","author":"example"},"matched_rules":[{"id":"rul_a1","tag":"outage"}],"first_seen_at":"2026-10-17T09:59:30Z"}}}]}} ``` Both verifiers above accept vectors 1 and 2, and reject vector 2 with one byte added to the body and vector 1 checked 301 seconds after its timestamp. ## What your receiver answers, and what SocialCrawl does | Your receiver answers | SocialCrawl | | --------------------- | ----------- | | `2xx` within 15 seconds | The delivery is done | | `401` for a bad signature or a stale timestamp | A failed attempt, retried, so a clock or secret mix-up can be fixed without losing events | | `410` | The endpoint is disabled at once (`endpoint.disabled`, reason `gone`). Answer it only to stop deliveries for good | | `429`, `502`, `503` or `504` with `Retry-After` | The next attempt waits for the later of the next retry and `Retry-After`, at most 24 hours | | Any other status, no answer within 15 seconds, or a redirect | A failed attempt, retried. Redirects are never followed | | Failed attempts for 5 days in a row, with no delivery in between | The endpoint is disabled (`endpoint.disabled`, reason `failing`). A day with no attempt at all ends the run, so one failure after a quiet week starts a new count instead | Retries follow this ladder after the first attempt: 30 seconds, 2 minutes, 10 minutes, 1 hour, 4 hours, 12 hours and 24 hours. After the eighth attempt the batch is dropped; its events stay readable from `GET /v1/events` for 30 days. Each retry sends the identical body under the same `webhook-id`. A receiver checklist: verify over the raw body before parsing, reject a timestamp more than 300 seconds from now, deduplicate on each inner event `id`, answer `2xx` within 15 seconds and do slow work afterwards. ## Managing endpoints | Call | What it does | | ---- | ------------ | | `GET /v1/webhook-endpoints` | The key's endpoints, newest first. `limit` 1 to 100 (default 50), `cursor`, `status` | | `GET /v1/webhook-endpoints/{id}` | One endpoint. The secret is never in it | | `PATCH /v1/webhook-endpoints/{id}` | Change `url` (checked again, and the account owner is emailed), `types`, `description`, `subject_ids` or `status` (`enabled` or `disabled`) | | `DELETE /v1/webhook-endpoints/{id}` | Delete the endpoint and its delivery history | | `POST /v1/webhook-endpoints/{id}/rotate-secret` | A new secret, shown once; the old one keeps signing for 24 hours | | `POST /v1/webhook-endpoints/{id}/test` | A signed sample, sent once | | `GET /v1/webhook-endpoints/{id}/attempts` | Delivery attempts, newest first: status, outcome, duration and the first 1 KB of your receiver's answer | `subject_ids` narrows an endpoint to events about the listed runs or schedules, for example `["dsch_..."]` for one discover schedule. A key holds at most 20 endpoints. The account owner gets a security email whenever an endpoint is added, and whenever a `PATCH` changes an endpoint's URL, naming the old and the new host, the API key and the time. A leaked key cannot quietly point your events somewhere else. An endpoint is disabled with a `disabled_reason`: `gone` (it answered `410`), `failing` (failed attempts for 5 days in a row), `key_revoked` (the API key that created it was revoked) or `user` (you sent `status: "disabled"`). To re-enable it, fix the receiver and send `PATCH` with `{ "status": "enabled" }`: batches held while it was disabled are sent again at once. Events created while it was disabled are not delivered; read them from `GET /v1/events`. ### Rotating the secret ```bash title="cURL" curl -X POST 'https://www.socialcrawl.dev/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/rotate-secret' \ -H 'x-api-key: YOUR_API_KEY' ``` The answer carries the new `data.secret` (shown once) and `secret_rotation.previous_secret_expires_at`. Until then every delivery carries two signatures, one per secret, so deploy the new secret to your receiver within 24 hours and nothing is rejected in between. ## Which URLs are accepted? The URL must be HTTPS on port 443, with no user name or password in it, at most 2,048 characters, on a host whose every DNS address is public. Private, loopback and link-local addresses are refused, and so are SocialCrawl's own hosts. A refused URL is a free `400 INVALID_REQUEST` with `details.reason: "invalid_webhook_url"`, a `why` and a `next_step`: | `why` | Meaning | | ----- | ------- | | `not_https`, `port_not_443`, `credentials_in_url`, `invalid_url`, `url_too_long` | The URL itself | | `private_host`, `private_dns`, `dns_failed` | Where the host points | | `own_host` | A SocialCrawl host or preview deployment | | `chat_incoming_webhook` | A chat service's incoming-webhook URL (below) | The host is checked again on every attempt, so a name that later points at a private address is refused then too. ## Send events to a chat channel Chat incoming-webhook URLs (Slack, Microsoft Teams, Discord, Google Chat) are refused: they cannot check a signature and expect their own message format. Run this short relay instead and register its URL. It verifies each delivery with the `verify` function above (saved as `verify.mjs`), posts one line per event, and answers `503` with `Retry-After` when the chat service fails, so SocialCrawl retries the batch. ```js title="JavaScript" import { createServer } from "node:http"; import { verify } from "./verify.mjs"; const { SOCIALCRAWL_WEBHOOK_SECRET: SECRET, CHAT_WEBHOOK_URL: CHAT, PORT = 8080 } = process.env; createServer(async (req, res) => { const chunks = []; for await (const chunk of req) chunks.push(chunk); const raw = Buffer.concat(chunks); if (!verify(SECRET, req.headers, raw)) return res.writeHead(401).end(); const lines = JSON.parse(raw).data.events.map(({ type, subject, data, links }) => `${type} ${subject?.id ?? ""}: ${data.status ?? data.reason ?? `${data.count ?? 0} new`} ${links?.run ?? ""}`.trim()); if (lines.length > 0) { const r = await fetch(CHAT, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ text: lines.join("\n") }) }); if (!r.ok) return res.writeHead(503, { "retry-after": "60" }).end(); } res.writeHead(204).end(); }).listen(PORT); ``` It does not deduplicate retried batches. If each event must be posted once, keep the inner event ids you have posted and skip those. ## Next steps - [Event feed](/docs/events.md): Read the same events by polling, with a cursor or a named consumer. - [Discover](/docs/prism/discover.md): Background and scheduled runs that send these events. - [Errors](/docs/errors.md): Every error type and whether to retry. - [Monitor webhooks](/docs/webhooks.md): The per-Monitor webhook, signed with its own header.