Event webhooks

Register one HTTPS endpoint with POST /v1/webhook-endpoints and SocialCrawl sends it every event your API key's work produces: a discover run that completes, pauses or finds new rows on a schedule. Each delivery is a signed JSON batch of events, retried for up to 41 hours until your endpoint answers 2xx. Every webhook endpoint route is free.

curl -X POST 'https://www.socialcrawl.dev/v1/webhook-endpoints' \
  -H 'x-api-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com/hooks/socialcrawl",
    "types": ["discover.*"],
    "description": "discover alerts"
  }'

{
  "success": true,
  "data": {
    "endpoint": {
      "id": "we_V1StGXR8Z5jdHi6BmyT",
      "url": "https://example.com/hooks/socialcrawl",
      "description": "discover alerts",
      "types": ["discover.*"],
      "subject_ids": null,
      "status": "enabled",
      "disabled_reason": null,
      "failing_since": null,
      "api_version": "2026-10-01",
      "secret_rotation": null,
      "created_at": "2026-10-19T10:00:00.000Z",
      "updated_at": "2026-10-19T10:00:00.000Z",
      "next": {
        "self_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT",
        "test_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/test",
        "attempts_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/attempts",
        "rotate_secret_url": "/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/rotate-secret",
        "events_url": "/v1/events"
      }
    },
    "secret": "whsec_...",
    "next_step": "Store data.secret now: it is shown once and never again. ..."
  },
  "credits_used": 0,
  "request_id": "req_..."
}

data.secret is the signing secret. It is shown in this answer only; SocialCrawl stores it encrypted and never returns it again. Leave types out to receive every event type, now and as new products start sending events. The endpoint belongs to the API key that created it: it receives the events of that key's runs and schedules and the events about the whole account, and only that key can read, change or delete it.

Send a test

curl -X POST 'https://www.socialcrawl.dev/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/test' \
  -H 'x-api-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{ "type": "discover.completed" }'

A test sends one signed sample of the type you name (default: the first type the endpoint subscribes to) and answers with what your receiver did:

{
  "success": true,
  "data": {
    "type": "discover.completed",
    "attempt": {
      "id": "wat_...",
      "delivery_id": "msg_...",
      "test": true,
      "status": 200,
      "outcome": "delivered",
      "duration_ms": 142,
      "response_excerpt": "ok",
      "error": null,
      "created_at": "2026-10-19T10:00:01.000Z"
    },
    "next_step": "The receiver answered 2xx. Real events are delivered the same way, in batches, signed per attempt."
  },
  "credits_used": 0,
  "request_id": "req_..."
}

The sample event carries "test": true beside its type and "cursor": null. A test is sent once, never retried, never enters the event log and never changes the endpoint's health, so a test answered 410 does not disable it.

What a delivery looks like

Every delivery is an event.batch, also when it holds one event. A batch holds at most 100 events or 256 KB.

{
  "id": "bat_V1StGXR8Z5jdHi6BmyT",
  "type": "event.batch",
  "data": {
    "events": [
      {
        "id": "evt_k3m7x2q7v4b6n5c5z6w2a2s4d3",
        "type": "discover.completed",
        "api_version": "2026-10-01",
        "created_at": "2026-10-19T10:14:52.118Z",
        "cursor": "c_...",
        "subject": { "kind": "discover_run", "id": "run_8Hq2LmNp4Rt6" },
        "data": {
          "run_id": "run_8Hq2LmNp4Rt6",
          "status": "completed",
          "stopped_because": null,
          "counts": {
            "legs": { "planned": 12, "done": 11, "empty": 1, "failed": 0, "skipped": 0, "pending": 0 },
            "rows_kept": 84
          },
          "credits": { "spent": 42 },
          "version": 7,
          "schedule_id": null
        },
        "links": {
          "run": "/v1/prism/discover/runs/run_8Hq2LmNp4Rt6",
          "results": "/v1/prism/discover/runs/run_8Hq2LmNp4Rt6/results.jsonl",
          "changes": "/v1/prism/discover/runs/run_8Hq2LmNp4Rt6?changes_since=0"
        }
      }
    ]
  }
}

Discover events carry counts and ids only, never your keywords, context or row text. counts.legs counts the run's searches by state and counts.rows_kept the rows it kept. Fetch the rows from the URLs in links.

Each event keeps its own id. Deliveries are at least once and unordered, so deduplicate on each inner event id. Every event also carries cursor, its position in your account's event feed: to fill a gap after an outage, read GET /v1/events?cursor=<id of the last event you processed>.

Event types

types takes exact types, a namespace wildcard such as discover.*, or *. A run that answered inside the original GET call sends no event: you already have its result.

How do I verify a delivery?

Every attempt carries three headers, in the Standard Webhooks format:

webhook-id: msg_V1StGXR8Z5jdHi6BmyT
webhook-timestamp: 1792231200
webhook-signature: v1,n4OBPwOJOeyYgw9sOfwuvt4WnxtW6CX30IHU5WLqf5Y=

The signature is the base64 HMAC-SHA256 of {webhook-id}.{webhook-timestamp}.{raw body}, keyed with the base64 decoding of your secret after its whsec_ prefix. webhook-id is the same on every retry of a batch; webhook-timestamp is fresh on each attempt. Verify over the raw body before you parse it, and reject a timestamp more than 300 seconds from your clock.

import { createHmac, timingSafeEqual } from "node:crypto";

// True if the Standard Webhooks signature over the raw body is valid and fresh.
export function verify(secret, headers, rawBody, nowSeconds = Date.now() / 1000) {
  const id = headers["webhook-id"], ts = headers["webhook-timestamp"];
  if (Math.abs(nowSeconds - Number(ts)) > 300) return false;
  const key = Buffer.from(secret.replace(/^whsec_/, ""), "base64");
  const expected = createHmac("sha256", key).update(`${id}.${ts}.`).update(rawBody).digest();
  return headers["webhook-signature"].split(" ").some((entry) => {
    const [version, sig] = entry.split(",");
    const got = Buffer.from(sig ?? "", "base64");
    return version === "v1" && got.length === expected.length && timingSafeEqual(got, expected);
  });
}

import base64, hashlib, hmac, time

def verify(secret: str, headers: dict, body: bytes, now: float | None = None) -> bool:
    """True if the Standard Webhooks signature over the raw body is valid and fresh."""
    msg_id, ts, sigs = headers["webhook-id"], headers["webhook-timestamp"], headers["webhook-signature"]
    if abs((now or time.time()) - int(ts)) > 300:
        return False
    key = base64.b64decode(secret.removeprefix("whsec_"))
    signed = f"{msg_id}.{ts}.".encode() + body
    expected = base64.b64encode(hmac.new(key, signed, hashlib.sha256).digest()).decode()
    for entry in sigs.split(" "):
        version, _, sig = entry.partition(",")
        if version == "v1" and hmac.compare_digest(sig, expected):
            return True
    return False

webhook-signature can hold several space-separated v1, entries: during a secret rotation each delivery is signed with the old secret and the new one, and a match on any entry is valid.

Test vectors

Check your verifier against these before you go live. Both use the made-up secret whsec_bGlzdGVuaW5nLXRlc3QtdmVjdG9yLWtleS0wMDAwMDE= and webhook-timestamp: 1792231200 (17/10/2026 10:00:00 UTC), so pass that time as "now". The body is the exact bytes shown, with no trailing newline.

{"id":"bat_test_0002","type":"event.batch","data":{"events":[{"id":"evt_test_0001","type":"mention.created","test":true,"api_version":"2026-10-01","created_at":"2026-10-17T10:00:00Z","cursor":"c_test_0001","subject":{"kind":"stream","id":"lst_test","name":"Client B: Globex Mobile"},"data":{"platform":"twitter","mention":{"mention_id":"men_test_0001","evidence":{"id":"tw_000000000000","platform":"twitter","url":"https://x.com/example/status/1","published_at":"2026-10-17","text":"Globex Mobile is down again","author":"example"},"matched_rules":[{"id":"rul_a1","tag":"outage"}],"first_seen_at":"2026-10-17T09:59:30Z"}}}]}}

Both verifiers above accept vectors 1 and 2, and reject vector 2 with one byte added to the body and vector 1 checked 301 seconds after its timestamp.

What your receiver answers, and what SocialCrawl does

Retries follow this ladder after the first attempt: 30 seconds, 2 minutes, 10 minutes, 1 hour, 4 hours, 12 hours and 24 hours. After the eighth attempt the batch is dropped; its events stay readable from GET /v1/events for 30 days. Each retry sends the identical body under the same webhook-id.

A receiver checklist: verify over the raw body before parsing, reject a timestamp more than 300 seconds from now, deduplicate on each inner event id, answer 2xx within 15 seconds and do slow work afterwards.

Managing endpoints

subject_ids narrows an endpoint to events about the listed runs or schedules, for example ["dsch_..."] for one discover schedule. A key holds at most 20 endpoints.

The account owner gets a security email whenever an endpoint is added, and whenever a PATCH changes an endpoint's URL, naming the old and the new host, the API key and the time. A leaked key cannot quietly point your events somewhere else.

An endpoint is disabled with a disabled_reason: gone (it answered 410), failing (failed attempts for 5 days in a row), key_revoked (the API key that created it was revoked) or user (you sent status: "disabled"). To re-enable it, fix the receiver and send PATCH with { "status": "enabled" }: batches held while it was disabled are sent again at once. Events created while it was disabled are not delivered; read them from GET /v1/events.

Rotating the secret

curl -X POST 'https://www.socialcrawl.dev/v1/webhook-endpoints/we_V1StGXR8Z5jdHi6BmyT/rotate-secret' \
  -H 'x-api-key: YOUR_API_KEY'

The answer carries the new data.secret (shown once) and secret_rotation.previous_secret_expires_at. Until then every delivery carries two signatures, one per secret, so deploy the new secret to your receiver within 24 hours and nothing is rejected in between.

Which URLs are accepted?

The URL must be HTTPS on port 443, with no user name or password in it, at most 2,048 characters, on a host whose every DNS address is public. Private, loopback and link-local addresses are refused, and so are SocialCrawl's own hosts. A refused URL is a free 400 INVALID_REQUEST with details.reason: "invalid_webhook_url", a why and a next_step:

The host is checked again on every attempt, so a name that later points at a private address is refused then too.

Send events to a chat channel

Chat incoming-webhook URLs (Slack, Microsoft Teams, Discord, Google Chat) are refused: they cannot check a signature and expect their own message format. Run this short relay instead and register its URL. It verifies each delivery with the verify function above (saved as verify.mjs), posts one line per event, and answers 503 with Retry-After when the chat service fails, so SocialCrawl retries the batch.

import { createServer } from "node:http";
import { verify } from "./verify.mjs";
const { SOCIALCRAWL_WEBHOOK_SECRET: SECRET, CHAT_WEBHOOK_URL: CHAT, PORT = 8080 } = process.env;
createServer(async (req, res) => {
  const chunks = [];
  for await (const chunk of req) chunks.push(chunk);
  const raw = Buffer.concat(chunks);
  if (!verify(SECRET, req.headers, raw)) return res.writeHead(401).end();
  const lines = JSON.parse(raw).data.events.map(({ type, subject, data, links }) =>
    `${type} ${subject?.id ?? ""}: ${data.status ?? data.reason ?? `${data.count ?? 0} new`} ${links?.run ?? ""}`.trim());
  if (lines.length > 0) {
    const r = await fetch(CHAT, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ text: lines.join("\n") }) });
    if (!r.ok) return res.writeHead(503, { "retry-after": "60" }).end();
  }
  res.writeHead(204).end();
}).listen(PORT);

It does not deduplicate retried batches. If each event must be posted once, keep the inner event ids you have posted and skip those.

Next steps