Monitor Webhooks

A Monitor runs a recipe on a schedule. Each time a run completes, SocialCrawl POSTs a signed JSON body to the webhook URL you registered, carrying the run result, any alerts that fired, and the deltas against the previous run. Add track and a Monitor keeps only the numbers you pick, per row, and can deliver them as spreadsheet rows or skip the webhook entirely (see Numbers-only Monitors).

Prerequisites

• An API key from socialcrawl.dev, under Dashboard → API Keys.
• An HTTPS endpoint on a public host that can read a raw request body. Loopback, private, and link-local addresses are rejected at registration.
• Enough credits to cover the recipe: each run bills the recipe's own cost plus 1 credit for scheduling.

How do I receive Monitor webhooks?

curl -X POST 'https://www.socialcrawl.dev/v1/monitors' \
  -H 'x-api-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "recipe": "search/everywhere",
    "params": { "query": "acme corp" },
    "cadence": "daily",
    "webhook_url": "https://example.com/hooks/socialcrawl",
    "alert_rules": [{ "metric": "coverage", "op": "lt", "value": 0.6 }],
    "suppress_webhook_unless_alert": false
  }'

{
  "monitor": {
    "id": "mon_7Qk2R9xLpV",
    "name": "search/everywhere monitor",
    "recipe": "search/everywhere",
    "cadence": "daily",
    "status": "active",
    "alert_rules": [{ "metric": "coverage", "op": "lt", "value": 0.6 }],
    "estimated_cost_per_run": 21,
    "estimated_monthly_cost": 630,
    "next_run_at": "2026-07-05T09:00:00.000Z",
    "warnings": []
  },
  "webhook_secret": "whsec_..."
}

The rest of the family: GET /v1/monitors lists them, GET /v1/monitors/{monitor_id} reads one, GET /v1/monitors/{monitor_id}/runs is the run history, GET /v1/monitors/{monitor_id}/timeseries projects stored runs into a metric series, GET /v1/monitors/{monitor_id}/export downloads the stored numbers as CSV or NDJSON, PATCH /v1/monitors/{monitor_id} with {"status": "paused"} or {"status": "active"} pauses and resumes, and DELETE /v1/monitors/{monitor_id} unschedules it.

Event catalogue

A Monitor webhook fires once per completed run. The run's own status decides whether a delivery happens at all.

One more filter sits on top: with suppress_webhook_unless_alert: true, deliveries are limited to runs where at least one alert rule fired. A quiet run sends nothing.

Payload

Every delivery is a JSON object with this shape.

{
  "monitor_id": "mon_7Qk2R9xLpV",
  "run_id": "run_9Fh1Ab3Cd7",
  "recipe": "search/everywhere",
  "status": "ok",
  "scheduled_for": "2026-07-04T09:00:00.000Z",
  "alerts_fired": [
    {
      "metric": "coverage",
      "op": "lt",
      "from": null,
      "to": 0.53,
      "delta": null,
      "pct_change": null
    }
  ],
  "result": {
    "...": "the recipe body for this run"
  },
  "deltas": {
    "coverage": -0.18
  }
}

A numbers-only Monitor sends a different body. See Numbers-only Monitors.

Fired alert objects

Each entry in alerts_fired describes one rule that matched.

{
  "metric": "coverage",
  "op": "pct_change_lt",
  "from": 0.71,
  "to": 0.53,
  "delta": -0.18,
  "pct_change": -25.35
}

Metric paths

metric and every deltas key is a dot-path into result (the recipe body), not into the envelope. For recipe: "search/everywhere" the top-level numeric leaf is coverage, so you write coverage, never result.coverage or data.coverage. For a single-platform endpoint, result is the supplier's body, so the paths follow that supplier's field names rather than the unified shape you get from a /v1 call. To watch unified paths such as author.followers or items[].post.engagement.views, create the Monitor with track.

Paths are discovered by walking the result for finite numeric leaves, up to 4 levels deep; arrays are never metric paths (with track they are, through items[].). A rule whose path does not resolve to a number is skipped with a warning. It never fires, and deltas stays empty for it.

Numbers-only Monitors

Pass track to keep only the numbers you care about. Each run reads them off the unified response (the same shape a /v1 call returns, whichever supplier answered), stores them per row with the change since the previous run, and stores nothing else.

curl -X POST 'https://www.socialcrawl.dev/v1/monitors' \
  -H 'x-api-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "recipe": "tiktok/profile/videos",
    "params": { "handle": "khaby.lame" },
    "cadence": "daily",
    "track": { "metrics": ["items[].post.engagement.views", "items[].post.engagement.likes"] }
  }'

Paths are checked when you create the Monitor, for free: each one has to land on a number in that recipe's unified response. items[].engagement.views on a post list returns 400 with the corrected path, items[].post.engagement.views, and a typo such as totl returns 400 with Did you mean 'total'?. A path to a text or true/false field returns 400 naming its type. On a recipe with no fixed response shape to check against, such as a Prism composite, the paths are accepted and listed under warnings in the response, so check the numbers of the first run. Billing is the same as any Monitor: the recipe's cost plus 1 credit per run, refunded in full when the run fails. The active-monitor limit is the same one above: Free 3, Starter 10, Growth 25, Pro 100, Enterprise 500, from the largest credit pack the account has bought. Paused Monitors do not count. A Monitor created without track is unchanged: webhook_url stays required, and each run still stores the recipe body.

With track:

• webhook_url is optional. Without it the Monitor is download-only: read it with /export or /timeseries, and the create response returns webhook_secret: null.
• Every alert_rules metric must be one of track.metrics. An items[]. rule is checked once per row against that row's previous number, and each fired alert carries row_id.
• Each run's legs holds one entry saying which source in the endpoint's chain answered: rung is primary or fallback, and rung_index is its position (0 is the primary). A fallback source can leave a number empty that the primary fills.
• A run on which none of the tracked metrics has a value is not charged. It is marked failed, its skip_reason says why, its credits are refunded in full, and it is never the run the next one is compared against, so an alert cannot fire from it.
• A run whose page comes back with no results is not charged either, the same as a call that returns an empty page. Its total of 0 would describe the empty page rather than the resource, so nothing is recorded: the run is marked failed with a skip_reason, refunded in full, and no alert fires from it.
• GET /v1/monitors/{monitor_id}/runs?include=numbers returns each run's numbers and deltas. result is always null. The numbers are what is stored. With webhook_format: "full" the page is in that delivery and is not kept. A rows delivery sends only the numbers. A Monitor with no webhook delivers nothing; the numbers are on /export and /timeseries.

Full payload

With the default webhook_format: "full" the delivery keeps the keys shown in Payload. result is the unified response for the run, and two keys change meaning:

{
  "numbers": {
    "7412233": { "items[].post.engagement.views": 184000 },
    "7419981": { "items[].post.engagement.views": 9200 }
  },
  "deltas": {
    "rows": {
      "7412233": {
        "items[].post.engagement.views": { "prev": 171500, "cur": 184000, "abs": 12500, "pct": 7.29 }
      }
    },
    "rows_new": ["7419981"],
    "rows_gone": []
  }
}

deltas is null on the first run. pct is null when the previous value was 0.

Rows payload for a spreadsheet

webhook_format: "rows" sends one flat entry per row and metric, signed the same way:

{
  "monitor_id": "mon_7Qk2R9xLpV",
  "run_id": "run_9Fh1Ab3Cd7",
  "recipe": "tiktok/profile/videos",
  "status": "ok",
  "scheduled_for": "2026-09-23T09:00:00.000Z",
  "alerts_fired": [],
  "rows": [
    {
      "t": "2026-09-23T09:00:00.000Z",
      "row_id": "7412233",
      "metric": "items[].post.engagement.views",
      "value": 184000,
      "prev": 171500,
      "abs_change": 12500,
      "pct_change": 7.29
    }
  ]
}

To append them to a Google Sheet, open Extensions, then Apps Script, paste this, and deploy it as a web app that anyone can access. Apps Script cannot read request headers, so it cannot check x-socialcrawl-signature. Put a long random token in the web app URL you register (.../exec?token=...) and compare it instead.

const TOKEN = "a-long-random-string";

function doPost(e) {
  if (e.parameter.token !== TOKEN) return ContentService.createTextOutput("no");
  const body = JSON.parse(e.postData.contents);
  const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Monitor");
  const values = body.rows.map((r) => [
    r.t, body.run_id, r.row_id, r.metric, r.value, r.prev, r.abs_change, r.pct_change,
  ]);
  if (values.length > 0) {
    sheet.getRange(sheet.getLastRow() + 1, 1, values.length, values[0].length).setValues(values);
  }
  return ContentService.createTextOutput("ok");
}

Google answers the POST with a redirect once the script has run. After the first run, check webhook_delivery on GET /v1/monitors/{monitor_id}/runs to see the status your deployment returned. A retried delivery repeats its run_id, so filter duplicates on that column.

Export

GET /v1/monitors/{monitor_id}/export downloads what the Monitor stored, one line per run, row, and metric, oldest first. It is free.

curl 'https://www.socialcrawl.dev/v1/monitors/mon_7Qk2R9xLpV/export?format=csv&from=2026-09-01T00:00:00Z' \
  -H 'x-api-key: YOUR_API_KEY'

t,row_id,metric,value,abs_change,pct_change
2026-09-23T09:00:00.000Z,7412233,items[].post.engagement.views,184000,12500,7.29

format=ndjson returns one JSON object per line with the same keys. A Monitor without track exports the numeric fields /timeseries finds in its stored results, under row id _. Both /export and /timeseries read the newest 1,000 runs in the window; on a numbers-only Monitor /timeseries also returns series, one { row_id, metric, points } per row and metric, and accepts row= to narrow it.

Verifying signatures

Every delivery carries an x-socialcrawl-signature header, Stripe-style:

x-socialcrawl-signature: t=1700000000,v1=<hex>

The signature is HMAC-SHA256(secret, "<t>.<rawBody>"), where <t> is the Unix timestamp in seconds and <rawBody> is the exact bytes of the request body. Fold t into your check to enforce a replay window.

Your signing secret is the whsec_... value returned once in the create response. Always verify against the raw request body: parsing and re-serializing the JSON changes the bytes and breaks the signature.

import crypto from "node:crypto";

const TOLERANCE_SECONDS = 300;

function verify(rawBody, header, secret) {
  const parts = Object.fromEntries(
    header.split(",").map((kv) => {
      const i = kv.indexOf("=");
      return [kv.slice(0, i), kv.slice(i + 1)];
    }),
  );
  const t = Number(parts.t);
  if (!Number.isFinite(t)) return false;
  // Replay window: reject anything older than the tolerance.
  if (Math.abs(Math.floor(Date.now() / 1000) - t) > TOLERANCE_SECONDS) {
    return false;
  }

  const expected = crypto
    .createHmac("sha256", secret)
    .update(`${t}.${rawBody}`)
    .digest("hex");
  const a = Buffer.from(parts.v1 ?? "", "hex");
  const b = Buffer.from(expected, "hex");
  // timingSafeEqual throws a RangeError on a length mismatch, and a truncated
  // or non-hex v1 produces a short buffer, so compare lengths first.
  if (a.length !== b.length) return false;
  return crypto.timingSafeEqual(a, b);
}

import hashlib, hmac, time

TOLERANCE_SECONDS = 300

def verify(raw_body: bytes, header: str, secret: str) -> bool:
    parts = dict(kv.split("=", 1) for kv in header.split(","))
    try:
        t = int(parts["t"])
    except (KeyError, ValueError):
        return False
    # Replay window: reject anything older than the tolerance.
    if abs(int(time.time()) - t) > TOLERANCE_SECONDS:
        return False

    expected = hmac.new(
        secret.encode(),
        f"{t}.".encode() + raw_body,
        hashlib.sha256,
    ).hexdigest()
    return hmac.compare_digest(parts.get("v1", ""), expected)

Delivery, retries, and auto-pause

Deliveries go out through a retrying queue: up to 6 attempts (the initial delivery plus 5 retries) with exponential backoff. Respond 2xx promptly to acknowledge; a non-2xx response or a timeout counts as a failed attempt.

A successful delivery resets the failure counter. After 10 consecutive failed deliveries the webhook is automatically paused and stops receiving events. Re-save the webhook URL on your dashboard to reactivate it.

URL requirements, enforced when you register a webhook:

• Must be HTTPS.
• Must be a public host. Loopback, private, and link-local addresses are rejected.
• Must be 2048 characters or fewer.

Idempotency

Handlers must be idempotent. Retries and occasional duplicate deliveries are normal, and the same run can arrive more than once if your endpoint acknowledged late.

De-duplicate on run_id: it is unique per run, stable across retries of that run, and already in the payload. Record it before you act, and treat a second delivery carrying a run_id you have seen as a no-op.

Testing locally

There is no test-fire endpoint, and a loopback URL is rejected at registration, so the loop is:

1. Expose your handler on a public HTTPS hostname (a tunnel, or a hosted request inspector).
2. Create a throwaway Monitor pointed at it with cadence: "hourly" and a cheap recipe such as tiktok/profile, and pass your own webhook_secret so a test fixture can hard-code a known value.
3. Capture one real delivery, then replay its raw body and header against your verify() function offline. That is the fastest way to debug a signature mismatch, because it removes the network from the loop.
4. Check GET /v1/monitors/{monitor_id}/runs. Each run row carries webhook_delivery with attempts, lastStatus, deliveredAt, and failedAt, which tells you what your endpoint actually returned. Add ?include=result to see the stored result too.
5. PATCH the Monitor to {"status": "paused"}, or DELETE it, when you are done, so it stops billing.

Troubleshooting

Next steps