Web page schema
The unified SocialCrawl web page schema. Every field, its type, per-platform availability, and a machine-readable JSON Schema, generated from the canonical Zod source.
Every SocialCrawl endpoint that returns a web page gives you this exact shape, whatever the source platform. Write your parser once and the same code reads web page data from every platform below. That is the unified schema: one contract instead of a dozen raw upstream formats.
Field reference
| Field | Type | Nullable | Description |
|---|---|---|---|
url | string | Yes | Requested URL. |
final_url | string | Yes | Final resolved URL after redirects. |
status_code | number | Yes | The target site's own HTTP status for the page, such as 200, 404 or 430. It is not the status of your SocialCrawl call: a web/scrape that answers success: true can carry status_code: 404, which means the site answered with its error page. Such a page is returned with a _warnings line and is not charged. |
scrape_id | string | Yes | Opaque scrape identifier for follow-up interactions. |
fetched_at | string | Yes | Fetch timestamp when reported. |
published_at | string | Yes | When the page was published: an ISO 8601 instant, or a YYYY-MM-DD date when only the day is printed. From the page's own article:published_time meta tag on a scrape, and from the news date or that tag on a search row. Null when the page states none. |
content.markdown | string | Yes | Markdown body. |
content.html | string | Yes | HTML body. |
content.raw_html | string | Yes | Raw HTML body. |
content.summary | string | Yes | Generated summary. |
media.screenshot_url | string | Yes | Screenshot URL when requested. |
media.audio_url | string | Yes | Audio URL. |
media.video_url | string | Yes | Video URL. |
title | string | Yes | The page title, or null when the page has none. Sent on scraped pages, on search and map results and on a parsed document. |
description | string | Yes | The page description or search snippet, or null when there is none. |
source_type | string | Yes | Which kind of result a search row is: web, news or image. Null when the result is not a search row. |
extraction | object | Yes | Structured extraction result when requested. The schema you send decides its keys, so it has no fixed shape. On web/extract every value is checked against the page text: a placeholder such as 'Not specified', or a value the page does not state, comes back as null and is listed in extraction_check.removed. |
answer | any | No | The source's answer to a question about the page, when you asked for the answer format. Null otherwise. Its shape follows the format, so it has no single fixed shape. |
highlights | any | No | The passages the source picked out of the page, when you asked for the highlights format. Null otherwise. Its shape follows the format, so it has no single fixed shape. |
change_tracking | any | No | What changed on the page since the last scrape of it, when you asked for the changeTracking format. Null otherwise. Its shape follows the format, so it has no single fixed shape. |
page_count | number | Yes | How many pages of a document were read, or null when the source does not report it. |
total_page_count | number | Yes | How many pages the document has in all, or null when the source does not report it. |
page_state.value | string | No | What the page is: content, login_or_signup_wall, consent_wall, challenge (a CAPTCHA or bot check), error_or_not_found, rate_limited, paywall_teaser, empty_shell or other. |
page_state.confidence | number | No | How sure the call is, from 0 to 1. |
page_state.skipped_judge | boolean | No | True when code decided the state and no judgement was needed: a long page with many headings (content), a known bot-check page (challenge), a blank page, a 'needs JavaScript' notice or a page of links only (empty_shell). |
native.endpoint | string | No | The endpoint that read the page, such as /v1/linkedin/profile. |
native.credits_used | number | No | The credits that endpoint charged. |
native.data | any | No | That endpoint's own data object, exactly as a direct call to it returns it. |
extraction_check.checked | integer | No | Non-null values examined. |
extraction_check.found_in_text | integer | No | Values kept because they appear word for word on the page. |
extraction_check.confirmed | integer | No | Values kept because a check found the page states them, for example in another language. |
extraction_check.removed | object[] | No | Fields set to null, with the value the extractor gave. |
extraction_check.unverified | string[] | No | Fields returned as extracted without a check. |
fetch.cache_state | string | Yes | The page scraper's own cache: hit when it served a copy it fetched earlier (see max_age), miss when it fetched the page now. It is not SocialCrawl's response cache, which is the envelope's cached flag. A hit here is still a charged call when cached is false. |
fetch.cached_at | string | Yes | When the page scraper fetched the copy it served, when cache_state is hit. |
fetch.proxy_tier | string | Yes | The proxy the fetch used: basic (a plain request) or enhanced (the stronger proxy, used with proxy=enhanced, or by proxy=auto when the site blocked the plain request). Null when the page was read by a source that does not report a proxy. |
Platform availability
1 platform returns the Web page shape. answer, highlights, change_tracking, page_state.value, page_state.confidence, page_state.skipped_judge, native.endpoint, native.credits_used, native.data, extraction_check.checked, extraction_check.found_in_text, extraction_check.confirmed, extraction_check.removed, extraction_check.unverified are never null on any platform. The fields below vary: yes means the platform populates it (the value may still be null); a blank means the platform never provides it, so it is always null.
| Platform | url | final_url | status_code | scrape_id | fetched_at | published_at | content.markdown | content.html | content.raw_html | content.summary | media.screenshot_url | media.audio_url |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Web Scraping | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
Machine-readable schema
Validate responses programmatically against the JSON Schema (2020-12):
Point a validator (Ajv, jsonschema, or your framework's) at that URL, or hand it to an agent so it can check the shape without a live call.
Returned by
Endpoints with the WebPage / WebPageList archetype return this shape:
See how the same fields map to each platform's raw upstream names in the cross-platform field equivalence table.
News article schema
The unified SocialCrawl news article schema. Every field, its type, per-platform availability, and a machine-readable JSON Schema, generated from the canonical Zod source.
Transcript schema
The unified SocialCrawl video or audio transcript schema. Every field, its type, per-platform availability, and a machine-readable JSON Schema, generated from the canonical Zod source.
