SocialCrawl
Unified schema

Web page schema

The unified SocialCrawl web page schema. Every field, its type, per-platform availability, and a machine-readable JSON Schema, generated from the canonical Zod source.

Every SocialCrawl endpoint that returns a web page gives you this exact shape, whatever the source platform. Write your parser once and the same code reads web page data from every platform below. That is the unified schema: one contract instead of a dozen raw upstream formats.

Field reference

FieldTypeNullableDescription
urlstringYesRequested URL.
final_urlstringYesFinal resolved URL after redirects.
status_codenumberYesThe target site's own HTTP status for the page, such as 200, 404 or 430. It is not the status of your SocialCrawl call: a web/scrape that answers success: true can carry status_code: 404, which means the site answered with its error page. Such a page is returned with a _warnings line and is not charged.
scrape_idstringYesOpaque scrape identifier for follow-up interactions.
fetched_atstringYesFetch timestamp when reported.
published_atstringYesWhen the page was published: an ISO 8601 instant, or a YYYY-MM-DD date when only the day is printed. From the page's own article:published_time meta tag on a scrape, and from the news date or that tag on a search row. Null when the page states none.
content.markdownstringYesMarkdown body.
content.htmlstringYesHTML body.
content.raw_htmlstringYesRaw HTML body.
content.summarystringYesGenerated summary.
media.screenshot_urlstringYesScreenshot URL when requested.
media.audio_urlstringYesAudio URL.
media.video_urlstringYesVideo URL.
titlestringYesThe page title, or null when the page has none. Sent on scraped pages, on search and map results and on a parsed document.
descriptionstringYesThe page description or search snippet, or null when there is none.
source_typestringYesWhich kind of result a search row is: web, news or image. Null when the result is not a search row.
extractionobjectYesStructured extraction result when requested. The schema you send decides its keys, so it has no fixed shape. On web/extract every value is checked against the page text: a placeholder such as 'Not specified', or a value the page does not state, comes back as null and is listed in extraction_check.removed.
answeranyNoThe source's answer to a question about the page, when you asked for the answer format. Null otherwise. Its shape follows the format, so it has no single fixed shape.
highlightsanyNoThe passages the source picked out of the page, when you asked for the highlights format. Null otherwise. Its shape follows the format, so it has no single fixed shape.
change_trackinganyNoWhat changed on the page since the last scrape of it, when you asked for the changeTracking format. Null otherwise. Its shape follows the format, so it has no single fixed shape.
page_countnumberYesHow many pages of a document were read, or null when the source does not report it.
total_page_countnumberYesHow many pages the document has in all, or null when the source does not report it.
page_state.valuestringNoWhat the page is: content, login_or_signup_wall, consent_wall, challenge (a CAPTCHA or bot check), error_or_not_found, rate_limited, paywall_teaser, empty_shell or other.
page_state.confidencenumberNoHow sure the call is, from 0 to 1.
page_state.skipped_judgebooleanNoTrue when code decided the state and no judgement was needed: a long page with many headings (content), a known bot-check page (challenge), a blank page, a 'needs JavaScript' notice or a page of links only (empty_shell).
native.endpointstringNoThe endpoint that read the page, such as /v1/linkedin/profile.
native.credits_usednumberNoThe credits that endpoint charged.
native.dataanyNoThat endpoint's own data object, exactly as a direct call to it returns it.
extraction_check.checkedintegerNoNon-null values examined.
extraction_check.found_in_textintegerNoValues kept because they appear word for word on the page.
extraction_check.confirmedintegerNoValues kept because a check found the page states them, for example in another language.
extraction_check.removedobject[]NoFields set to null, with the value the extractor gave.
extraction_check.unverifiedstring[]NoFields returned as extracted without a check.
fetch.cache_statestringYesThe page scraper's own cache: hit when it served a copy it fetched earlier (see max_age), miss when it fetched the page now. It is not SocialCrawl's response cache, which is the envelope's cached flag. A hit here is still a charged call when cached is false.
fetch.cached_atstringYesWhen the page scraper fetched the copy it served, when cache_state is hit.
fetch.proxy_tierstringYesThe proxy the fetch used: basic (a plain request) or enhanced (the stronger proxy, used with proxy=enhanced, or by proxy=auto when the site blocked the plain request). Null when the page was read by a source that does not report a proxy.

Platform availability

1 platform returns the Web page shape. answer, highlights, change_tracking, page_state.value, page_state.confidence, page_state.skipped_judge, native.endpoint, native.credits_used, native.data, extraction_check.checked, extraction_check.found_in_text, extraction_check.confirmed, extraction_check.removed, extraction_check.unverified are never null on any platform. The fields below vary: yes means the platform populates it (the value may still be null); a blank means the platform never provides it, so it is always null.

Platformurlfinal_urlstatus_codescrape_idfetched_atpublished_atcontent.markdowncontent.htmlcontent.raw_htmlcontent.summarymedia.screenshot_urlmedia.audio_url
Web Scrapingyesyesyesyesyesyesyesyesyesyesyesyes

Machine-readable schema

Validate responses programmatically against the JSON Schema (2020-12):

/schemas/web-page.json

Point a validator (Ajv, jsonschema, or your framework's) at that URL, or hand it to an agent so it can check the shape without a live call.

Returned by

Endpoints with the WebPage / WebPageList archetype return this shape:

Web Scraping

See how the same fields map to each platform's raw upstream names in the cross-platform field equivalence table.

On this page