Web page schema
The unified SocialCrawl web page schema. Every field, its type, per-platform availability, and a machine-readable JSON Schema, generated from the canonical Zod source.
Web page schema
Every SocialCrawl endpoint that returns a web page gives you this exact shape, whatever the source platform. Write your parser once and the same code reads web page data from every platform below. That is the unified schema: one contract instead of a dozen raw upstream formats.
Field reference
| Field | Type | Nullable | Description |
|---|---|---|---|
url | string | Yes | Requested URL. |
final_url | string | Yes | Final resolved URL after redirects. |
status_code | number | Yes | HTTP status code observed while fetching the page. |
scrape_id | string | Yes | Opaque scrape identifier for follow-up interactions. |
fetched_at | string | Yes | Fetch timestamp when reported. |
content.markdown | string | Yes | Markdown body. |
content.html | string | Yes | HTML body. |
content.raw_html | string | Yes | Raw HTML body. |
content.summary | string | Yes | Generated summary. |
media.screenshot_url | string | Yes | Screenshot URL when requested. |
media.audio_url | string | Yes | Audio URL. |
media.video_url | string | Yes | Video URL. |
extraction | object | Yes | Structured extraction result when requested. |
fetch.cache_state | string | Yes | Upstream cache state when reported. |
fetch.cached_at | string | Yes | Upstream cache timestamp when reported. |
fetch.proxy_tier | string | Yes | Proxy tier used for the fetch. |
Platform availability
1 platform return the Web page shape. yes means the platform populates the field (the value may still be null); a blank means the platform never provides it, so it is always null.
| Platform | url | final_url | status_code | scrape_id | fetched_at | content.markdown | content.html | content.raw_html | content.summary | media.screenshot_url | media.audio_url | media.video_url |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Web Scraping | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes | yes |
Machine-readable schema
Validate responses programmatically against the JSON Schema (2020-12):
Point a validator (Ajv, jsonschema, or your framework's) at that URL, or hand it to an agent so it can check the shape without a live call.
Returned by
Endpoints with the WebPage / WebPageList archetype return this shape:
See how the same fields map to each platform's raw upstream names in the cross-platform field equivalence table.
News article schema
The unified SocialCrawl news article schema. Every field, its type, per-platform availability, and a machine-readable JSON Schema, generated from the canonical Zod source.
Cross-platform field equivalence
How every platform's raw metric maps to one unified SocialCrawl field. TikTok play_count, YouTube viewCount, and Twitter views.count all become engagement.views.