Web Scraping Run Agent API
Scrape Web Scraping Run Agent data with one API call. Runs a model-neutral browser agent task and returns a SocialCrawl async job id (job_...). Poll it via GET /v1/web/jobs/{job_id}.
Last updated August 2026Maintained by the SocialCrawl team
Starts a job where a browser agent follows your plain-language instruction on a site, returning a job id to poll for its answer.
Use it for tasks that need clicking and navigating to reach the data; for a page you can already name, scrape it.
Searching 50 platforms in parallel
TikTok·
Instagram·
YouTube·
Facebook·
X·
LinkedIn·
Reddit·
Threads·
Pinterest·
Twitch·
Truth Social·
Snapchat·
Kick·
Bluesky·
Kwai·
Rumble·
Spotify·Apple Music·TikTok Shop·
Amazon Shop·
Google Shopping·
Trustpilot·
TripAdvisor·
Linktree·
Komi·
Pillar·
lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Google Finance·
Polymarket·
Tavily·
Hacker News·
GitHub·
Perplexity·
Naver·Utility·
Universal SearchTikTok·Instagram·YouTube·Facebook·X·LinkedIn·Reddit·Threads·Pinterest·Twitch·Truth Social·Snapchat·Kick·Bluesky·Kwai·Rumble·Spotify·Apple Music·TikTok Shop·Amazon Shop·Google Shopping·Trustpilot·TripAdvisor·Linktree·Komi·Pillar·lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Google Finance·Polymarket·Tavily·Hacker News·GitHub·Perplexity·Naver·Utility·Universal SearchTikTok·Instagram·YouTube·Facebook·X·LinkedIn·Reddit·Threads·Pinterest·Twitch·Truth Social·Snapchat·Kick·Bluesky·Kwai·Rumble·Spotify·Apple Music·TikTok Shop·Amazon Shop·Google Shopping·Trustpilot·TripAdvisor·Linktree·Komi·Pillar·lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Google Finance·Polymarket·Tavily·Hacker News·GitHub·Perplexity·Naver·Utility·Universal SearchTikTok·Instagram·YouTube·Facebook·X·LinkedIn·Reddit·Threads·Pinterest·Twitch·Truth Social·Snapchat·Kick·Bluesky·Kwai·Rumble·Spotify·Apple Music·TikTok Shop·Amazon Shop·Google Shopping·Trustpilot·TripAdvisor·Linktree·Komi·Pillar·lnk.bio·Facebook Ads·Google Ads·LinkedIn Ads·Google Search·Google News·Google Finance·Polymarket·Tavily·Hacker News·GitHub·Perplexity·Naver·Utility·Universal SearchWhat can you do with the Run Agent API?
The Run Agent endpoint gives you structured Web Scraping data with computed fields in a single request. No scraping infrastructure to build or maintain.
Example Request
curl -X POST -H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","prompt":"Find the pricing page."}' \
"https://www.socialcrawl.dev/v1/web/agent"import requests
response = requests.post(
"https://www.socialcrawl.dev/v1/web/agent",
json={
'url': 'https://example.com',
'prompt': 'Find the pricing page.',
},
headers={"x-api-key": "YOUR_API_KEY"},
)
data = response.json()const response = await fetch(
"https://www.socialcrawl.dev/v1/web/agent",
{
method: "POST",
headers: {
"x-api-key": "YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
url: "https://example.com",
prompt: "Find the pricing page.",
}),
},
);
const data = await response.json();Parameters
| Parameter | Required | Description |
|---|---|---|
| url | Yes | Starting URL folded into the agent instruction. |
| prompt | Yes | Task instruction. |
| model | No | Agent model: spark-1-mini or spark-1-pro. (spark-1-mini | spark-1-pro) |
What does the Web Scraping Run Agent API return?
This example shows the endpoint's response shape and fields. Values are illustrative; this is not a live capture.
Example response
{
"success": true,
"platform": "web",
"endpoint": "/v1/web/agent",
"data": {
"job_id": "job_2q8w6r4v3",
"kind": "agent",
"resource": "agent",
"status": "queued",
"progress": null,
"invalid_urls": null,
"result_ref": null,
"result": null,
"error": null,
"credits_hold": 25,
"credits_charged": 0,
"created_at": "2026-07-09T09:30:00.000Z",
"updated_at": "2026-07-09T09:30:00.000Z",
"expires_at": "2026-07-16T09:30:00.000Z"
},
"credits_used": 25,
"credits_remaining": 975,
"request_id": "req-a1b2c3d4e5f6",
"cached": false
}Illustrative fixture for the documented response shape. Values are examples, not a production capture.
How does the Web Scraping Run Agent API work?
Send a GET request with your API key and get back clean, structured JSON in our unified schema. Supported computed fields are populated when the source provides the required inputs.
Method
POST
Response
JSON
How do you scrape social media data in seconds?
The fastest social media scraping API for developers. Scrape profiles, posts, comments, and analytics from 50 platforms covering 10B+ monthly active users.
One schema, every platform
Query 50 platforms with identical response structures. Write your integration once.
Computed fields, not just scraped
When an endpoint supports these metrics and the source provides the required inputs, the normalized record includes engagement_rate, estimated_reach, content_category, and language — ready to use.
See your data before you code
Visual Data Explorer — paste any URL, get rich result cards, sortable tables, CSV export.
import requests
response = requests.get(
'https://www.socialcrawl.dev/v1/tiktok/profile',
params={'handle': 'charlidamelio'},
headers={'x-api-key': 'sc_YOUR_API_KEY'}
)
data = response.json(){
"success": true,
"platform": "tiktok",
"data": {
"author": {
"username": "charlidamelio",
"followers": 152400000
},
"engagement": {
"likes": 12400000000,
"engagement_rate": 0.087
},
"metadata": {
"language": "en",
"content_category": "lifestyle"
}
}
}Ready to scrape Web Scraping Run Agent data?
Get your API key and start pulling Web Scraping data in under 60 seconds.