SocialCrawl

Use SocialCrawl with OpenAI

Give an OpenAI agent social media data, with the price shown before it spends.

OpenAI function calling gives your agent two tools over the SocialCrawl REST API, one that prices a call and one that runs it. SocialCrawl publishes no OpenAI package, so the wrapper on this page is the whole integration.

Setup

Install the OpenAI client and requests, then put your key in SOCIALCRAWL_API_KEY. Create the key in the dashboard under API Keys; it looks like sc_your_key_here. Your OpenAI key goes in OPENAI_API_KEY.

Terminal
pip install openai requests
export SOCIALCRAWL_API_KEY=sc_your_key_here

Connect

Declare each tool as a function tool whose parameters is a JSON schema. The files under /agent-tools/ use Anthropic's field name, input_schema, so rename it to parameters before you pass one to OpenAI. The loop below runs whatever the model asks for and sends back a function_call_output.

Python
import os

import requests

BASE = "https://www.socialcrawl.dev"
HEADERS = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}


def estimate_cost(endpoint: str, params: dict) -> dict:
    """Quote the credits a call will hold. Free. endpoint looks like tiktok/profile."""
    r = requests.get(
        f"{BASE}/v1/utility/estimate",
        headers=HEADERS,
        params={"id": endpoint, **params},
        timeout=30,
    )
    body = r.json()
    return body.get("data", body)


def call_endpoint(platform: str, resource: str, params: dict) -> dict:
    """Run one SocialCrawl call, for example tiktok and profile with a handle."""
    r = requests.get(
        f"{BASE}/v1/{platform}/{resource}", headers=HEADERS, params=params, timeout=60
    )
    body = r.json()
    if body.get("success"):
        return body["data"]
    # Give the model the error type, not the whole envelope.
    err = body.get("error", {})
    return {"error": err.get("type"), "message": err.get("message"), "status": r.status_code}
import json

from openai import OpenAI

client = OpenAI()
MODEL = os.environ.get("OPENAI_MODEL", "gpt-5.5")

TOOLS = [
    {
        "type": "function",
        "name": "estimate_cost",
        "description": "Quote the credits a SocialCrawl call will hold. Free. Call it before call_endpoint and show the user the answer.",
        "parameters": {"type": "object",
            "properties": {
                "endpoint": {"type": "string", "description": "platform/resource, for example tiktok/profile"},
                "params": {"type": "object", "description": "The endpoint's query parameters, for example {\"handle\": \"nasa\"}"},
            },
            "required": ["endpoint", "params"]},
    },
    {
        "type": "function",
        "name": "call_endpoint",
        "description": "Run one SocialCrawl call to read public social media data. Paid, so quote it first. A 402 means out of credits: tell the user and stop.",
        "parameters": {"type": "object",
            "properties": {
                "platform": {"type": "string", "description": "For example tiktok"},
                "resource": {"type": "string", "description": "For example profile or profile/videos"},
                "params": {"type": "object", "description": "The endpoint's query parameters"},
            },
            "required": ["platform", "resource", "params"]},
    },
]
FUNCS = {"estimate_cost": estimate_cost, "call_endpoint": call_endpoint}


def run(prompt: str, max_steps: int = 5) -> str:
    previous, next_input = None, prompt
    for _ in range(max_steps):
        resp = client.responses.create(
            model=MODEL, input=next_input, tools=TOOLS, previous_response_id=previous
        )
        previous = resp.id
        calls = [item for item in resp.output if item.type == "function_call"]
        if not calls:
            return resp.output_text
        next_input = [
            {
                "type": "function_call_output",
                "call_id": c.call_id,
                "output": json.dumps(FUNCS[c.name](**json.loads(c.arguments))),
            }
            for c in calls
        ]
    return "Stopped at the step limit."

The Responses API can also attach a hosted MCP server through its mcp tool type. Skills and MCP lists the server URL and where the key goes.

Read a profile and its posts

Price the call yourself first, then hand the same ask to the model. estimate_cost costs nothing, and its hold is the most the call can take.

Python
quote = estimate_cost("tiktok/profile", {"handle": "nasa"})
print("hold:", quote["hold"], "valid:", quote["valid"])

print(run("Read the TikTok profile for nasa, then its latest videos, and summarise both."))

Keep spend in check

Call estimate_cost before any paid call and show the answer to the person who owns the key. Keep max_steps small, and send max_credits on list endpoints so a walk stops where you set it. A 402 means the account is out of credits. Tell the user it is an offer to top up and stop, because a retry cannot succeed. See Errors and Credits.

Structured output

To get a typed answer, pass a Pydantic model as text_format and read output_parsed. Give the model the rows as input so no second paid call is needed.

Python
from pydantic import BaseModel


class ProfileSummary(BaseModel):
    handle: str
    followers: int | None
    one_line_summary: str


profile = call_endpoint("tiktok", "profile", {"handle": "nasa"})
parsed = client.responses.parse(
    model=MODEL,
    input=f"Summarise this profile: {json.dumps(profile)}",
    text_format=ProfileSummary,
)
print(parsed.output_parsed)

Resources

On this page