SocialCrawl

Use SocialCrawl with CrewAI

Give a CrewAI agent social media data, with the price shown before it spends.

CrewAI agents use tools that subclass BaseTool, so two small classes over the REST API give a crew SocialCrawl data. SocialCrawl publishes no CrewAI package, and the wrapper below needs none.

Setup

Install CrewAI and requests, then set your keys. Create the SocialCrawl key in the dashboard under API Keys; it looks like sc_your_key_here. CrewAI uses the model named by your environment, so set the key for that provider, for example OPENAI_API_KEY.

Terminal
pip install crewai requests
export SOCIALCRAWL_API_KEY=sc_your_key_here
export OPENAI_API_KEY=your_openai_key

Connect

Each tool has a name, a description, an args_schema and a _run method. The call tool quotes the call itself and refuses it when the hold is above MAX_HOLD, so the cap is set once for every agent that shares the tool.

Python
import os

import requests

BASE = "https://www.socialcrawl.dev"
HEADERS = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}


def estimate_cost(endpoint: str, params: dict[str, str | int]) -> dict:
    """Quote the credits a call will hold. Free. endpoint looks like tiktok/profile."""
    r = requests.get(
        f"{BASE}/v1/utility/estimate",
        headers=HEADERS,
        params={"id": endpoint, **params},
        timeout=30,
    )
    body = r.json()
    return body.get("data", body)


def call_endpoint(platform: str, resource: str, params: dict[str, str | int]) -> dict:
    """Run one SocialCrawl call, for example tiktok and profile with a handle."""
    r = requests.get(
        f"{BASE}/v1/{platform}/{resource}", headers=HEADERS, params=params, timeout=60
    )
    body = r.json()
    if body.get("success"):
        return body["data"]
    # Give the model the error type, not the whole envelope.
    err = body.get("error", {})
    return {"error": err.get("type"), "message": err.get("message"), "status": r.status_code}
import json

from crewai import Agent, Crew, Task
from crewai.tools import BaseTool
from pydantic import BaseModel

MAX_HOLD = 25  # credits one call may hold


class EstimateArgs(BaseModel):
    endpoint: str
    params: dict[str, str | int]


class CallArgs(BaseModel):
    platform: str
    resource: str
    params: dict[str, str | int]


class EstimateCostTool(BaseTool):
    name: str = "estimate_cost"
    description: str = "Quote the credits a SocialCrawl call will hold. Free. Call it before call_endpoint and tell the user the answer."
    args_schema: type[BaseModel] = EstimateArgs

    def _run(self, endpoint: str, params: dict) -> str:
        return json.dumps(estimate_cost(endpoint, params))


class CallEndpointTool(BaseTool):
    name: str = "call_endpoint"
    description: str = "Run one SocialCrawl call, for example platform tiktok and resource profile. Paid, so quote it first. A 402 means out of credits: tell the user and stop."
    args_schema: type[BaseModel] = CallArgs

    def _run(self, platform: str, resource: str, params: dict) -> str:
        quote = estimate_cost(f"{platform}/{resource}", params)
        if not quote.get("valid") or quote.get("expected_max", 0) > MAX_HOLD:
            return json.dumps({"refused": "The hold is above the cap or the call is invalid.", "quote": quote})
        return json.dumps(call_endpoint(platform, resource, params))

Read a profile and its posts

Quote the call yourself, then give one researcher the tools and one task. max_iter is the step limit.

Python
quote = estimate_cost("tiktok/profile", {"handle": "nasa"})
print("hold:", quote["hold"], "valid:", quote["valid"])

researcher = Agent(
    role="Researcher",
    goal="Read public social media data with SocialCrawl and report what it says.",
    backstory="You always call estimate_cost and report the hold before call_endpoint.",
    tools=[EstimateCostTool(), CallEndpointTool()],
    max_iter=5,
)
research = Task(
    description="Read the TikTok profile for nasa and its latest videos (tiktok/profile and tiktok/profile/videos).",
    expected_output="Follower count and the topics of the latest videos.",
    agent=researcher,
)
print(Crew(agents=[researcher], tasks=[research]).kickoff())

Keep spend in check

Call estimate_cost before any paid call and show the answer to the person who owns the key. MAX_HOLD refuses an expensive call, max_iter caps the loop, and max_credits stops a list walk where you set it. A 402 means the account is out of credits. Tell the user it is an offer to top up and stop, because a retry cannot succeed. See Errors and Credits.

A researcher and a writer

Add a second agent with no tools. The writer reads the researcher's output through the task context, so only the researcher can spend credits.

Python
writer = Agent(
    role="Writer",
    goal="Turn research notes into a short brief.",
    backstory="You write plain, accurate summaries and never invent numbers.",
)
brief = Task(
    description="Write a four-sentence brief from the research.",
    expected_output="A four-sentence brief.",
    agent=writer,
    context=[research],
)
print(Crew(agents=[researcher, writer], tasks=[research, brief]).kickoff())

Resources

On this page