Use SocialCrawl with CrewAI
Give a CrewAI agent social media data, with the price shown before it spends.
CrewAI agents use tools that subclass BaseTool, so two small classes over the REST API give a crew SocialCrawl data. SocialCrawl publishes no CrewAI package, and the wrapper below needs none.
Setup
Install CrewAI and requests, then set your keys. Create the SocialCrawl key in the dashboard under API Keys; it looks like sc_your_key_here. CrewAI uses the model named by your environment, so set the key for that provider, for example OPENAI_API_KEY.
pip install crewai requests
export SOCIALCRAWL_API_KEY=sc_your_key_here
export OPENAI_API_KEY=your_openai_keyConnect
Each tool has a name, a description, an args_schema and a _run method. The call tool quotes the call itself and refuses it when the hold is above MAX_HOLD, so the cap is set once for every agent that shares the tool.
import os
import requests
BASE = "https://www.socialcrawl.dev"
HEADERS = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}
def estimate_cost(endpoint: str, params: dict[str, str | int]) -> dict:
"""Quote the credits a call will hold. Free. endpoint looks like tiktok/profile."""
r = requests.get(
f"{BASE}/v1/utility/estimate",
headers=HEADERS,
params={"id": endpoint, **params},
timeout=30,
)
body = r.json()
return body.get("data", body)
def call_endpoint(platform: str, resource: str, params: dict[str, str | int]) -> dict:
"""Run one SocialCrawl call, for example tiktok and profile with a handle."""
r = requests.get(
f"{BASE}/v1/{platform}/{resource}", headers=HEADERS, params=params, timeout=60
)
body = r.json()
if body.get("success"):
return body["data"]
# Give the model the error type, not the whole envelope.
err = body.get("error", {})
return {"error": err.get("type"), "message": err.get("message"), "status": r.status_code}
import json
from crewai import Agent, Crew, Task
from crewai.tools import BaseTool
from pydantic import BaseModel
MAX_HOLD = 25 # credits one call may hold
class EstimateArgs(BaseModel):
endpoint: str
params: dict[str, str | int]
class CallArgs(BaseModel):
platform: str
resource: str
params: dict[str, str | int]
class EstimateCostTool(BaseTool):
name: str = "estimate_cost"
description: str = "Quote the credits a SocialCrawl call will hold. Free. Call it before call_endpoint and tell the user the answer."
args_schema: type[BaseModel] = EstimateArgs
def _run(self, endpoint: str, params: dict) -> str:
return json.dumps(estimate_cost(endpoint, params))
class CallEndpointTool(BaseTool):
name: str = "call_endpoint"
description: str = "Run one SocialCrawl call, for example platform tiktok and resource profile. Paid, so quote it first. A 402 means out of credits: tell the user and stop."
args_schema: type[BaseModel] = CallArgs
def _run(self, platform: str, resource: str, params: dict) -> str:
quote = estimate_cost(f"{platform}/{resource}", params)
if not quote.get("valid") or quote.get("expected_max", 0) > MAX_HOLD:
return json.dumps({"refused": "The hold is above the cap or the call is invalid.", "quote": quote})
return json.dumps(call_endpoint(platform, resource, params))Read a profile and its posts
Quote the call yourself, then give one researcher the tools and one task. max_iter is the step limit.
quote = estimate_cost("tiktok/profile", {"handle": "nasa"})
print("hold:", quote["hold"], "valid:", quote["valid"])
researcher = Agent(
role="Researcher",
goal="Read public social media data with SocialCrawl and report what it says.",
backstory="You always call estimate_cost and report the hold before call_endpoint.",
tools=[EstimateCostTool(), CallEndpointTool()],
max_iter=5,
)
research = Task(
description="Read the TikTok profile for nasa and its latest videos (tiktok/profile and tiktok/profile/videos).",
expected_output="Follower count and the topics of the latest videos.",
agent=researcher,
)
print(Crew(agents=[researcher], tasks=[research]).kickoff())Keep spend in check
Call estimate_cost before any paid call and show the answer to the person who owns the key. MAX_HOLD refuses an expensive call, max_iter caps the loop, and max_credits stops a list walk where you set it. A 402 means the account is out of credits. Tell the user it is an offer to top up and stop, because a retry cannot succeed. See Errors and Credits.
A researcher and a writer
Add a second agent with no tools. The writer reads the researcher's output through the task context, so only the researcher can spend credits.
writer = Agent(
role="Writer",
goal="Turn research notes into a short brief.",
backstory="You write plain, accurate summaries and never invent numbers.",
)
brief = Task(
description="Write a four-sentence brief from the research.",
expected_output="A four-sentence brief.",
agent=writer,
context=[research],
)
print(Crew(agents=[researcher, writer], tasks=[research, brief]).kickoff())