SocialCrawl

Use SocialCrawl with AutoGen

Give an AutoGen agent social media data, with the price shown before it spends.

AutoGen can attach an MCP server to an agent through McpWorkbench, so the hosted SocialCrawl server gives an agent its tools with no function to write. SocialCrawl publishes no AutoGen package, and the one install line below is AutoGen's own.

Setup

Install AutoGen with its MCP and OpenAI extras, then set your keys. Create the SocialCrawl key in the dashboard under API Keys; it looks like sc_your_key_here.

Terminal
pip install -U autogen-agentchat "autogen-ext[mcp,openai]"
export SOCIALCRAWL_API_KEY=sc_your_key_here
export OPENAI_API_KEY=your_openai_key

Connect

Point a workbench at the hosted server at https://mcp.socialcrawl.dev/mcp and send your key as a bearer header. The server brings a free estimate tool, socialcrawl_estimate, so the system message can ask for the price first. max_tool_iterations is the step limit.

Python
import asyncio
import os

from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_ext.tools.mcp import McpWorkbench, StreamableHttpServerParams

server = StreamableHttpServerParams(
    url="https://mcp.socialcrawl.dev/mcp",
    headers={"Authorization": f"Bearer {os.environ['SOCIALCRAWL_API_KEY']}"},
)

SYSTEM = (
    "Read public social media data with SocialCrawl. Call socialcrawl_estimate and report the cost "
    "before any paid call. If a call says the account is out of credits, tell the user and stop."
)


async def main() -> None:
    model_client = OpenAIChatCompletionClient(model="gpt-4.1-mini")
    async with McpWorkbench(server) as workbench:
        agent = AssistantAgent(
            "researcher",
            model_client=model_client,
            workbench=workbench,
            system_message=SYSTEM,
            reflect_on_tool_use=True,
            max_tool_iterations=5,
        )
        result = await agent.run(task="Read the TikTok profile for nasa and summarise it.")
        print(result.messages[-1].content)
    await model_client.close()


asyncio.run(main())

Read a profile and its posts

The run above asks the agent to quote before it reads, so the estimate appears in the tool calls that result.messages records. Widen the task to ask for the latest videos as well, for example Read the TikTok profile for nasa and its latest videos, then summarise both.

Keep spend in check

Call estimate_cost before any paid call and show the answer to the person who owns the key. max_tool_iterations caps the loop, and max_credits stops a list walk where you set it. A 402 means the account is out of credits. Tell the user it is an offer to top up and stop, because a retry cannot succeed. See Errors and Credits.

Researcher and writer

Put a second agent in a round-robin team. Only the researcher gets the workbench, so only the researcher can spend credits, and the writer ends the run by saying DONE.

Python
from autogen_agentchat.conditions import TextMentionTermination
from autogen_agentchat.teams import RoundRobinGroupChat


async def team_run() -> None:
    model_client = OpenAIChatCompletionClient(model="gpt-4.1-mini")
    async with McpWorkbench(server) as workbench:
        researcher = AssistantAgent(
            "researcher", model_client=model_client, workbench=workbench,
            system_message=SYSTEM, reflect_on_tool_use=True, max_tool_iterations=5,
        )
        writer = AssistantAgent(
            "writer", model_client=model_client,
            system_message="Write a four-sentence brief from the researcher's notes, then say DONE.",
        )
        team = RoundRobinGroupChat([researcher, writer], termination_condition=TextMentionTermination("DONE"))
        result = await team.run(task="Read the TikTok profile for nasa and brief me on it.")
        print(result.messages[-1].content)
    await model_client.close()


asyncio.run(team_run())

Resources

On this page