SocialCrawl

Use SocialCrawl with LlamaIndex

Give a LlamaIndex agent social media data, with the price shown before it spends.

LlamaIndex turns a Python function into an agent tool with FunctionTool, so two small functions give an agent SocialCrawl data. SocialCrawl publishes no LlamaIndex tool package, and the wrapper below needs none.

Setup

Install LlamaIndex and requests, then set your keys. Create the SocialCrawl key in the dashboard under API Keys; it looks like sc_your_key_here. The model key goes in OPENAI_API_KEY.

Terminal
pip install llama-index requests
export SOCIALCRAWL_API_KEY=sc_your_key_here
export OPENAI_API_KEY=your_openai_key

Connect

FunctionTool.from_defaults reads the type hints and the docstring, and FunctionAgent runs the tool loop.

Python
import os

import requests

BASE = "https://www.socialcrawl.dev"
HEADERS = {"x-api-key": os.environ["SOCIALCRAWL_API_KEY"]}


def estimate_cost(endpoint: str, params: dict[str, str | int]) -> dict:
    """Quote the credits a call will hold. Free. endpoint looks like tiktok/profile."""
    r = requests.get(
        f"{BASE}/v1/utility/estimate",
        headers=HEADERS,
        params={"id": endpoint, **params},
        timeout=30,
    )
    body = r.json()
    return body.get("data", body)


def call_endpoint(platform: str, resource: str, params: dict[str, str | int]) -> dict:
    """Run one SocialCrawl call, for example tiktok and profile with a handle."""
    r = requests.get(
        f"{BASE}/v1/{platform}/{resource}", headers=HEADERS, params=params, timeout=60
    )
    body = r.json()
    if body.get("success"):
        return body["data"]
    # Give the model the error type, not the whole envelope.
    err = body.get("error", {})
    return {"error": err.get("type"), "message": err.get("message"), "status": r.status_code}
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI

agent = FunctionAgent(
    tools=[FunctionTool.from_defaults(fn=estimate_cost), FunctionTool.from_defaults(fn=call_endpoint)],
    llm=OpenAI(model="gpt-4.1-mini"),
    system_prompt=(
        "Call estimate_cost and report the hold before any call_endpoint. "
        "If a call returns INSUFFICIENT_CREDITS, tell the user and stop."
    ),
)

Read a profile and its posts

Quote the call yourself, then run the agent. max_iterations is the step limit.

Python
import asyncio

quote = estimate_cost("tiktok/profile", {"handle": "nasa"})
print("hold:", quote["hold"], "valid:", quote["valid"])

answer = asyncio.run(
    agent.run("Read the TikTok profile for nasa and its latest videos, then summarise them.", max_iterations=5)
)
print(answer)

Keep spend in check

Call estimate_cost before any paid call and show the answer to the person who owns the key. max_iterations caps the loop, and max_credits stops a list walk where you set it. A 402 means the account is out of credits. Tell the user it is an offer to top up and stop, because a retry cannot succeed. See Errors and Credits.

Resources

On this page