SocialCrawl

Sentiment analysis

Build a social sentiment pipeline — universal search finds the conversation, comment endpoints pull what real people said, and your LLM labels it. 23 credits per run.

Sentiment analysis

Build a "how do people actually feel about X" pipeline: find the conversation across the social web, harvest the comments (where the real sentiment lives), and hand an LLM a clean, labeled corpus to classify.

How do you analyze social media sentiment with an API?

Two stages: data, then judgment. GET /v1/search/everywhere returns posts from 12 platforms with each post's top comments already attached; per-platform comment endpoints (/v1/tiktok/post/comments, /v1/youtube/video/comments, /v1/reddit/post/comments, 1 credit each) deepen the sample. Then a single LLM call labels every comment positive, negative, or neutral.

The problem

Sentiment lives in comments, not posts. The post says "we launched a new pricing page"; the comments say what people think about it. Harvesting comments across platforms means a different endpoint, pagination model, and response shape per platform — and naive keyword-based sentiment ("bad" = negative) falls over on sarcasm immediately.

The solution

Discovery plus enrichment plus an LLM:

  • GET /v1/search/everywhere — finds posts about the topic across 12 platforms, with top_comments pre-attached to every result (20 credits flat)
  • GET /v1/tiktok/post/comments — deeper TikTok comment pages (param: url, 1 credit)
  • GET /v1/youtube/video/comments — deeper YouTube comment pages (param: url, 1 credit)
  • GET /v1/reddit/post/comments — deeper Reddit comment threads (param: url, 1 credit)

Want an aggregate without running an LLM at all? GET /v1/content_analysis/sentiment (param: keyword, advanced tier, 5 credits) returns a positive/negative/neutral split plus a 6-axis emotional breakdown (anger, happiness, love, sadness, share, fun) over the keyword's web mentions in one call.

// recipe-sentiment.ts
// Harvests comments about a topic, builds an LLM-ready classification batch.
// Run with: SOCIALCRAWL_KEY=sc_... npx tsx recipe-sentiment.ts

const KEY = process.env.SOCIALCRAWL_KEY;
if (!KEY) throw new Error("Set SOCIALCRAWL_KEY");

const BASE = "https://www.socialcrawl.dev/v1";
const topic = "github copilot pricing";

async function get(path: string, params: Record<string, string>) {
  const url = new URL(`${BASE}/${path}`);
  for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
  const res = await fetch(url, {
    headers: { "x-api-key": KEY!, accept: "application/json" },
  });
  return (await res.json()) as {
    success: boolean;
    data?: Record<string, any>;
  };
}

// ── Step 1: Find the conversation (comments come pre-attached) ────────────
const search = await get("search/everywhere", {
  query: topic,
  lookback_days: "14",
});
if (!search.success) throw new Error("search failed");

type Sample = { platform: string; post_url: string; text: string };
const samples: Sample[] = [];

for (const item of search.data!.items as Array<{
  source: string;
  url: string;
  source_items: Array<{
    metadata?: { top_comments?: Array<{ text: string }> };
  }>;
}>) {
  for (const c of item.source_items[0]?.metadata?.top_comments ?? []) {
    samples.push({ platform: item.source, post_url: item.url, text: c.text });
  }
}

// ── Step 2: Deepen the sample on the 3 hottest posts ──────────────────────
const COMMENTS_PATH: Record<string, string> = {
  tiktok: "tiktok/post/comments",
  youtube: "youtube/video/comments",
  reddit: "reddit/post/comments",
};

const hot = (search.data!.items as Array<{ source: string; url: string }>)
  .filter((i) => COMMENTS_PATH[i.source])
  .slice(0, 3);

for (const post of hot) {
  const res = await get(COMMENTS_PATH[post.source]!, { url: post.url });
  for (const item of (res.data?.items ?? []) as Array<{
    comment?: { text?: string | null };
  }>) {
    if (item.comment?.text)
      samples.push({
        platform: post.source,
        post_url: post.url,
        text: item.comment.text,
      });
  }
}

console.log(`${samples.length} comments harvested about "${topic}"`);

// ── Step 3: Hand the batch to your LLM of choice ──────────────────────────
const prompt = `Classify the sentiment of each social media comment about
"${topic}" as "positive", "negative", or "neutral". Judge sentiment toward
the topic itself, not general mood. Account for sarcasm. Return a JSON array
of { index, sentiment, confidence } objects.

Comments:
${samples.map((s, i) => `${i}. [${s.platform}] ${s.text.slice(0, 280)}`).join("\n")}`;

// Send `prompt` to any LLM — OpenAI, Anthropic, Gemini, a local model.
// Then aggregate: share positive vs negative per platform, over time, etc.
console.log(prompt.slice(0, 600), "…");

What you get back

// `samples` after Step 2 — the LLM input batch:
[
  { "platform": "reddit", "post_url": "https://reddit.com/r/programming/...", "text": "Honestly worth every cent, it writes half my tests." },
  { "platform": "youtube", "post_url": "https://www.youtube.com/watch?v=...", "text": "$39/mo for autocomplete, lol no thanks" },
  { "platform": "tiktok", "post_url": "https://www.tiktok.com/@.../video/...", "text": "the new pricing killed it for students fr" },
  // ... 80-200 more, depending on how active the topic is
]

// And the LLM's output after Step 3:
[
  { "index": 0, "sentiment": "positive", "confidence": 0.95 },
  { "index": 1, "sentiment": "negative", "confidence": 0.9 },
  { "index": 2, "sentiment": "negative", "confidence": 0.85 },
]

Credits cost

Cost per run: 23 credits (20 for the universal search + 3 deep comment fetches × 1 credit), plus your LLM tokens. The no-LLM alternative — GET /v1/content_analysis/sentiment — is 5 credits flat for an aggregate distribution without per-comment labels.

Take it further

How to Run Sentiment Analysis on Social Media Posts and Comments | Socialcrawl