How to Build a Social Data Pipeline for 0 Credits

How to build a social data pipeline, on SocialCrawl, is the call list for a job written in plain English. GET /v1/utility/plan?query= returns that list before any paid call runs. Pass the job on query. The route does not accept q.

On 17 October 2026, six plan envelopes each came back with credits_used 0 and cached false. The calls printed inside those plans were not executed. The 0 in the title is credits_used on that planner read. The printed calls have their own prices.

One creator job asked for a creator's TikTok profile, their recent videos, and the comments on them. It came back as 3 blocked steps. Each step label is 1 credit. total.calls is 31, and hold is 31.

The same pull returned a TikTok search that dropped "in the US", a Nike watch with the search price and the monitor price in separate fields, and fill rates from one sample day, 2026-09-23. The base is https://www.socialcrawl.dev. The header is x-api-key.

How do you plan the API calls in a social data pipeline?

Send the creator job on query. The response is the ordered list of social data API calls for that job. Each step names the parameters that were filled, the ones still missing, a credit cost, and a curl.

curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  "https://www.socialcrawl.dev/v1/utility/plan?query=a%20creator%27s%20TikTok%20profile%2C%20their%20recent%20videos%20and%20the%20comments%20on%20them"

The samples in this post use $SOCIALCRAWL_API_KEY. The captured curls carried a placeholder in that header, not a live key.

On 17 October 2026 the envelope was credits_used 0 and cached false. The plan body is recipe creator_profile, uncertain false, reason planned, and confidence 1. cannot is empty. ask names handle on the profile step. The block below keeps those envelope fields next to a trimmed plan body.

{
  "recipe": "creator_profile",
  "uncertain": false,
  "reason": "planned",
  "confidence": 1,
  "credits_used": 0,
  "cached": false,
  "total": {
    "calls": 31,
    "hold": 31,
    "expected_min": 31,
    "expected_max": 31,
    "exact": true,
    "priced_with_examples": ["handle", "url"]
  },
  "steps": [
    {
      "id": "tiktok_profile",
      "method": "GET",
      "path": "/v1/tiktok/profile",
      "missing": ["handle"],
      "credits": 1,
      "run": "blocked"
    },
    {
      "id": "tiktok_posts",
      "method": "GET",
      "path": "/v1/tiktok/profile/videos",
      "missing": ["handle"],
      "credits": 1,
      "run": "blocked",
      "depends_on": "tiktok_profile",
      "binds": {
        "handle": "tiktok_profile.author.username"
      }
    },
    {
      "id": "tiktok_comments",
      "method": "GET",
      "path": "/v1/tiktok/post/comments",
      "missing": ["url"],
      "credits": 1,
      "run": "blocked",
      "depends_on": "tiktok_posts",
      "binds": {
        "url": "tiktok_posts.items[].post.url"
      }
    }
  ]
}

All three steps are blocked, and each credit label is 1. tiktok_profile is GET /v1/tiktok/profile with handle missing. tiktok_posts is GET /v1/tiktok/profile/videos, also missing handle. It depends on tiktok_profile and binds handle from tiktok_profile.author.username. tiktok_comments is GET /v1/tiktok/post/comments, missing url. It depends on tiktok_posts and binds url from tiktok_posts.items[].post.url.

Read total for the price. total.calls, hold, expected_min, and expected_max are 31, and exact is true. fan_out takes the profile into 5 video calls, then each of those into 5 comment calls. The reconstruction is 1 + 5 + 25 = 31, which matches total.calls.

GET /v1/tiktok/profile
  -> 5 x GET /v1/tiktok/profile/videos
    -> 25 x GET /v1/tiktok/post/comments

That chain is the fan-out the plan printed. None of those calls was run on 17 October 2026.

priced_with_examples lists handle and url. The 31 was priced with stand-ins for parameters the sentence did not supply. The query did not name an account. run stays blocked until you supply handle, and the comment step stays blocked until a video URL exists to bind.

size disagrees with that total. The description is "profile, one page of videos, first comment page on five videos", and size.items is 5, while total.calls is 31. The fan-out is five video calls and 25 comment calls. Use total.calls and the steps.

What does a social media data pipeline need before the first paid call?

Before the first paid call, GET /v1/utility/plan?query= has to show which parameters the words filled, which are still missing, and a total that matches the steps when size says something else.

The plan read in this pull returned credits_used 0 and cached false. You can stop before running the calls it prints. Missing handle and url stay missing, with run blocked. None of these plans filled in an account. Words that are not parameters stay out of the parameter object.

This job was search TikTok for running shoes in the US:

curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  "https://www.socialcrawl.dev/v1/tiktok/search?query=running+shoes"

The plan is recipe topic_search, uncertain false, reason planned, and confidence 0.99. cannot and ask are empty, credits_used is 0, and cached is false. One step, tiktok_search, is GET /v1/tiktok/search with query set to running shoes, missing empty, a credit label of 1, and run ready. total.calls, hold, expected_min, and expected_max are 1, exact is true, and fan_out is empty.

"in the US" produced no parameter. The plan filled neither a country field nor a region field, so this search was not geo-filtered. The guide text for post.ext.region says the region request parameter sets the proxy and is not itself the country of the row, and this plan did not send region. That ready step is a social search for the words that remained.

size disagrees again. The description says "one page from each of three platforms", and size.items is 60. The response is one TikTok search. total.calls is 1. Use that.

The Korean job 나이키 틱톡 프로필이랑 최근 영상, 댓글까지 returned the same recipe creator_profile, the same three paths, confidence 1, and total.calls 31 with hold 31 and exact true. Fan-out is the same profile-to-posts times 5 and posts-to-comments times 5. 나이키 was not written into handle. All three steps are still missing handle or url, and run is blocked on each. This pull only sent that sentence and the English jobs above.

SociaVault hardcodes its routes. Captapi, Fivetran, and the AWS guidance describe layers or steps. This pull did not measure those products.

How do you collect social media data through one set of API calls?

Collection in this pull is one ready search plus a monitor the plan leaves blocked. The job was watch mentions of Nike on TikTok and Reddit. recipe is brand_monitoring, uncertain is false, reason is planned, and confidence is 0.64 across two steps. credits_used is 0 and cached is false.

The first step, mentions, is GET /v1/search/everywhere, which is how you search TikTok and Reddit in one call. Filled parameters are query Nike and sources tiktok,reddit, with missing empty, credits 20, and run ready. The curl was present:

curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  "https://www.socialcrawl.dev/v1/search/everywhere?query=Nike&sources=tiktok%2Creddit"

The second step, monitor, is POST /v1/monitors. The body recipe is search/everywhere with those same params. cadence and webhook_url are missing, so run is blocked and ask lists both. credits on that step is null. It depends on mentions. The step note says: "Each run bills the same credits as GET /v1/search/everywhere (20 per run)." That blocked POST is how you schedule that search as a monitor.

total.calls is 1, hold is 20, expected_min and expected_max are 20, and exact is true. fan_out is empty. Step credits are 20 and null. recurring.per is 21, per_day is 1, cadence is "1 run per day", and cadence_assumed is true, while cadence is still missing. hold 20 and recurring.per 21 stay in their own fields. The response does not show 21 as a sum.

cannot has one entry, id change_alert: "There is no new-post or spike webhook. A monitor re-runs one call on a schedule and delivers each result; its alert rules compare numbers (gt, lt, pct_change)."

size says "one brand, one window" and size.items is 1, which matches total.calls 1. This plan has no priced_with_examples. The search call was not executed, so this pull has no posts, mention count, or fused result body.

How do you check field fill rate before a collection call?

Before a social media data collection call, read GET /v1/utility/endpoint?id= and check quality.fill_rate, the API field fill rate. This pull asked for three ids: tiktok/profile, tiktok/profile/videos, and search/everywhere. The plan step id tiktok_profile is a step name. The guide id for /v1/tiktok/profile is tiktok/profile. The same path-to-id rule gives the other two. All three guide reads were cached true and credits_used 0.

quality.method on the measured guides says: "Production calls made with the endpoint's example inputs, bypassing the cache; fill_rate is the share of returned rows carrying a value for each field over the rolling window, fields that are null by design are listed separately and left out of it, and freshness_hours is the median age of the newest dated row per call." Those sample calls are what bypass the cache. The guide reads in this pull were cache hits. A 1 in fill_rate is the JSON number 1. sampled_calls 1 is one call, and the quality object has no row count.

tiktok/profile came back measured. Thirteen fill_rate values are exactly 1. Minimum and maximum are both 1. Guide credits.cost for the profile call is 1, matching the plan step. contract.quality.fill_rate_window is 2026-09-23..2026-09-23 with sampled_calls 1. field_health is 2026-09-28, a different date from that window.

{
  "measured": true,
  "window_days": 30,
  "measured_from": "2026-09-23",
  "measured_to": "2026-09-23",
  "sampled_calls": 1,
  "freshness_hours": null,
  "null_by_design": [],
  "by_rung": [],
  "fill_rate": {
    "author.avatar_url": 1,
    "author.bio": 1,
    "author.display_name": 1,
    "author.ext.bio_link": 1,
    "author.ext.business_category": 1,
    "author.followers": 1,
    "author.following": 1,
    "author.id": 1,
    "author.likes_count": 1,
    "author.posts_count": 1,
    "author.private": 1,
    "author.username": 1,
    "author.verified": 1
  }
}

The profile contract lists 21 output fields. Thirteen have a fill number. Eight have fill null: author.joined_at, author.url, author.location, and five computed.* leaves (computed.content_category, computed.engagement_rate, computed.estimated_reach, computed.language, computed.save_rate). author.location has source_hint sample. A null fill is unmeasured, and null_by_design is empty.

tiktok/profile/videos uses the same window dates, the same window_days 30, and the same sampled_calls 1. freshness_hours is 0.8. null_by_design and by_rung are empty. Nineteen fields are exactly 1. The profile map is separate, and its freshness_hours is null. field_health is 2026-09-28 again, still separate from the fill window. The guide price label is metered, "1-5", at 1 credit per page unless a paid label is set. The plan priced this step at 1 and did not set label.

The videos contract lists 38 output fields. Nineteen have a fill number. Nineteen have fill null, including post.url, post.author.verified, post.flags.deleted, post.ext.on_screen_texts, and the computed.* leaves. post.ext.region is inside that unmeasured set. fill is null and live is true.

search/everywhere, the first path on the Nike plan, was not measured:

{
  "measured": false,
  "reason": "not sampled in the last 30 days",
  "null_by_design": []
}

The quality object stops at measured, reason, and an empty null_by_design. contract.quality.fill_rate_window, sampled_calls, and field_health are null. Guide credits.cost is 20, the same figure as the Nike search step, and that 20 is the call price. All 30 contract output fields have fill null, in line with measured false.

window_days is 30 on the two measured endpoints. This pull did not measure 30 days. Both are one call on 2026-09-23, and the pull date is 17 October 2026, so that sample day is 24 days old. A fill rate of 1 means every row on that one example-input call had the field. The quality object has no row count, and the 1 is not a cross-account benchmark. by_rung is empty on both, so there is no per-rung split.

How do you run the plan without paying for the calls?

Resend the planner and the guide, then stop. Those reads returned credits_used 0 on 17 October 2026. The holds printed inside the plans are later prices.

curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  "https://www.socialcrawl.dev/v1/utility/plan?query=QUERY"

curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
  "https://www.socialcrawl.dev/v1/utility/endpoint?id=ID"

Resend them with one API key. Compare steps, cannot, and total on these six jobs:

1. a creator's TikTok profile, their recent videos and the comments on them
2. watch mentions of Nike on TikTok and Reddit
3. search TikTok for running shoes in the US
4. make a promo video of my product
5. follower history chart for a TikTok account
6. 나이키 틱톡 프로필이랑 최근 영상, 댓글까지

Then compare quality for tiktok/profile, tiktok/profile/videos, and search/everywhere. On 17 October 2026 those reads returned credits_used 0. You do not need to run the TikTok calls or the monitor call the plan prints.

make a promo video of my product came back uncertain true, reason no_recipe, steps empty, and cannot empty, with confidence 1. The body has no total object and no curl.

follower history chart for a TikTok account still returns steps. uncertain is false, recipe is creator_profile, confidence is 0.53, and there are two blocked steps: profile and videos, with no comments step. total.calls is 6, hold is 6, and exact is true. Fan-out is the profile plus 5 video calls, which is 1 + 5 = 6. priced_with_examples lists handle only. size still describes comments on five videos, and there is no comments step, so use total.calls 6. cannot id history says: "Past values are not stored: each call returns today's number, so a follower chart or a history cannot be fetched. Re-run the call on a schedule and keep your own series."

The route is documented on the utility plan route. The dashboard assistant can show the same plan card, with confirmation still in front of any credit spend. The count strip under a run is not built, and this pull did not open the card, the chips, or a pasted-error reply.

FAQ

Is the social data pipeline planner free?

The plan read and the endpoint guide read returned credits_used 0 on 17 October 2026. All six plans had cached false. All three guide reads had credits_used 0. The calls those plans print were not made. The creator hold is 31, the Nike search step is 20, and the running-shoes step is 1.

What does the plan endpoint return (calls, parameters, credit cost)?

GET /v1/utility/plan?query= takes query. Each step has a method, a path, filled parameters, missing parameters, a credit label, run of ready or blocked, and a curl. depends_on and binds appear when a later step needs an earlier field. total.calls and hold are the priced fan-out. cannot is a separate list. The read does not run the steps. On the creator job the three labels are 1 and total.calls is 31.

What does fill rate mean on an endpoint you are about to call?

quality.fill_rate on GET /v1/utility/endpoint is the share of returned rows that carried a value. In this pull, tiktok/profile has 13 fields at 1 and tiktok/profile/videos has 19 fields at 1, each from sampled_calls 1 on 2026-09-23, while window_days is 30. freshness_hours is null on the profile and 0.8 on the videos. search/everywhere is measured false, reason "not sampled in the last 30 days", so it has no fill rate. A null fill is unmeasured. null_by_design was empty on all three guides.

What will the planner not plan?

make a promo video of my product returned uncertain true, reason no_recipe, empty steps, and empty cannot. It does not guess a call. Follower history is separate. It returns two blocked steps, total.calls 6, and cannot id history, because past values are not stored and each call returns today's number. Nike's cannot id change_alert says there is no new-post or spike webhook. A monitor re-runs one call on a schedule.

Which languages can you use to describe the job?

Only the sentences this pull sent. The English creator sentence and 나이키 틱톡 프로필이랑 최근 영상, 댓글까지 both returned recipe creator_profile, 3 steps, confidence 1, and total.calls 31. The Korean sentence left handle missing. No other language was sent, and this pull did not test Korean error copy in the dashboard.