# How to Build a Social Data Pipeline for 0 Credits (https://www.socialcrawl.dev/blog/how-to-build-a-social-data-pipeline)
> A plain-English job returns the exact API calls, parameters, and credit cost at 0 credits. A creator plan is 3 steps and 31 calls. Fill rates cover one day.
How to build a social data pipeline, on SocialCrawl, is the call list for a job written in plain English. `GET /v1/utility/plan?query=` returns that list before any paid call runs. Pass the job on `query`. The route does not accept `q`.
On 17 October 2026, six plan envelopes each came back with `credits_used` 0 and `cached` false. The calls printed inside those plans were not executed. The 0 in the title is `credits_used` on that planner read. The printed calls have their own prices.
One creator job asked for a creator's TikTok profile, their recent videos, and the comments on them. It came back as 3 blocked steps. Each step label is 1 credit. `total.calls` is 31, and `hold` is 31.
The same pull returned a TikTok search that dropped "in the US", a Nike watch with the search price and the monitor price in separate fields, and fill rates from one sample day, 2026-09-23. The base is `https://www.socialcrawl.dev`. The header is `x-api-key`.
## How do you plan the API calls in a social data pipeline?
Send the creator job on `query`. The response is the ordered list of social data API calls for that job. Each step names the parameters that were filled, the ones still missing, a credit cost, and a curl.
```bash
curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
"https://www.socialcrawl.dev/v1/utility/plan?query=a%20creator%27s%20TikTok%20profile%2C%20their%20recent%20videos%20and%20the%20comments%20on%20them"
```
The samples in this post use `$SOCIALCRAWL_API_KEY`. The captured curls carried a placeholder in that header, not a live key.
On 17 October 2026 the envelope was `credits_used` 0 and `cached` false. The plan body is `recipe` `creator_profile`, `uncertain` false, `reason` `planned`, and confidence 1. `cannot` is empty. `ask` names `handle` on the profile step. The block below keeps those envelope fields next to a trimmed plan body.
```json
{
"recipe": "creator_profile",
"uncertain": false,
"reason": "planned",
"confidence": 1,
"credits_used": 0,
"cached": false,
"total": {
"calls": 31,
"hold": 31,
"expected_min": 31,
"expected_max": 31,
"exact": true,
"priced_with_examples": ["handle", "url"]
},
"steps": [
{
"id": "tiktok_profile",
"method": "GET",
"path": "/v1/tiktok/profile",
"missing": ["handle"],
"credits": 1,
"run": "blocked"
},
{
"id": "tiktok_posts",
"method": "GET",
"path": "/v1/tiktok/profile/videos",
"missing": ["handle"],
"credits": 1,
"run": "blocked",
"depends_on": "tiktok_profile",
"binds": {
"handle": "tiktok_profile.author.username"
}
},
{
"id": "tiktok_comments",
"method": "GET",
"path": "/v1/tiktok/post/comments",
"missing": ["url"],
"credits": 1,
"run": "blocked",
"depends_on": "tiktok_posts",
"binds": {
"url": "tiktok_posts.items[].post.url"
}
}
]
}
```
All three steps are `blocked`, and each credit label is 1. `tiktok_profile` is `GET /v1/tiktok/profile` with `handle` missing. `tiktok_posts` is `GET /v1/tiktok/profile/videos`, also missing `handle`. It depends on `tiktok_profile` and binds `handle` from `tiktok_profile.author.username`. `tiktok_comments` is `GET /v1/tiktok/post/comments`, missing `url`. It depends on `tiktok_posts` and binds `url` from `tiktok_posts.items[].post.url`.
Read `total` for the price. `total.calls`, `hold`, `expected_min`, and `expected_max` are 31, and `exact` is true. `fan_out` takes the profile into 5 video calls, then each of those into 5 comment calls. The reconstruction is 1 + 5 + 25 = 31, which matches `total.calls`.
```text
GET /v1/tiktok/profile
-> 5 x GET /v1/tiktok/profile/videos
-> 25 x GET /v1/tiktok/post/comments
```
That chain is the fan-out the plan printed. None of those calls was run on 17 October 2026.
`priced_with_examples` lists `handle` and `url`. The 31 was priced with stand-ins for parameters the sentence did not supply. The query did not name an account. `run` stays `blocked` until you supply `handle`, and the comment step stays blocked until a video URL exists to bind.
`size` disagrees with that total. The description is "profile, one page of videos, first comment page on five videos", and `size.items` is 5, while `total.calls` is 31. The fan-out is five video calls and 25 comment calls. Use `total.calls` and the steps.
## What does a social media data pipeline need before the first paid call?
Before the first paid call, `GET /v1/utility/plan?query=` has to show which parameters the words filled, which are still missing, and a total that matches the steps when `size` says something else.
The plan read in this pull returned `credits_used` 0 and `cached` false. You can stop before running the calls it prints. Missing `handle` and `url` stay missing, with `run` `blocked`. None of these plans filled in an account. Words that are not parameters stay out of the parameter object.
This job was `search TikTok for running shoes in the US`:
```bash
curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
"https://www.socialcrawl.dev/v1/tiktok/search?query=running+shoes"
```
The plan is `recipe` `topic_search`, `uncertain` false, `reason` `planned`, and confidence 0.99. `cannot` and `ask` are empty, `credits_used` is 0, and `cached` is false. One step, `tiktok_search`, is `GET /v1/tiktok/search` with `query` set to `running shoes`, `missing` empty, a credit label of 1, and `run` `ready`. `total.calls`, `hold`, `expected_min`, and `expected_max` are 1, `exact` is true, and `fan_out` is empty.
"in the US" produced no parameter. The plan filled neither a country field nor a region field, so this search was not geo-filtered. The guide text for `post.ext.region` says the `region` request parameter sets the proxy and is not itself the country of the row, and this plan did not send `region`. That ready step is a [social search](/blog/social-search-engine) for the words that remained.
`size` disagrees again. The description says "one page from each of three platforms", and `size.items` is 60. The response is one TikTok search. `total.calls` is 1. Use that.
The Korean job `나이키 틱톡 프로필이랑 최근 영상, 댓글까지` returned the same `recipe` `creator_profile`, the same three paths, confidence 1, and `total.calls` 31 with `hold` 31 and `exact` true. Fan-out is the same profile-to-posts times 5 and posts-to-comments times 5. `나이키` was not written into `handle`. All three steps are still missing `handle` or `url`, and `run` is `blocked` on each. This pull only sent that sentence and the English jobs above.
[SociaVault](https://sociavault.com/blog/unified-social-media-pipeline-2025) hardcodes its routes. [Captapi](https://captapi.com/blog/data-pipeline-automation), [Fivetran](https://www.fivetran.com/blog/build-a-data-pipeline), and the [AWS guidance](https://docs.aws.amazon.com/solutions/social-media-data-pipeline-on-aws/) describe layers or steps. This pull did not measure those products.
## How do you collect social media data through one set of API calls?
Collection in this pull is one ready search plus a monitor the plan leaves blocked. The job was `watch mentions of Nike on TikTok and Reddit`. `recipe` is `brand_monitoring`, `uncertain` is false, `reason` is `planned`, and confidence is 0.64 across two steps. `credits_used` is 0 and `cached` is false.
The first step, `mentions`, is `GET /v1/search/everywhere`, which is how you [search TikTok and Reddit in one call](/blog/best-multi-platform-social-search-api). Filled parameters are `query` `Nike` and `sources` `tiktok,reddit`, with `missing` empty, `credits` 20, and `run` `ready`. The curl was present:
```bash
curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
"https://www.socialcrawl.dev/v1/search/everywhere?query=Nike&sources=tiktok%2Creddit"
```
The second step, `monitor`, is `POST /v1/monitors`. The body recipe is `search/everywhere` with those same params. `cadence` and `webhook_url` are missing, so `run` is `blocked` and `ask` lists both. `credits` on that step is null. It depends on `mentions`. The step note says: "Each run bills the same credits as `GET /v1/search/everywhere` (20 per run)." That blocked POST is how you [schedule that search as a monitor](/blog/social-media-monitoring-api).
`total.calls` is 1, `hold` is 20, `expected_min` and `expected_max` are 20, and `exact` is true. `fan_out` is empty. Step credits are 20 and null. `recurring.per` is 21, `per_day` is 1, `cadence` is "1 run per day", and `cadence_assumed` is true, while `cadence` is still missing. `hold` 20 and `recurring.per` 21 stay in their own fields. The response does not show 21 as a sum.
`cannot` has one entry, id `change_alert`: "There is no new-post or spike webhook. A monitor re-runs one call on a schedule and delivers each result; its alert rules compare numbers (gt, lt, pct_change)."
`size` says "one brand, one window" and `size.items` is 1, which matches `total.calls` 1. This plan has no `priced_with_examples`. The search call was not executed, so this pull has no posts, mention count, or fused result body.
## How do you check field fill rate before a collection call?
Before a social media data collection call, read `GET /v1/utility/endpoint?id=` and check `quality.fill_rate`, the API field fill rate. This pull asked for three ids: `tiktok/profile`, `tiktok/profile/videos`, and `search/everywhere`. The plan step id `tiktok_profile` is a step name. The guide id for `/v1/tiktok/profile` is `tiktok/profile`. The same path-to-id rule gives the other two. All three guide reads were `cached` true and `credits_used` 0.
`quality.method` on the measured guides says: "Production calls made with the endpoint's example inputs, bypassing the cache; fill_rate is the share of returned rows carrying a value for each field over the rolling window, fields that are null by design are listed separately and left out of it, and freshness_hours is the median age of the newest dated row per call." Those sample calls are what bypass the cache. The guide reads in this pull were cache hits. A 1 in `fill_rate` is the JSON number 1. `sampled_calls` 1 is one call, and the quality object has no row count.
`tiktok/profile` came back measured. Thirteen `fill_rate` values are exactly 1. Minimum and maximum are both 1. Guide `credits.cost` for the profile call is 1, matching the plan step. `contract.quality.fill_rate_window` is `2026-09-23..2026-09-23` with `sampled_calls` 1. `field_health` is 2026-09-28, a different date from that window.
```json
{
"measured": true,
"window_days": 30,
"measured_from": "2026-09-23",
"measured_to": "2026-09-23",
"sampled_calls": 1,
"freshness_hours": null,
"null_by_design": [],
"by_rung": [],
"fill_rate": {
"author.avatar_url": 1,
"author.bio": 1,
"author.display_name": 1,
"author.ext.bio_link": 1,
"author.ext.business_category": 1,
"author.followers": 1,
"author.following": 1,
"author.id": 1,
"author.likes_count": 1,
"author.posts_count": 1,
"author.private": 1,
"author.username": 1,
"author.verified": 1
}
}
```
The profile contract lists 21 output fields. Thirteen have a fill number. Eight have `fill` null: `author.joined_at`, `author.url`, `author.location`, and five `computed.*` leaves (`computed.content_category`, `computed.engagement_rate`, `computed.estimated_reach`, `computed.language`, `computed.save_rate`). `author.location` has `source_hint` `sample`. A null `fill` is unmeasured, and `null_by_design` is empty.
`tiktok/profile/videos` uses the same window dates, the same `window_days` 30, and the same `sampled_calls` 1. `freshness_hours` is 0.8. `null_by_design` and `by_rung` are empty. Nineteen fields are exactly 1. The profile map is separate, and its `freshness_hours` is null. `field_health` is 2026-09-28 again, still separate from the fill window. The guide price label is metered, "1-5", at 1 credit per page unless a paid label is set. The plan priced this step at 1 and did not set `label`.
| Field | fill_rate |
| --- | --- |
| post.author.avatar_url | 1 |
| post.author.display_name | 1 |
| post.author.username | 1 |
| post.content.duration_seconds | 1 |
| post.content.media_urls | 1 |
| post.content.text | 1 |
| post.content.thumbnail_url | 1 |
| post.engagement.comments | 1 |
| post.engagement.likes | 1 |
| post.engagement.saves | 1 |
| post.engagement.shares | 1 |
| post.engagement.views | 1 |
| post.ext.author_followers | 1 |
| post.ext.author_id | 1 |
| post.ext.download_count | 1 |
| post.ext.music_id | 1 |
| post.flags.pinned | 1 |
| post.id | 1 |
| post.published_at | 1 |
The videos contract lists 38 output fields. Nineteen have a fill number. Nineteen have `fill` null, including `post.url`, `post.author.verified`, `post.flags.deleted`, `post.ext.on_screen_texts`, and the `computed.*` leaves. `post.ext.region` is inside that unmeasured set. `fill` is null and `live` is true.
`search/everywhere`, the first path on the Nike plan, was not measured:
```json
{
"measured": false,
"reason": "not sampled in the last 30 days",
"null_by_design": []
}
```
The quality object stops at `measured`, `reason`, and an empty `null_by_design`. `contract.quality.fill_rate_window`, `sampled_calls`, and `field_health` are null. Guide `credits.cost` is 20, the same figure as the Nike search step, and that 20 is the call price. All 30 contract output fields have `fill` null, in line with `measured` false.
`window_days` is 30 on the two measured endpoints. This pull did not measure 30 days. Both are one call on 2026-09-23, and the pull date is 17 October 2026, so that sample day is 24 days old. A fill rate of 1 means every row on that one example-input call had the field. The quality object has no row count, and the 1 is not a cross-account benchmark. `by_rung` is empty on both, so there is no per-rung split.
## How do you run the plan without paying for the calls?
Resend the planner and the guide, then stop. Those reads returned `credits_used` 0 on 17 October 2026. The holds printed inside the plans are later prices.
```bash
curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
"https://www.socialcrawl.dev/v1/utility/plan?query=QUERY"
```
```bash
curl -H "x-api-key: $SOCIALCRAWL_API_KEY" \
"https://www.socialcrawl.dev/v1/utility/endpoint?id=ID"
```
Resend them with one API key. Compare `steps`, `cannot`, and `total` on these six jobs:
1. `a creator's TikTok profile, their recent videos and the comments on them`
2. `watch mentions of Nike on TikTok and Reddit`
3. `search TikTok for running shoes in the US`
4. `make a promo video of my product`
5. `follower history chart for a TikTok account`
6. `나이키 틱톡 프로필이랑 최근 영상, 댓글까지`
Then compare `quality` for `tiktok/profile`, `tiktok/profile/videos`, and `search/everywhere`. On 17 October 2026 those reads returned `credits_used` 0. You do not need to run the TikTok calls or the monitor call the plan prints.
`make a promo video of my product` came back `uncertain` true, `reason` `no_recipe`, `steps` empty, and `cannot` empty, with confidence 1. The body has no `total` object and no curl.
`follower history chart for a TikTok account` still returns steps. `uncertain` is false, `recipe` is `creator_profile`, confidence is 0.53, and there are two blocked steps: profile and videos, with no comments step. `total.calls` is 6, `hold` is 6, and `exact` is true. Fan-out is the profile plus 5 video calls, which is 1 + 5 = 6. `priced_with_examples` lists `handle` only. `size` still describes comments on five videos, and there is no comments step, so use `total.calls` 6. `cannot` id `history` says: "Past values are not stored: each call returns today's number, so a follower chart or a history cannot be fetched. Re-run the call on a schedule and keep your own series."
The route is documented on the [utility plan route](/docs/utility). The dashboard assistant can show the same plan card, with confirmation still in front of any credit spend. The count strip under a run is not built, and this pull did not open the card, the chips, or a pasted-error reply.
## FAQ
### Is the social data pipeline planner free?
The plan read and the endpoint guide read returned `credits_used` 0 on 17 October 2026. All six plans had `cached` false. All three guide reads had `credits_used` 0. The calls those plans print were not made. The creator hold is 31, the Nike search step is 20, and the running-shoes step is 1.
### What does the plan endpoint return (calls, parameters, credit cost)?
`GET /v1/utility/plan?query=` takes `query`. Each step has a method, a path, filled parameters, missing parameters, a credit label, `run` of ready or blocked, and a curl. `depends_on` and `binds` appear when a later step needs an earlier field. `total.calls` and `hold` are the priced fan-out. `cannot` is a separate list. The read does not run the steps. On the creator job the three labels are 1 and `total.calls` is 31.
### What does fill rate mean on an endpoint you are about to call?
`quality.fill_rate` on `GET /v1/utility/endpoint` is the share of returned rows that carried a value. In this pull, `tiktok/profile` has 13 fields at 1 and `tiktok/profile/videos` has 19 fields at 1, each from `sampled_calls` 1 on 2026-09-23, while `window_days` is 30. `freshness_hours` is null on the profile and 0.8 on the videos. `search/everywhere` is `measured` false, reason "not sampled in the last 30 days", so it has no fill rate. A null `fill` is unmeasured. `null_by_design` was empty on all three guides.
### What will the planner not plan?
`make a promo video of my product` returned `uncertain` true, `reason` `no_recipe`, empty `steps`, and empty `cannot`. It does not guess a call. Follower history is separate. It returns two blocked steps, `total.calls` 6, and `cannot` id `history`, because past values are not stored and each call returns today's number. Nike's `cannot` id `change_alert` says there is no new-post or spike webhook. A monitor re-runs one call on a schedule.
### Which languages can you use to describe the job?
Only the sentences this pull sent. The English creator sentence and `나이키 틱톡 프로필이랑 최근 영상, 댓글까지` both returned `recipe` `creator_profile`, 3 steps, confidence 1, and `total.calls` 31. The Korean sentence left `handle` missing. No other language was sent, and this pull did not test Korean error copy in the dashboard.