# Computed fields (/docs/computed-fields) Computed fields [#computed-fields] Most APIs hand you the raw numbers and walk away. SocialCrawl runs every payload through a transformer that attaches a `computed` block alongside the upstream data — the same shape on every platform, so a TikTok creator's engagement rate is directly comparable to an Instagram one. Four fields live in `computed`: | Field | Type | Range | When it's `null` | | ------------------ | ---------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `engagement_rate` | `number \| null` | `0.0` – `1.0` | Divisor missing or zero; on posts, also when all of `likes`/`comments`/`shares` are `null`; on profiles, when the raw ratio exceeds `1.0` | | `language` | `string \| null` | ISO 639-1 (`en`, `ko`, `ja`, ...) | Latin scripts: cleaned prose under 30 chars, under 3 words, or a title-cased headline. Non-Latin scripts: never length-gated. Also when unrecognised | | `content_category` | `string \| null` | 14 categories or `"other"` | Input text under 10 trimmed chars (a looser, separate gate from `language`'s) | | `estimated_reach` | `number \| null` | integer ≥ 0 | Posts: `views` missing or `0` (so any post without a view count, e.g. Instagram photos). Profiles: always `null` — see below | Every value is either a real number or honestly `null`. We never substitute `0` for "we don't know" — the difference matters when you're sorting or filtering. If a value was forced into range (typically `engagement_rate` exceeding `1.0`), an explanatory string lands in [`data._warnings`](/docs/response-schema.md#data_warnings--partial-data-channel) so you can see it happened. Computed fields attach to three archetypes: * **`Author`** — `data.computed` on profile responses. * **`Post`** — `data.computed` on single-post responses **and on every item** of a `PostList`. * **`Comment`** — `data.computed` on single-comment responses **and on every item** of a `CommentList`. Comments carry a slimmer block: just `{ language }`. The other three fields need signals a comment doesn't have (`engagement_rate` needs a views divisor, `content_category` and `estimated_reach` don't generalize to one-line replies), and always-null columns would be shape noise. `Audience`, `Transcript`, and `SearchResult` archetypes don't carry a `computed` block. *** `engagement_rate` [#engagement_rate] A normalised, comparable engagement signal in the range `[0.0, 1.0]`. Rounded to 6 decimals. Author variant (profiles) [#author-variant-profiles] ``` engagement_rate = author.likes_count / author.followers ``` Returns `null` when `followers` is `0` or `likes_count` is absent. "Zero engagement" and "we don't have the data" are different states; we report the second one honestly. **Instagram fallback.** Instagram's profile payload never populates `author.likes_count`. When you call a profile endpoint on Instagram, we instead read up to \~12 recent posts embedded in the same response and compute: ``` engagement_rate = (mean(post_likes) + mean(post_comments)) / followers ``` A post counts only if both `likes_count` and `comments_count` are present. Zero usable posts → `null`. This fallback is **Instagram-only** today. Post variant (single posts and PostList items) [#post-variant-single-posts-and-postlist-items] ``` engagement_rate = (likes + comments + shares) / views ``` Returns `null` when `views` is missing or `0`. This is intentional — pre-views-era tweets and platforms that don't report views must be honest. The previous implementation fell back to `divisor = 1`, which silently surfaced the numerator (e.g. `26,573`) as an engagement rate. That's worse than `null`. It also returns `null` when the numerator is structurally unavailable — i.e. **all three** of `likes`, `comments`, and `shares` are `null` (Twitch VODs, Spotify tracks, Rumble list items). An explicit upstream `0` in any one of the three is real data, so a genuine zero-engagement post still reports `0`. Note this means a post can carry a populated `estimated_reach` (views present) alongside a `null` engagement\_rate (no engagement signals) — the two fields gate independently on `views`. **One formula, everywhere.** This is the same calculation on every platform. There is no per-platform variant of the post-level formula. What changes between platforms is not the formula, it's which inputs the platform exposes: * **`likes`, `comments`, `shares` are treated as `0` when absent.** If a platform does not report a shares count (Instagram and YouTube do not), the numerator simply becomes `likes + comments`. We do not invent a shares number, and we do not null the whole rate just because one addend is missing. This is the single most common reason a hand-recomputed rate won't match ours: if you reproduce the formula assuming a non-null `shares` on Instagram or YouTube, your result will differ from the value we return, because on our side `shares` contributed `0`. * **`views` is treated as `null` when absent, and a `null` divisor makes the whole rate `null`.** Unlike the addends, a missing `views` is not coerced to `0` (dividing by zero is meaningless), so the rate is honestly `null`. Why your recomputation might not match ours (per-platform) [#why-your-recomputation-might-not-match-ours-per-platform] If you validate our `engagement_rate` by recomputing it from the raw fields, this table explains every case of disagreement. In each case the value we return is correct; the mismatch comes from an input you may not have that we also don't (or vice versa). | Platform | Reproducible from `(likes+comments+shares)/views`? | Why | | -------------------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **TikTok** | Yes, exactly | All five engagement fields (`views, likes, comments, shares, saves`) are present. Your recomputation will match. | | **Twitter/X** | Yes, on rows where `engagement_rate` is not `null` | Where we return a rate, it reproduces exactly. A large share of tweets return `engagement_rate: null` because Twitter did not report a native `views` count for that tweet (older tweets and many replies). `null` here means "no view count available", not "zero engagement". Filter on `engagement_rate IS NOT NULL` before using it as a feature. | | **Instagram** | No | `shares` (and `saves`) are not exposed on the public upstream, so on our side `shares = 0` and the rate is effectively `(likes + comments) / views`. Photo posts also have no view count, so their rate is `null`; Reels have views and do get a rate. A shares-inclusive reproduction cannot match either branch. Use our value directly. | | **YouTube** | No | Same mechanism as Instagram: `shares` is not returned, so the rate is `(likes + comments) / views`. | | **Facebook** | Partially | We map the upstream `like_count` verbatim into `likes`. That field appears to be the aggregate reaction count (like + love + haha + ...), not likes alone, so our numerator can run higher than a like-only reproduction. We are confirming this upstream; treat our Facebook rate as reaction-inclusive for now. | | **YouTube livestreams** | No (rate is `0` or `null`) | The live/upcoming list shape carries a view count but no like or comment counts, so the numerator is `0` (rate `0`) or the view count is absent pre-broadcast (rate `null`). Hydrate a finished stream through the single-video endpoint to get full engagement. | | **Reddit, Threads (list items)** | No (rate is `null`) | Neither exposes a per-post `views` count on list/feed items, so no rate is possible. For Reddit, use `score` (upvotes) as the engagement proxy. A single Threads post fetched by URL does carry a view count and will get a rate. | The small share of **post** rows where the raw math exceeds `1.0` are clamped to `1.0` with a `_warnings` note (see below); this is a fraction of a percent of rows and does not affect the population. Out-of-range values: post rows clamp, profile rows go `null` [#out-of-range-values-post-rows-clamp-profile-rows-go-null] The two archetypes handle a raw value above `1.0` differently, because the causes are different: * **Post rows clamp to `1.0`.** A post ratio above `1.0` (an old tweet whose likes outweigh its under-reported views, heavy reshare activity exceeding views) is a real if unbounded signal, so it is pinned to `1.0` and a warning is appended: ```json { "data": { "computed": { "engagement_rate": 1.0 }, "_warnings": [ "computed.engagement_rate: value exceeded 1.0 (raw: 1.42); clamped" ] } } ``` * **Profile (Author) rows return `null`.** An author ratio above `1.0` only ever arises from dividing a *cumulative lifetime* like count (TikTok, YouTube, Facebook profiles) by the current follower count. That is not a real engagement rate, so instead of fabricating `1.0` we return `null` with an explanatory warning: ```json { "data": { "computed": { "engagement_rate": null }, "_warnings": [ "computed.engagement_rate: author ratio exceeded 1.0 (raw: 86.32); returned null — a lifetime likes/followers ratio is not a real engagement rate" ] } } ``` The defensive lower clamp at `0` exists for the same reason but rarely fires — our arithmetic can't produce negatives from positive inputs. *** `language` [#language] ISO 639-1 two-letter code (`en`, `ko`, `ja`, `pt-BR` is **not** used — we emit `pt`). * **Input on `Author`**: `author.bio`. * **Input on `Post`**: `post.content.text`. * **Input on `Comment`**: `comment.text` — via a comment-tuned detector, described [below](#on-comments). * **Returns `null`** when the input is missing or non-string. Beyond that, the floors depend on the script: * **Non-Latin scripts** (Korean, Japanese, Chinese, Arabic, Devanagari, Thai) are detected by Unicode range and classify at **any length**. A three-character Korean bio returns `ko`. * **Latin scripts** go through trigram classification, which needs real prose to be trustworthy. We first strip URLs, `@mentions`, `#hashtags`, and list markers, then require the remaining prose to be at least **30 characters** and at least **3 alphabetic words**. A short mostly-Title-Cased string (a headline or a name, not prose) also returns `null`. These floors are confidence gates. Below them the classifier misfires more often than it helps, so we return nothing rather than a confident wrong code. Detection strategy [#detection-strategy] Two passes: 1. **Unicode fast-path** for non-Latin scripts, applied first and at any input length. If the input contains characters in these ranges, we return the language code directly: * Korean (`가–힯`, Hangul Jamo) → `ko` * Japanese (Hiragana / Katakana) → `ja` * CJK Unified Ideographs → `zh` * Arabic → `ar` * Devanagari → `hi` * Thai → `th` 2. **Trigram classification** via [`franc-min`](https://github.com/wooorm/franc) for everything else, run on the cleaned prose that cleared the floors above. franc returns `und` when below its internal confidence threshold; we map that to `null` rather than guessing. Supported codes [#supported-codes] If franc's ISO 639-3 code maps to one of the codes below, you get the two-letter form. Anything else collapses to `null` — we'd rather emit nothing than leak an obscure 3-letter code through the public surface. ``` ar bg ca cs da de el en es fa fi fr he hi hu id it ja ko nl no pl pt ro ru sv th tr uk vi zh ``` 31 codes total. If you need a language we don't surface, let us know. On comments [#on-comments] Comments are far shorter and noisier than captions or bios, and the trigram classifier above misfires badly below its floors — which would leave roughly half of all real comments at `null`. Comment responses therefore use a separate detection stack, in order: 1. **Prose strip** — URLs, `@mentions`, `#hashtags`, and emoji are removed first. A comment that is only emoji, only a date stamp, or only a handle returns `null`. 2. **Unicode fast-path** — the same non-Latin script ranges as above, plus Hebrew (`he`) and Greek (`el`), at any length. `좋아요` → `ko`. 3. **Noise gate** — what's left must be at least 2 words and 4 letters of prose. Below that a comment is a name or an interjection; no classifier is trustworthy there, so we return `null` rather than guess. 4. **The platform's own label** — TikTok attaches a per-comment language label upstream, and Threads sometimes does; when it is one of the 31 published codes we return it directly. 5. **Short-text classification** — everything else runs through [ELD](https://github.com/nitotm/efficient-language-detector-js), a detector built for short text, and only a reliable, clear-margin answer is returned. This is why `Great video!` gets `en` on a comment while the same string in a bio would fall under the 30-character floor. The published code set is identical to the table above; anything outside it collapses to `null`. Expect roughly a quarter of real-world comments to return `null` — emoji-only reactions, date stamps, and one-word replies genuinely carry no language signal, and an honest `null` beats a confident wrong label when you're filtering a corpus. *** `content_category` [#content_category] One of 14 hand-curated categories or `"other"`. Returns `null` if the input text is missing or under 10 trimmed characters. This is a **different, looser gate** than `language`'s — a bio can carry a category and no language, or the reverse. | Category | Sample matched keywords | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `tech` | programming, developer, software, ai, saas, blockchain, frontend | | `food` | cooking, recipe, chef, restaurant, baking, vegan | | `gaming` | gaming, esports, twitch, fortnite, valorant, fps | | `fashion` | fashion, outfit, designer, ootd, streetwear | | `beauty` | makeup, skincare, lipstick, serum, moisturizer | | `fitness` | workout, gym, cardio, yoga, marathon, protein | | `travel` | adventure, destination, vacation, backpacking, wanderlust | | `music` | song, artist, album, concert, producer, spotify | | `education` | learning, course, tutorial, university, lecture | | `entertainment` | movie, tv, netflix, celebrity, comedy, series | | `sports` | football, basketball, nba, olympics, championship | | `business` | entrepreneur, ceo, marketing, finance, fundraising | | `news` | politics, economy, election, journalist, parliament | | `lifestyle` | wellness, mindfulness, productivity, minimalism, diy | | `other` | fewer than two distinct in-category keywords matched (the input was long enough to evaluate, but nothing corroborated a single category) | Matching rules [#matching-rules] * **Short keywords** (≤3 characters, single word) — e.g. `"ai"`, `"tv"`, `"dj"` — require an **exact token match**. They match `"building with ai"` but not `"hair"` or `"said"`. * Those same three tokens (`ai`, `tv`, `dj`) are ambiguous across languages — `"ai"` is colloquial Portuguese and French — so they only count when `computed.language` is `en` or `null`. `content_category` is therefore computed **after** `language` and can depend on it: the same bio scores differently once we know it isn't English. * **Longer or multi-word keywords** use a Unicode word-boundary regex. `"machine learning"` matches inside `"I love machine learning!"` but doesn't bleed into adjacent words. * **A category must match at least two distinct keywords to win at all.** A single incidental word ("team", "protein", "producer") resolves to `"other"` rather than a confident wrong label. This is the most common reason a bio you'd call obviously on-topic comes back as `"other"`. * Among categories that clear that bar, the highest-scoring one wins. Ties resolve to whichever category is declared first in the source — `tech, food, gaming, fashion, beauty, fitness, travel, music, education, entertainment, sports, business, news, lifestyle` — not alphabetically. If you build product on top of `content_category`, treat it as a **rough first-pass classifier**, not a taxonomy. It's keyword-based — fast, deterministic, and noisy. For nuanced classification, layer your own model on top of the bio/text fields. *** `estimated_reach` [#estimated_reach] A rough upper-bound estimate of how many distinct accounts a post is reaching. Always an integer ≥ 0, or `null`. Author variant (profiles): always `null` [#author-variant-profiles-always-null] On the `Author` archetype, `estimated_reach` is **always `null`**. Reach is not computable from a follower count plus a followers-normalised engagement rate — there is no impressions or views signal in a profile payload the way the `Post` archetype has `views`. An earlier formula (`followers * engagement_rate * 0.1`) produced numbers that could fall *below the like count of a single post* on large accounts — a fabrication, not an estimate — so it was removed. The field is kept in the block for shape stability. Post variant [#post-variant] ``` estimated_reach = round(views * 1.2) ``` For posts, views is already a stronger reach signal than impressions, so we apply a modest multiplier to estimate unique-account reach (assuming a small fraction of repeat views). Returns `null` whenever `views` is missing or `0`. **`views` is the only gate** — so whether reach populates on a given post is entirely a question of whether the platform exposes a view count for that media type: * **Instagram**: reels and videos carry a play count, so they get a reach estimate. **Photo posts and photo-only carousels have no view count on Instagram, so `estimated_reach` (and `engagement_rate`) are always `null` on them** — while `likes` and `comments` still populate normally. A carousel that contains a video child does carry a play count and does get a reach estimate. * **Twitter/X**: tweets without a native view count (older tweets, many replies) return `null`. * **Reddit, Threads list items**: no per-post views, so always `null` there. Caveats [#caveats] This is a heuristic, not a measurement. It's useful for: * **Sorting** posts or creators by approximate reach when the platform doesn't expose reach directly. * **Estimating** order-of-magnitude impact for influencer outreach. It is **not** useful for: * Forecasting paid-media ROI. * Comparing reach across platforms with very different view-counting rules (TikTok's auto-loop views vs YouTube's 30-second threshold, for example). *** Example response [#example-response] ```json { "success": true, "platform": "tiktok", "endpoint": "/v1/tiktok/profile", "data": { "author": { "username": "mrbeast", "followers": 95000000, "likes_count": 8200000000, "bio": "I want to make the world a better place before I die." }, "computed": { "engagement_rate": null, "language": "en", "content_category": "other", "estimated_reach": null }, "_warnings": [ "computed.engagement_rate: author ratio exceeded 1.0 (raw: 86.315789); returned null — a lifetime likes/followers ratio is not a real engagement rate" ] }, "credits_used": 1, "credits_remaining": 8431 } ``` Three things to read off that response. The `likes_count / followers` ratio blows past `1.0` because TikTok reports cumulative lifetime hearts against current followers — not a real engagement rate, so the author path reports `null` with the warning rather than fabricating `1.0`. `estimated_reach` is `null` on the `Author` archetype, as above. And `content_category` is `"other"` because that bio, despite reading as broadly lifestyle-ish, matches no category keyword twice — see the two-hit rule. When you shouldn't trust the value [#when-you-shouldnt-trust-the-value] Three patterns mean "look at the warnings before using this number": 1. **`_warnings` mentions `clamped`** (post rows) **or `returned null`** (profile rows) — the raw value blew through `[0, 1]`. Decide whether you want our value or to recompute from the raw upstream fields yourself. 2. **`computed.engagement_rate` is `null` on a `Post` archetype** — the post lacks a `views` field. Instagram photo posts, pre-2020 tweets, some Reddit endpoints, and a few Facebook surfaces are common offenders. On Instagram this is per-media-type: a reel gets a rate and a reach estimate, the photo posted an hour later by the same account gets `null` for both. 3. **`computed.language` is `null` on a clearly-textual bio** — on Latin scripts, what's left after stripping URLs, handles, and hashtags was probably under 30 characters or under 3 words, or the bio reads as a title rather than prose. Read `author.bio` directly to confirm. This does not apply to Korean, Japanese, Chinese, Arabic, Devanagari, or Thai bios, which classify at any length. See also [#see-also] * [Response schema → Successful response](/docs/response-schema.md#successful-response) — where `computed` sits in the response. * [Response schema → `data._warnings[]`](/docs/response-schema.md#data_warnings--partial-data-channel) — how warning strings are surfaced. * [Endpoint pricing](/docs/endpoint-pricing.md) — every endpoint and what each call costs.