# YouTube Comment Scraper: 100 Per Page in Python (https://www.socialcrawl.dev/blog/youtube-comment-scraper)
> A YouTube comment scraper that pages 100 comments per call. Curl and Python pagination, a full replies walk, and the quota table ranking pages still get wrong.
A YouTube comment scraper is a script that pages a comments endpoint until the public thread is exhausted. It is not a Chrome extension, and it is not a one-shot `maxResults=100` dump that pretends the rest of the thread does not exist.
You leave with the same first page in curl and Python, a `continuationToken` loop with a `max_pages` guard, and a replies walker that pages past the truncated `replies` object. After that: a quota table that reconciles the "1 million comments/day" claim against the "3,000–10,000/day" claim, and a ToS paragraph sitting next to the working code.
**Stack:** Python 3.10+ · `requests` · curl · [`GET /v1/youtube/video/comments`](/platforms/youtube/video-comments) + `GET /v1/youtube/video/comment/replies` · `x-api-key` (100 welcome credits, no card). No GCP project, no OAuth consent screen, no YouTube Data API v3 library.
---
## What do you need to scrape YouTube comments?
Python 3.10+, curl, and one API key. The hook is what you do **not** need: a Google Cloud project, an OAuth consent screen, or the YouTube Data API v3 client library. Public comments are a read of a public thread. The official path still makes you enable an API and mint a key inside GCP. This path does not.
- Python 3.10+ and `pip install requests`. `pandas` is optional; Step 4 uses the `csv` stdlib so the export runs with zero extra packages.
- curl. Built in on macOS and Linux. On Windows, `curl.exe`.
- [One API key](/docs/quickstart) from signup. 100 welcome credits, no card. Keys look like `sc_` plus random bytes and are shown in full once.
```bash
```
- A **public** video URL: `youtube.com/watch?v=`, `youtu.be`, or Shorts. Comments-off, private, and made-for-kids videos fail later; the error matrix at the bottom is the skip list, not a bypass list.
---
## How do I fetch the first page of YouTube comments?
Call `GET /v1/youtube/video/comments` with the watch URL and `x-api-key`. Default page is already 100. Most tutorials show either a Python `commentThreads` snippet or a vendor curl; almost none print the **same** first page both ways. Here is that page, then the official `googleapis.com` call it replaces.
Yes, the sample is `dQw4w9WgXcQ`. Every comments tutorial uses it. This one does too, so the first page you get matches the curl you just copied.
```bash
curl "https://www.socialcrawl.dev/v1/youtube/video/comments?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ&order=top" \
-H "x-api-key: YOUR_API_KEY"
```
Same request in Python. `timeout=30` because a hung socket is not a successful empty page, and `success: false` is a real failure even when HTTP is 200.
```python
API_KEY = os.environ["SOCIALCRAWL_API_KEY"]
BASE = "https://www.socialcrawl.dev/v1"
VIDEO_URL = "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
resp = requests.get(
f"{BASE}/youtube/video/comments",
params={
"url": VIDEO_URL,
"order": "top",
"format": "plainText", # default is html; CSV wants text
},
headers={"x-api-key": API_KEY},
timeout=30,
)
payload = resp.json()
if not payload.get("success"):
raise RuntimeError(
f"{payload['error']['type']}: {payload['error']['message']} "
f"(request_id={payload['request_id']})"
)
# CommentList items are { "comment": {...}, "computed": {...} }, not a flat row.
comments = [item["comment"] for item in payload["data"]["items"]]
print(f"page size: {len(comments)}")
print(comments[0]["text"][:120])
```
The official counterpart, for contrast. Default `maxResults` on this method is **20**, not 100. You have to ask for 100. You also need a GCP key, and the call costs 1 quota unit ([`commentThreads.list`](https://developers.google.com/youtube/v3/docs/commentThreads/list)).
```bash
curl "https://www.googleapis.com/youtube/v3/commentThreads?part=snippet&videoId=dQw4w9WgXcQ&maxResults=100&key=YOUR_GOOGLE_KEY"
```
The parts that are easy to get wrong:
- **`url` is required.** Full watch URL, Shorts URL, or `youtu.be`. The official method wants an 11-character `videoId`; this one wants the URL you actually have.
- **`order=top|newest`.** `top` is official `relevance`. `newest` is exact newest-first on `published_at`, and later pages continue strictly older with no overlap. That is the lane for "last N days." There is no server-side date parameter on either API ([`commentThreads.list` filters](https://developers.google.com/youtube/v3/docs/commentThreads/list)).
- **`format`.** Default is `html`, matching official `textFormat=html`. Pass `plainText` when you are about to write a CSV. Out-of-range `max_results` (0, or 101+) is a free `400`, not a silently smaller page.
- **`searchTerm`** filters the thread to comments containing that string. **`channel_id`** fetches that channel's community comments. It is not "every comment on every video this channel uploaded."
- **Cost:** 1 credit, 300s cache. Cache hits are 0 credits. Default page is already **100**.
- **Fields on each comment:** `id`, `text`, `author.username` / `display_name` / `avatar_url`, `engagement.likes`, `engagement.replies`, `published_at`, `flags.pinned`, `ext.replies_token`, `ext.author_channel_id`.
- **Pagination:** send `continuationToken` (the universal alias is `cursor`). Read `pagination.next_cursor` and `pagination.has_more` on the envelope ([pagination contract](/docs/pagination)). A `data.next_cursor` string may also appear; do not loop on it instead of the pagination block, and do not loop on `engagement.replies` or a `total` estimate.
Trimmed envelope, so you know what you are parsing:
```json
{
"success": true,
"platform": "youtube",
"endpoint": "/v1/youtube/video/comments",
"data": {
"items": [
{
"comment": {
"id": "UgzdQw4w9WgXcQ000000000000000000",
"parent_id": null,
"text": "first comment on the thread",
"author": {
"username": "some_handle",
"display_name": "Some Handle",
"avatar_url": "https://yt3.ggpht.com/..."
},
"engagement": { "likes": 412, "replies": 18 },
"flags": { "pinned": null, "deleted": false },
"published_at": "2024-11-02T14:08:11Z",
"ext": { "replies_token": "Eg0SC2RRdzR3OVdnWGNRGAYy...", "author_channel_id": "UC..." }
}
}
]
},
"credits_used": 1,
"cached": false,
"pagination": {
"next_cursor": "sc.eyJ2IjoyLCJjIjoiRWcwU0My...",
"has_more": true,
"page_size": 100
}
}
```
Full field notes live in the [YouTube endpoint docs](/docs/youtube). Working-example page: [`GET /v1/youtube/video/comments`](/platforms/youtube/video-comments).
---
## How do I walk continuationToken until the thread ends?
Official pagination is `pageToken` in, `nextPageToken` out, default 20 per page, 1 unit per call ([`commentThreads.list`](https://developers.google.com/youtube/v3/docs/commentThreads/list)). Ours is `continuationToken` in, `pagination.next_cursor` out, default 100 per page, 1 credit per call (cache hit 0). Same idea. Different meter.
Guard `max_pages`. A viral thread will happily drain a key if you write `while True`.
```python
def scrape_youtube_comments(
video_url: str,
*,
order: str = "newest",
max_pages: int = 50,
) -> list[dict]:
"""Page a public YouTube comment thread. One credit per page, 0 on cache hit."""
comments: list[dict] = []
token: str | None = None
for _ in range(max_pages):
params: dict[str, str | int] = {
"url": video_url,
"order": order,
"format": "plainText",
"max_results": 100,
}
if token:
# YouTube-native name. The universal alias is `cursor`.
params["continuationToken"] = token
resp = requests.get(
f"{BASE}/youtube/video/comments",
params=params,
headers={"x-api-key": API_KEY},
timeout=30,
)
payload = resp.json()
if not payload.get("success"):
raise RuntimeError(
f"{payload['error']['type']}: {payload['error']['message']} "
f"(request_id={payload['request_id']})"
)
comments.extend(item["comment"] for item in payload["data"]["items"])
pag = payload.get("pagination") or {}
token = pag.get("next_cursor")
if not pag.get("has_more") or not token:
break
return comments
comments = scrape_youtube_comments(VIDEO_URL)
print(f"fetched {len(comments)} top-level comments")
```
Loop on `pagination.has_more` / `next_cursor`, never on `engagement.replies` and never on a `total`. Parent reply counts are YouTube's own tally and can include held-for-review, spam-filtered, and deleted rows that will never appear in the public list ([YouTube held-for-review](https://support.google.com/youtube/answer/9483359)).
`order=newest` is the date-window lane. Pages are strictly older, no overlap. A pinned comment can still sit first even on newest ([YouTube comments docs](/docs/youtube)). Filter by `published_at`. Do not stop at the first out-of-window row; stop when a page's oldest **in-window** row is before the cutoff, after you have skipped the pin.
---
## How do I page every reply, not just the first 100?
Official `commentThreads` "does not necessarily contain all replies to a comment, and you need to use the `comments.list` method if you want to retrieve all replies for a particular comment" ([commentThreads resource](https://developers.google.com/youtube/v3/docs/commentThreads)). The tree is two levels: a top-level comment and a flat reply list. There is no reply-to-reply thread. Later answers sit in the same list, usually with an @mention.
The truncated `replies` object on a `commentThreads` resource is a short prefix of the thread, not the full list. Helpers that call `comments.list?parentId=` once at `maxResults=100` and stop are the same bug with a bigger page size. A full walk pages `comments.list` until `nextPageToken` dies. Same here: page `GET /v1/youtube/video/comment/replies` until `has_more` is false.
The replies endpoint takes a **token**, not a comment id. That token is `ext.replies_token` on the parent. 1 credit, 300s cache. A parent with zero replies returns an **empty list, not a 404**.
```python
def fetch_all_replies(replies_token: str, *, max_pages: int = 20) -> list[dict]:
"""Drain one reply thread. Stop on has_more, not parent engagement.replies."""
replies: list[dict] = []
token: str | None = replies_token
for _ in range(max_pages):
resp = requests.get(
f"{BASE}/youtube/video/comment/replies",
params={"continuationToken": token, "format": "plainText"},
headers={"x-api-key": API_KEY},
timeout=30,
)
payload = resp.json()
if not payload.get("success"):
raise RuntimeError(
f"{payload['error']['type']}: {payload['error']['message']} "
f"(request_id={payload['request_id']})"
)
replies.extend(item["comment"] for item in payload["data"]["items"])
pag = payload.get("pagination") or {}
token = pag.get("next_cursor")
if not pag.get("has_more") or not token:
break
return replies
def scrape_comments_and_replies(video_url: str, *, max_pages: int = 50) -> list[dict]:
rows = scrape_youtube_comments(video_url, max_pages=max_pages)
out: list[dict] = []
for comment in rows:
out.append(comment)
token = (comment.get("ext") or {}).get("replies_token")
if not token:
continue
for reply in fetch_all_replies(token):
if reply.get("parent_id") is None:
reply["parent_id"] = comment["id"]
out.append(reply)
return out
thread = scrape_comments_and_replies(VIDEO_URL)
print(f"top-level + replies: {len(thread)}")
```
Keep `parent_id` so the CSV can rebuild chains. Page replies on `pagination.has_more`, not the parent's `engagement.replies`. That tally can exceed the public list.
---
## How do I extract YouTube comments and replies to CSV?
Once the list is flat, write it with the `csv` stdlib and utf-8-sig so Excel does not mangle emoji. That is the download: a file you can sort, not a UI dump.
```python
COLUMNS = [
"id",
"parent_id",
"author",
"text",
"likes",
"reply_count",
"published_at",
"pinned",
"video_url",
]
def comments_to_csv(rows: list[dict], video_url: str, path: str = "youtube_comments.csv") -> None:
flattened = []
for c in rows:
if c.get("text") is None:
continue # deleted rows arrive as text: null; drop before sentiment
flattened.append(
{
"id": c.get("id"),
"parent_id": c.get("parent_id"),
"author": (c.get("author") or {}).get("username")
or (c.get("author") or {}).get("display_name"),
"text": c.get("text"),
"likes": (c.get("engagement") or {}).get("likes") or 0,
"reply_count": (c.get("engagement") or {}).get("replies") or 0,
"published_at": c.get("published_at"),
"pinned": (c.get("flags") or {}).get("pinned") or False,
"video_url": video_url,
}
)
flattened.sort(key=lambda r: r["likes"], reverse=True)
with open(path, "w", newline="", encoding="utf-8-sig") as f:
writer = csv.DictWriter(f, fieldnames=COLUMNS)
writer.writeheader()
writer.writerows(flattened)
comments_to_csv(thread, VIDEO_URL)
```
Sort-by-likes is a practical cut: the highest-liked rows are the ones people actually engaged with. If the next step is classification, [sentiment analysis tools](/blog/sentiment-analysis-tools) is the follow-up, not another comments endpoint.
---
## What's the difference between a YouTube comment scraper and the comments API?
"YouTube comments API" usually means YouTube Data API v3 `commentThreads.list` / `comments.list`. A comments scraper, in practice, often means an unofficial InnerTube client. Those are not the same contract, and they are not the same ToS posture. The [YouTube platform hub](/platforms/youtube) lists the rest of the comments surface. This table is the decision matrix.
| Path | Auth | Default page | Pagination token | Replies | Daily ceiling | ToS posture |
| --- | --- | --- | --- | --- | --- | --- |
| YouTube Data API v3 `commentThreads.list` / `comments.list` | API key for public reads; OAuth for `moderationStatus` | 20 (max 100) | `pageToken` / `nextPageToken` | `comments.list?parentId=` | 10,000 units/day, midnight Pacific | Official / ToS-clean |
| InnerTube / [youtube-comment-downloader](https://github.com/egbertbouman/youtube-comment-downloader) / `yt-dlp --write-comments` | No Google key | Iterator / `--limit` | Unofficial continuation | README is silent on a replies walk | Whatever YouTube's bot defenses allow | Against the automated-access clause |
| `GET /v1/youtube/video/comments` + `/comment/replies` | `x-api-key` | 100 | `continuationToken` / `pagination.next_cursor` | Dedicated replies walk | 1 credit/page, 600 req/min, 300s cache | Documented API path, credits, not the Data API |
The unified schema is the reason the CSV columns above do not change if you later point the same flatten at TikTok or Reddit comments.
### Quota math the SERP still gets wrong
Documented costs: `commentThreads.list` = **1 unit**, `comments.list` = **1 unit**, default **10,000 units/day** ([quota cost table](https://developers.google.com/youtube/v3/determine_quota_cost), [getting started](https://developers.google.com/youtube/v3/getting-started)).
Worked example, `part=snippet`, 100 comments/page: 10,000 units ÷ 1 unit/call = **10,000 pages/day** of top-level comments = a theoretical **1,000,000 top-level comments/day**. That is a ceiling on `commentThreads.list` **before replies**. It is also the figure behind the "~1 million comments/day" line you will see on comparison posts.
A popular how-to that prices `commentThreads` at "1–3 units" and concludes "roughly 3,000–10,000 comments per day" does **not** match the published 1-unit cost for `commentThreads.list`. Do not budget from that range.
Add replies and the arithmetic moves. Each `comments.list` page is another 1 unit. A 10,000-comment video with 2,000 replies is about 100 top-level pages + 20 reply pages = **120 units**, not 100. `part=replies` does not replace `comments.list` for a full walk (official quote above).
Quota increases go through YouTube's [compliance/quota audit](https://developers.google.com/youtube/v3/guides/quota_and_compliance_audits), not a self-serve slider. There is no "buy more units" button. That is a real reason teams search for a comments scraper. `search.list` is a separate daily cap from comment reads; that story is [YouTube API quota](/blog/youtube-data-api-2026), not this page.
SocialCrawl: 1 credit per page of up to 100, 100 welcome credits ≈ 100 pages before paying, cache hits free, no midnight-Pacific reset. 600 requests/minute per key; the 429 is unbilled ([error handling](/docs/errors)).
---
## How does commentThreads pagination actually work?
BeautifulSoup on the watch page fails because comments load via AJAX. The HTML you download is a shell; the thread arrives later from YouTube's InnerTube continuation endpoints. That is why a `requests.get` of the watch URL plus a CSS selector returns an empty list, and why unofficial libraries look like scrapers but are really continuation clients.
The official path is quota-gated and ToS-clean: `pageToken` in, `nextPageToken` out, 1 unit per call, 10,000 units/day. Unofficial continuation is what most "scraper" GitHub READMEs actually hit; they rarely name InnerTube. Three token systems, side by side:
```
Official Data API Unofficial InnerTube SocialCrawl
commentThreads.list youtube-comment-downloader GET /v1/youtube/video/comments
pageToken → nextPageToken continuation (AJAX/InnerTube) continuationToken
1 unit / call, 10k/day no Google key, ToS risk 1 credit / call, 300s cache
maxResults default 20 iterator / --limit max_results default 100
comments.list?parentId= (README is silent on replies) GET /v1/.../comment/replies
order=time|relevance sort 0 popular / 1 recent order=newest|top
```
Sort maps `time|relevance` ↔ `newest|top`. The working pagination loop is in "How do I walk continuationToken until the thread ends?" above. This section is the map, not a second `while`.
---
## Is it legal to scrape YouTube comments?
Two facts, both true at once.
1. YouTube's Terms prohibit accessing the service by automated means (robots, botnets, scrapers) except public search engines honoring `robots.txt`, or with YouTube's prior written permission ([YouTube Terms of Service](https://www.youtube.com/t/terms)).
2. The permitted programmatic path is YouTube API Services, governed by the [API Services ToS](https://developers.google.com/youtube/terms/api-services-terms-of-service) and [Developer Policies](https://developers.google.com/youtube/terms/developer-policies).
`hiQ Labs v. LinkedIn` is the case every optimistic how-to cites. The Ninth Circuit in 2022 narrowed the CFAA for *public* data. Later that year hiQ still lost on the contract claim and entered a stipulated judgment. Do not ship the one-liner "scraping publicly accessible data generally does not violate the CFAA" without that second half. The longer read is the [social media scraping legal primer](/blog/social-media-scraping-legal-technical-guide). Technique context: [web scraping](/blog/web-scraping). What we publish about public data: the [public data notice](/legal/public-data-notice).
Enforcement in practice is rate-limits, IP blocks, and quota revocation. A quiet docket is not permission.
The working code on this page is the documented API path (key, credits, two endpoints). It is not an InnerTube recipe, and it is not the official YouTube Data API. This is not legal advice.
---
## What returns empty: commentsDisabled, kids, Shorts, dash-prefix IDs
| Symptom | Official Data API | What to do here |
| --- | --- | --- |
| Comments turned off | 403 `commentsDisabled` | Expect empty or an error; skip the video. Do not invent a JSON body. |
| Made-for-kids / held-for-review | Public list omits them; `moderationStatus=heldForReview` needs OAuth + `allThreadsRelatedToChannelId` | A public key cannot see held-for-review |
| Private / age-restricted | 403 / empty | Skip. Do not bypass. |
| Unlisted but comments on | Works with the `videoId` if you have the URL | Same: URL in, public comments out |
| Shorts / `youtu.be` | Extract the 11-character `videoId` | Pass the full URL to `url=` |
| Dash-prefix video ids | Fine as `videoId` | GitHub CLIs need `-y=id`; `url=` is fine if the URL is quoted |
| Displayed count > export | Deleted/spam included in the public number | Page until the token dies; parent reply count can exceed the public list |
| `max_results` 0 or 101 | n/a | Free `400`, no credit |
| Zero-reply parent | n/a | Replies endpoint → empty list, not 404 |
| 600 req/min | n/a | Unbilled `429 RATE_LIMITED` |
Official `quotaExceeded` is a 403 on the Data API and resets at midnight Pacific. The body and the reset math are in [YouTube API quota](/blog/youtube-data-api-2026); this page does not re-teach that clock.
A 502 or 503 (`UPSTREAM_ERROR`, `SERVICE_UNAVAILABLE`) refunds the credit. Retry the same token; do not advance past a page you did not read ([errors](/docs/errors)).
---
## Where does a comment scraper go next?
One video is the unit. A channel harvest is "list that channel's videos, then this comments loop per video" — point the [YouTube platform hub](/platforms/youtube) or the visual explorer at the channel, then reuse the functions above. This post does not add a third endpoint.
If the CSV is a monitoring input, the [social media monitoring API](/blog/social-media-monitoring-api) is the pipeline shape. The sibling how-to is [scrape TikTok comments with Python](/blog/how-to-scrape-tiktok-comments-python). Endpoint contract: [`GET /v1/youtube/video/comments`](/platforms/youtube/video-comments).
---
## Frequently asked questions
### How do I scrape all YouTube comments from a video?
Page `GET /v1/youtube/video/comments` until `pagination.has_more` is false, with `max_pages` set so a viral thread cannot drain the key. That walk covers public top-level comments. Held-for-review, likely-spam, and rejected comments are invisible without owner OAuth on the Data API. Replies are a second endpoint, not extra rows on page one.
### How do I get YouTube comment replies, not just top-level comments?
Take `ext.replies_token` from each parent and call `GET /v1/youtube/video/comment/replies` with that value as `continuationToken`. Page until `has_more` is false. Official equivalent: `comments.list?parentId=` until `nextPageToken` dies. The tree is two levels; @mentions in the flat list are how later answers attach.
### Can I export YouTube comments to CSV or JSON?
Yes. The Python above writes utf-8-sig CSV with `id`, `parent_id`, `author`, `text`, `likes`, `reply_count`, `published_at`, `pinned`, `video_url`. For JSON, `json.dump` the `items` list as-is. Filter `text is None` before sentiment; deleted rows arrive as null, not a `"[deleted]"` string.
### Does the YouTube Data API let me download comments without a quota?
No. Default allowance is 10,000 units/day for comment reads and the rest of the non-search surface. `commentThreads.list` and `comments.list` each cost 1 unit ([quota table](https://developers.google.com/youtube/v3/determine_quota_cost), [getting started](https://developers.google.com/youtube/v3/getting-started)). There is no self-serve top-up; extra quota goes through an [audit form](https://developers.google.com/youtube/v3/guides/quota_and_compliance_audits). Search quota is a different pain: [YouTube API quota](/blog/youtube-data-api-2026).
### How do I paginate YouTube comments after the first page?
Official: read `nextPageToken`, send it back as `pageToken`. Here: read `pagination.next_cursor`, send it back as `continuationToken` (or the universal `cursor`), stop when `has_more` is false. Do not build the token, do not decode it, do not loop on `total`.
### Is a YouTube comment downloader the same as a scraper API?
No. A YouTube comment downloader (or comment extractor) is usually a Chrome extension or a GitHub CLI (`youtube-comment-downloader`, `yt-dlp --write-comments`) that walks unofficial InnerTube continuations. A scraper API is a documented endpoint, a key, and a pagination contract. The working code on this page is the API path, not an extension.
### Can I filter YouTube comments by keyword or date?
Keyword: `searchTerm` here, `searchTerms` on `commentThreads.list`. Date: **no server-side date parameter** on either API. Client-side: `order=newest`, keep in-window rows, stop when a page's oldest in-window row is before the cutoff. Do not stop at a pinned row; pins can sit first on newest.