YouTube Comment Scraper: 100 Per Page in Python

A YouTube comment scraper is a script that pages a comments endpoint until the public thread is exhausted. It is not a Chrome extension, and it is not a one-shot maxResults=100 dump that pretends the rest of the thread does not exist.

You leave with the same first page in curl and Python, a continuationToken loop with a max_pages guard, and a replies walker that pages past the truncated replies object. After that: a quota table that reconciles the "1 million comments/day" claim against the "3,000–10,000/day" claim, and a ToS paragraph sitting next to the working code.

Stack: Python 3.10+ · requests · curl · GET /v1/youtube/video/comments + GET /v1/youtube/video/comment/replies · x-api-key (100 welcome credits, no card). No GCP project, no OAuth consent screen, no YouTube Data API v3 library.

---

What do you need to scrape YouTube comments?

Python 3.10+, curl, and one API key. The hook is what you do not need: a Google Cloud project, an OAuth consent screen, or the YouTube Data API v3 client library. Public comments are a read of a public thread. The official path still makes you enable an API and mint a key inside GCP. This path does not.

• Python 3.10+ and pip install requests. pandas is optional; Step 4 uses the csv stdlib so the export runs with zero extra packages.
• curl. Built in on macOS and Linux. On Windows, curl.exe.
• One API key from signup. 100 welcome credits, no card. Keys look like sc_ plus random bytes and are shown in full once.

export SOCIALCRAWL_API_KEY=sc_your_api_key_here

• A public video URL: youtube.com/watch?v=, youtu.be, or Shorts. Comments-off, private, and made-for-kids videos fail later; the error matrix at the bottom is the skip list, not a bypass list.

---

How do I fetch the first page of YouTube comments?

Call GET /v1/youtube/video/comments with the watch URL and x-api-key. Default page is already 100. Most tutorials show either a Python commentThreads snippet or a vendor curl; almost none print the same first page both ways. Here is that page, then the official googleapis.com call it replaces.

Yes, the sample is dQw4w9WgXcQ. Every comments tutorial uses it. This one does too, so the first page you get matches the curl you just copied.

curl "https://www.socialcrawl.dev/v1/youtube/video/comments?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ&order=top" \
  -H "x-api-key: YOUR_API_KEY"

Same request in Python. timeout=30 because a hung socket is not a successful empty page, and success: false is a real failure even when HTTP is 200.

import os
import requests

API_KEY = os.environ["SOCIALCRAWL_API_KEY"]
BASE = "https://www.socialcrawl.dev/v1"
VIDEO_URL = "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

resp = requests.get(
    f"{BASE}/youtube/video/comments",
    params={
        "url": VIDEO_URL,
        "order": "top",
        "format": "plainText",  # default is html; CSV wants text
    },
    headers={"x-api-key": API_KEY},
    timeout=30,
)
payload = resp.json()
if not payload.get("success"):
    raise RuntimeError(
        f"{payload['error']['type']}: {payload['error']['message']} "
        f"(request_id={payload['request_id']})"
    )

# CommentList items are { "comment": {...}, "computed": {...} }, not a flat row.
comments = [item["comment"] for item in payload["data"]["items"]]
print(f"page size: {len(comments)}")
print(comments[0]["text"][:120])

The official counterpart, for contrast. Default maxResults on this method is 20, not 100. You have to ask for 100. You also need a GCP key, and the call costs 1 quota unit (commentThreads.list).

curl "https://www.googleapis.com/youtube/v3/commentThreads?part=snippet&videoId=dQw4w9WgXcQ&maxResults=100&key=YOUR_GOOGLE_KEY"

The parts that are easy to get wrong:

• url is required. Full watch URL, Shorts URL, or youtu.be. The official method wants an 11-character videoId; this one wants the URL you actually have.
• order=top|newest. top is official relevance. newest is exact newest-first on published_at, and later pages continue strictly older with no overlap. That is the lane for "last N days." There is no server-side date parameter on either API (commentThreads.list filters).
• format. Default is html, matching official textFormat=html. Pass plainText when you are about to write a CSV. Out-of-range max_results (0, or 101+) is a free 400, not a silently smaller page.
• searchTerm filters the thread to comments containing that string. channel_id fetches that channel's community comments. It is not "every comment on every video this channel uploaded."
• Cost: 1 credit, 300s cache. Cache hits are 0 credits. Default page is already 100.
• Fields on each comment: id, text, author.username / display_name / avatar_url, engagement.likes, engagement.replies, published_at, flags.pinned, ext.replies_token, ext.author_channel_id.
• Pagination: send continuationToken (the universal alias is cursor). Read pagination.next_cursor and pagination.has_more on the envelope (pagination contract). A data.next_cursor string may also appear; do not loop on it instead of the pagination block, and do not loop on engagement.replies or a total estimate.

Trimmed envelope, so you know what you are parsing:

{
  "success": true,
  "platform": "youtube",
  "endpoint": "/v1/youtube/video/comments",
  "data": {
    "items": [
      {
        "comment": {
          "id": "UgzdQw4w9WgXcQ000000000000000000",
          "parent_id": null,
          "text": "first comment on the thread",
          "author": {
            "username": "some_handle",
            "display_name": "Some Handle",
            "avatar_url": "https://yt3.ggpht.com/..."
          },
          "engagement": { "likes": 412, "replies": 18 },
          "flags": { "pinned": null, "deleted": false },
          "published_at": "2024-11-02T14:08:11Z",
          "ext": { "replies_token": "Eg0SC2RRdzR3OVdnWGNRGAYy...", "author_channel_id": "UC..." }
        }
      }
    ]
  },
  "credits_used": 1,
  "cached": false,
  "pagination": {
    "next_cursor": "sc.eyJ2IjoyLCJjIjoiRWcwU0My...",
    "has_more": true,
    "page_size": 100
  }
}

Full field notes live in the YouTube endpoint docs. Working-example page: GET /v1/youtube/video/comments.

---

How do I walk continuationToken until the thread ends?

Official pagination is pageToken in, nextPageToken out, default 20 per page, 1 unit per call (commentThreads.list). Ours is continuationToken in, pagination.next_cursor out, default 100 per page, 1 credit per call (cache hit 0). Same idea. Different meter.

Guard max_pages. A viral thread will happily drain a key if you write while True.

def scrape_youtube_comments(
    video_url: str,
    *,
    order: str = "newest",
    max_pages: int = 50,
) -> list[dict]:
    """Page a public YouTube comment thread. One credit per page, 0 on cache hit."""
    comments: list[dict] = []
    token: str | None = None

    for _ in range(max_pages):
        params: dict[str, str | int] = {
            "url": video_url,
            "order": order,
            "format": "plainText",
            "max_results": 100,
        }
        if token:
            # YouTube-native name. The universal alias is `cursor`.
            params["continuationToken"] = token

        resp = requests.get(
            f"{BASE}/youtube/video/comments",
            params=params,
            headers={"x-api-key": API_KEY},
            timeout=30,
        )
        payload = resp.json()
        if not payload.get("success"):
            raise RuntimeError(
                f"{payload['error']['type']}: {payload['error']['message']} "
                f"(request_id={payload['request_id']})"
            )

        comments.extend(item["comment"] for item in payload["data"]["items"])

        pag = payload.get("pagination") or {}
        token = pag.get("next_cursor")
        if not pag.get("has_more") or not token:
            break

    return comments


comments = scrape_youtube_comments(VIDEO_URL)
print(f"fetched {len(comments)} top-level comments")

Loop on pagination.has_more / next_cursor, never on engagement.replies and never on a total. Parent reply counts are YouTube's own tally and can include held-for-review, spam-filtered, and deleted rows that will never appear in the public list (YouTube held-for-review).

order=newest is the date-window lane. Pages are strictly older, no overlap. A pinned comment can still sit first even on newest (YouTube comments docs). Filter by published_at. Do not stop at the first out-of-window row; stop when a page's oldest in-window row is before the cutoff, after you have skipped the pin.

---

How do I page every reply, not just the first 100?

Official commentThreads "does not necessarily contain all replies to a comment, and you need to use the comments.list method if you want to retrieve all replies for a particular comment" (commentThreads resource). The tree is two levels: a top-level comment and a flat reply list. There is no reply-to-reply thread. Later answers sit in the same list, usually with an @mention.

The truncated replies object on a commentThreads resource is a short prefix of the thread, not the full list. Helpers that call comments.list?parentId= once at maxResults=100 and stop are the same bug with a bigger page size. A full walk pages comments.list until nextPageToken dies. Same here: page GET /v1/youtube/video/comment/replies until has_more is false.

The replies endpoint takes a token, not a comment id. That token is ext.replies_token on the parent. 1 credit, 300s cache. A parent with zero replies returns an empty list, not a 404.

def fetch_all_replies(replies_token: str, *, max_pages: int = 20) -> list[dict]:
    """Drain one reply thread. Stop on has_more, not parent engagement.replies."""
    replies: list[dict] = []
    token: str | None = replies_token

    for _ in range(max_pages):
        resp = requests.get(
            f"{BASE}/youtube/video/comment/replies",
            params={"continuationToken": token, "format": "plainText"},
            headers={"x-api-key": API_KEY},
            timeout=30,
        )
        payload = resp.json()
        if not payload.get("success"):
            raise RuntimeError(
                f"{payload['error']['type']}: {payload['error']['message']} "
                f"(request_id={payload['request_id']})"
            )

        replies.extend(item["comment"] for item in payload["data"]["items"])

        pag = payload.get("pagination") or {}
        token = pag.get("next_cursor")
        if not pag.get("has_more") or not token:
            break

    return replies


def scrape_comments_and_replies(video_url: str, *, max_pages: int = 50) -> list[dict]:
    rows = scrape_youtube_comments(video_url, max_pages=max_pages)
    out: list[dict] = []
    for comment in rows:
        out.append(comment)
        token = (comment.get("ext") or {}).get("replies_token")
        if not token:
            continue
        for reply in fetch_all_replies(token):
            if reply.get("parent_id") is None:
                reply["parent_id"] = comment["id"]
            out.append(reply)
    return out


thread = scrape_comments_and_replies(VIDEO_URL)
print(f"top-level + replies: {len(thread)}")

Keep parent_id so the CSV can rebuild chains. Page replies on pagination.has_more, not the parent's engagement.replies. That tally can exceed the public list.

---

How do I extract YouTube comments and replies to CSV?

Once the list is flat, write it with the csv stdlib and utf-8-sig so Excel does not mangle emoji. That is the download: a file you can sort, not a UI dump.

import csv

COLUMNS = [
    "id",
    "parent_id",
    "author",
    "text",
    "likes",
    "reply_count",
    "published_at",
    "pinned",
    "video_url",
]


def comments_to_csv(rows: list[dict], video_url: str, path: str = "youtube_comments.csv") -> None:
    flattened = []
    for c in rows:
        if c.get("text") is None:
            continue  # deleted rows arrive as text: null; drop before sentiment
        flattened.append(
            {
                "id": c.get("id"),
                "parent_id": c.get("parent_id"),
                "author": (c.get("author") or {}).get("username")
                or (c.get("author") or {}).get("display_name"),
                "text": c.get("text"),
                "likes": (c.get("engagement") or {}).get("likes") or 0,
                "reply_count": (c.get("engagement") or {}).get("replies") or 0,
                "published_at": c.get("published_at"),
                "pinned": (c.get("flags") or {}).get("pinned") or False,
                "video_url": video_url,
            }
        )
    flattened.sort(key=lambda r: r["likes"], reverse=True)

    with open(path, "w", newline="", encoding="utf-8-sig") as f:
        writer = csv.DictWriter(f, fieldnames=COLUMNS)
        writer.writeheader()
        writer.writerows(flattened)


comments_to_csv(thread, VIDEO_URL)

Sort-by-likes is a practical cut: the highest-liked rows are the ones people actually engaged with. If the next step is classification, sentiment analysis tools is the follow-up, not another comments endpoint.

---

What's the difference between a YouTube comment scraper and the comments API?

"YouTube comments API" usually means YouTube Data API v3 commentThreads.list / comments.list. A comments scraper, in practice, often means an unofficial InnerTube client. Those are not the same contract, and they are not the same ToS posture. The YouTube platform hub lists the rest of the comments surface. This table is the decision matrix.

Path	Auth	Default page	Pagination token	Replies	Daily ceiling	ToS posture
YouTube Data API v3 commentThreads.list / comments.list	API key for public reads; OAuth for moderationStatus	20 (max 100)	pageToken / nextPageToken	comments.list?parentId=	10,000 units/day, midnight Pacific	Official / ToS-clean
InnerTube / youtube-comment-downloader / yt-dlp --write-comments	No Google key	Iterator / --limit	Unofficial continuation	README is silent on a replies walk	Whatever YouTube's bot defenses allow	Against the automated-access clause
GET /v1/youtube/video/comments + /comment/replies	x-api-key	100	continuationToken / pagination.next_cursor	Dedicated replies walk	1 credit/page, 600 req/min, 300s cache	Documented API path, credits, not the Data API

The unified schema is the reason the CSV columns above do not change if you later point the same flatten at TikTok or Reddit comments.

Quota math the SERP still gets wrong

Documented costs: commentThreads.list = 1 unit, comments.list = 1 unit, default 10,000 units/day (quota cost table, getting started).

Worked example, part=snippet, 100 comments/page: 10,000 units ÷ 1 unit/call = 10,000 pages/day of top-level comments = a theoretical 1,000,000 top-level comments/day. That is a ceiling on commentThreads.list before replies. It is also the figure behind the "~1 million comments/day" line you will see on comparison posts.

A popular how-to that prices commentThreads at "1–3 units" and concludes "roughly 3,000–10,000 comments per day" does not match the published 1-unit cost for commentThreads.list. Do not budget from that range.

Add replies and the arithmetic moves. Each comments.list page is another 1 unit. A 10,000-comment video with 2,000 replies is about 100 top-level pages + 20 reply pages = 120 units, not 100. part=replies does not replace comments.list for a full walk (official quote above).

Quota increases go through YouTube's compliance/quota audit, not a self-serve slider. There is no "buy more units" button. That is a real reason teams search for a comments scraper. search.list is a separate daily cap from comment reads; that story is YouTube API quota, not this page.

SocialCrawl: 1 credit per page of up to 100, 100 welcome credits ≈ 100 pages before paying, cache hits free, no midnight-Pacific reset. 600 requests/minute per key; the 429 is unbilled (error handling).

---

How does commentThreads pagination actually work?

BeautifulSoup on the watch page fails because comments load via AJAX. The HTML you download is a shell; the thread arrives later from YouTube's InnerTube continuation endpoints. That is why a requests.get of the watch URL plus a CSS selector returns an empty list, and why unofficial libraries look like scrapers but are really continuation clients.

The official path is quota-gated and ToS-clean: pageToken in, nextPageToken out, 1 unit per call, 10,000 units/day. Unofficial continuation is what most "scraper" GitHub READMEs actually hit; they rarely name InnerTube. Three token systems, side by side:

Official Data API          Unofficial InnerTube           SocialCrawl
commentThreads.list        youtube-comment-downloader     GET /v1/youtube/video/comments
pageToken → nextPageToken  continuation (AJAX/InnerTube)  continuationToken
1 unit / call, 10k/day     no Google key, ToS risk        1 credit / call, 300s cache
maxResults default 20      iterator / --limit             max_results default 100
comments.list?parentId=    (README is silent on replies)  GET /v1/.../comment/replies
order=time|relevance       sort 0 popular / 1 recent      order=newest|top

Sort maps time|relevance ↔ newest|top. The working pagination loop is in "How do I walk continuationToken until the thread ends?" above. This section is the map, not a second while.

---

Is it legal to scrape YouTube comments?

Two facts, both true at once.

1. YouTube's Terms prohibit accessing the service by automated means (robots, botnets, scrapers) except public search engines honoring robots.txt, or with YouTube's prior written permission (YouTube Terms of Service).
2. The permitted programmatic path is YouTube API Services, governed by the API Services ToS and Developer Policies.

hiQ Labs v. LinkedIn is the case every optimistic how-to cites. The Ninth Circuit in 2022 narrowed the CFAA for public data. Later that year hiQ still lost on the contract claim and entered a stipulated judgment. Do not ship the one-liner "scraping publicly accessible data generally does not violate the CFAA" without that second half. The longer read is the social media scraping legal primer. Technique context: web scraping. What we publish about public data: the public data notice.

Enforcement in practice is rate-limits, IP blocks, and quota revocation. A quiet docket is not permission.

The working code on this page is the documented API path (key, credits, two endpoints). It is not an InnerTube recipe, and it is not the official YouTube Data API. This is not legal advice.

---

What returns empty: commentsDisabled, kids, Shorts, dash-prefix IDs

Symptom	Official Data API	What to do here
Comments turned off	403 commentsDisabled	Expect empty or an error; skip the video. Do not invent a JSON body.
Made-for-kids / held-for-review	Public list omits them; moderationStatus=heldForReview needs OAuth + allThreadsRelatedToChannelId	A public key cannot see held-for-review
Private / age-restricted	403 / empty	Skip. Do not bypass.
Unlisted but comments on	Works with the videoId if you have the URL	Same: URL in, public comments out
Shorts / youtu.be	Extract the 11-character videoId	Pass the full URL to url=
Dash-prefix video ids	Fine as videoId	GitHub CLIs need -y=id; url= is fine if the URL is quoted
Displayed count > export	Deleted/spam included in the public number	Page until the token dies; parent reply count can exceed the public list
max_results 0 or 101	n/a	Free 400, no credit
Zero-reply parent	n/a	Replies endpoint → empty list, not 404
600 req/min	n/a	Unbilled 429 RATE_LIMITED

Official quotaExceeded is a 403 on the Data API and resets at midnight Pacific. The body and the reset math are in YouTube API quota; this page does not re-teach that clock.

A 502 or 503 (UPSTREAM_ERROR, SERVICE_UNAVAILABLE) refunds the credit. Retry the same token; do not advance past a page you did not read (errors).

---

Where does a comment scraper go next?

One video is the unit. A channel harvest is "list that channel's videos, then this comments loop per video" — point the YouTube platform hub or the visual explorer at the channel, then reuse the functions above. This post does not add a third endpoint.

If the CSV is a monitoring input, the social media monitoring API is the pipeline shape. The sibling how-to is scrape TikTok comments with Python. Endpoint contract: GET /v1/youtube/video/comments.

---

Frequently asked questions

How do I scrape all YouTube comments from a video?

Page GET /v1/youtube/video/comments until pagination.has_more is false, with max_pages set so a viral thread cannot drain the key. That walk covers public top-level comments. Held-for-review, likely-spam, and rejected comments are invisible without owner OAuth on the Data API. Replies are a second endpoint, not extra rows on page one.

How do I get YouTube comment replies, not just top-level comments?

Take ext.replies_token from each parent and call GET /v1/youtube/video/comment/replies with that value as continuationToken. Page until has_more is false. Official equivalent: comments.list?parentId= until nextPageToken dies. The tree is two levels; @mentions in the flat list are how later answers attach.

Can I export YouTube comments to CSV or JSON?

Yes. The Python above writes utf-8-sig CSV with id, parent_id, author, text, likes, reply_count, published_at, pinned, video_url. For JSON, json.dump the items list as-is. Filter text is None before sentiment; deleted rows arrive as null, not a "[deleted]" string.

Does the YouTube Data API let me download comments without a quota?

No. Default allowance is 10,000 units/day for comment reads and the rest of the non-search surface. commentThreads.list and comments.list each cost 1 unit (quota table, getting started). There is no self-serve top-up; extra quota goes through an audit form. Search quota is a different pain: YouTube API quota.

How do I paginate YouTube comments after the first page?

Official: read nextPageToken, send it back as pageToken. Here: read pagination.next_cursor, send it back as continuationToken (or the universal cursor), stop when has_more is false. Do not build the token, do not decode it, do not loop on total.

Is a YouTube comment downloader the same as a scraper API?

No. A YouTube comment downloader (or comment extractor) is usually a Chrome extension or a GitHub CLI (youtube-comment-downloader, yt-dlp --write-comments) that walks unofficial InnerTube continuations. A scraper API is a documented endpoint, a key, and a pagination contract. The working code on this page is the API path, not an extension.

Can I filter YouTube comments by keyword or date?

Keyword: searchTerm here, searchTerms on commentThreads.list. Date: no server-side date parameter on either API. Client-side: order=newest, keep in-window rows, stop when a page's oldest in-window row is before the cutoff. Do not stop at a pinned row; pins can sit first on newest.