Web Scraping in 2026: 3 Techniques + When to Use an API

Web scraping is the automated extraction of data from websites: a script requests a page, parses the HTML, and pulls out the fields you need instead of a human copying and pasting them [Source: https://en.wikipedia.org/wiki/Web_scraping]. By the end of this post you'll know the three techniques scrapers actually use, have a working Python example with real error handling, and know how to tell when writing your own scraper stops making sense.

That last part matters more than it used to. In 2024, automated traffic crossed 51% of all web requests for the first time since Imperva started tracking the metric in 2013 [Source: https://www.ficstar.com/state-of-anti-bot-technology-2026]. "Just write a scraper" is a harder problem in 2026 than it was five years ago, and we'll get into exactly why.

What web scraping actually does

Web scraping converts unstructured HTML into structured, usable data [Source: https://en.wikipedia.org/wiki/Web_scraping]. Fortinet frames it more operationally: bots that browse and copy content from websites at a scale no human could replicate [Source: https://www.fortinet.com/resources/cyberglossary/web-scraping].

One term gets used interchangeably with scraping, but it means something different: crawling. Crawling is discovering URLs, following links across a site (or the whole web) to build a list of pages worth visiting. Scraping is extracting structured data from a page you already have the URL for. A search engine crawls; a price-tracking script scrapes. Most real projects do both: crawl to find product pages, then scrape each one for price and stock data. You'll also see "web data extraction" and "web harvesting" used as synonyms for scraping in vendor docs and Wikipedia, respectively [Source: https://en.wikipedia.org/wiki/Web_scraping].

How to scrape data from a web page

There are three broad techniques, and the right one depends entirely on how the target page is built. Going in order of cost and complexity saves you from reaching for a browser automation framework when a plain HTTP request would have worked.

HTTP requests + HTML parsing

The cheapest and fastest method: send a GET request, get back raw HTML, and parse it with a library like BeautifulSoup using CSS selectors or XPath to pull out the elements you want [Source: https://medium.com/@joerosborne/intro-to-web-scraping-build-your-first-scraper-in-5-minutes-1c36b5c4b110]. This only works if the data you need is actually present in the HTML the server sends back, before any JavaScript runs.

Headless browsers for JavaScript-rendered pages

Most modern sites, especially social platforms and single-page apps, render their content client-side. The data isn't in the raw HTML; it's built by JavaScript after the page loads. To scrape that, you need a real browser engine executing the page first. Playwright, Puppeteer, and Selenium all automate actual browser instances (Chromium, Firefox, WebKit) for exactly this reason [Source: https://bytetunnels.com/posts/playwright-vs-puppeteer-vs-selenium-vs-scrapy-2026-mega-comparison/].

The tooling landscape here has a clear old-guard/new-guard split. Selenium, first released in 2004, remains the most enterprise-installed option: 55,785 to 63,549 verified companies depending on the tracker, roughly 12 times Playwright's verified install base [Source: https://testdino.com/blog/selenium-market-share]. But developer attention has shifted hard toward Playwright, which leads in GitHub stars (80,100+ versus Selenium's 34,000+) and hit a 94% "would use again" retention rate in the 2024 State of JS survey, the highest of any end-to-end tool measured that year [Source: https://testdino.com/blog/selenium-market-share]. Read that as: old reliable versus new default. If you're inheriting a legacy test suite, it's probably Selenium. If you're starting something new, it's probably Playwright.

Reverse-engineered or official APIs

Some platforms expose a documented API; others have undocumented internal APIs scrapers find by inspecting network requests in the browser. Skipping HTML entirely and hitting a JSON endpoint directly is generally the most stable approach when it's available. The catch, which we'll come back to later in this post, is that this path is increasingly closed off by the platforms themselves.

A working example: scraping a page with Python

Here's a minimal scraper using requests and BeautifulSoup. Most tutorials stop at the happy path: send a request, assume it works, parse the response. That's fine until the target site rate-limits you, returns a 403, or times out, at which point a script with no error handling just crashes or silently returns garbage.

import time
import requests
from bs4 import BeautifulSoup

def fetch_page(url, retries=1, timeout=10):
    headers = {"User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"}
    for attempt in range(retries + 1):
        try:
            response = requests.get(url, headers=headers, timeout=timeout)
            response.raise_for_status()
            return response.text
        except requests.exceptions.RequestException as e:
            if attempt < retries:
                time.sleep(2 ** attempt)
                continue
            raise RuntimeError(f"Failed to fetch {url}: {e}") from e

html = fetch_page("https://example.com")
soup = BeautifulSoup(html, "html.parser")

title = soup.select_one("h1")
print(title.get_text(strip=True) if title else "No title found")

Three things this example does that most beginner tutorials skip: it checks the status code with raise_for_status() instead of assuming a 200, it sets a timeout so a hung connection doesn't block your script forever, and it retries once with exponential backoff before giving up. None of that is exotic, but it's the difference between a scraper that works in a demo and one that survives a real run against a real site.

If you run this against a page and get back an empty shell with none of the content you can see in your browser, the data probably isn't in the initial HTML at all: it's rendered by JavaScript after load. That's your cue to swap requests.get() for a headless browser's page.goto() instead of debugging your CSS selectors for another hour.

What are the best web scraping tools?

The honest answer depends on your target, not a universal ranking. Here's the shortlist and who each one is actually for:

• Scrapy: a production-scale, asynchronous crawling framework with no browser involved, built for large jobs across many pages. 92,720 GitHub stars and counting [Source: https://scrapy.org/].
• BeautifulSoup + requests: the smallest surface area of anything on this list. Best for one-off jobs against static, server-rendered pages.
• Playwright: the modern default for JavaScript-rendered targets, and the one most new projects reach for first in 2026.
• Selenium: the largest installed base of any tool here despite being the oldest. Worth knowing if you inherit a legacy automation suite, even if you wouldn't start a new project on it.
• webscraper.io and Octoparse: no-code browser extensions aimed at non-developers who want point-and-click scraping without writing anything. Fine for a one-time export job; not built for the kind of programmatic, scheduled extraction a developer usually needs.

If your actual target is social platform data specifically (TikTok, Instagram, Reddit, YouTube, and similar), a general-purpose crawl website tool isn't really competing with a managed API purpose-built for that data and its unified schema, which we'll get into below.

The real cost of scraping at scale in 2026

Writing the scraper is the easy part. Keeping it working is where the real cost lives, and the numbers here explain why.

Automated traffic hit 51% of all web requests in 2024, the first time bots outnumbered humans since Imperva began tracking it in 2013, with "bad bots" specifically making up 37% of all traffic, up from 32% the year before [Source: https://www.ficstar.com/state-of-anti-bot-technology-2026]. Sites have responded with layered defenses that go well beyond a simple CAPTCHA: behavioral analysis of mouse movement and click timing, device and browser fingerprinting, ML-based anomaly detection retrained continuously on threat feeds, and IP-reputation-based rate limiting [Source: https://www.ficstar.com/state-of-anti-bot-technology-2026]. This isn't hypothetical infrastructure. DataDome runs more than 85,000 customer-specific machine learning models for bot detection, and Cloudflare started blocking AI-based scraping by default in July 2025 [Source: https://tendem.ai/blog/how-anti-bot-systems-work-scrape-anyway]. Akamai, as reported by Mordor Intelligence, claims its bot-manager suite blocks 82.3% of automated traffic on select protected pages [Source: https://www.mordorintelligence.com/industry-reports/web-scraping-market].

That's why proxies and IP rotation have gone from "nice to have" to mandatory infrastructure. A single IP hammering a site gets flagged fast, which means reliable scraping at any real scale now requires rotating residential proxy pools and human-like request timing as an ongoing arms race, not a one-time setup [Source: https://www.ficstar.com/state-of-anti-bot-technology-2026]. And even a scraper that works today can silently break tomorrow: the moment a target site changes its markup or anti-bot configuration, your extraction either fails loudly or, worse, keeps running and returns wrong data. Ficstar puts it well: the real cost of scraping infrastructure "is rarely the sticker price, it's the engineering hours spent rebuilding crawlers every time a target site changes" [Source: https://www.ficstar.com/state-of-anti-bot-technology-2026]. Downstream, bad or bot-polluted data isn't free either; Gartner estimates poor data quality costs organizations an average of $12.9 million a year [Source: https://www.ficstar.com/state-of-anti-bot-technology-2026, citing Gartner].

Is web scraping legal?

The honest answer is: it depends, and the nuance matters more than a yes/no.

In hiQ Labs, Inc. v. LinkedIn Corp, the Ninth Circuit ruled that scraping publicly available website data does not violate the Computer Fraud and Abuse Act [Source: https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn]. That's a real, meaningful precedent, but it's not a blanket green light. Legal commentary on the case notes that scraping public data can still be legal under the CFAA while creating separate liability under a breach-of-contract claim, meaning violating a site's Terms of Service can expose you to civil risk even when no criminal statute was broken [Source: https://www.morganlewis.com/blogs/sourcingatmorganlewis/2022/12/linkedin-v-hiq-landmark-data-scraping-suit-provides-guidance-to-data-scrapers-and-web-operators].

The more recent case worth knowing is Meta Platforms v. Bright Data (N.D. Cal.). In January 2024, the court denied Meta's motion for summary judgment on its breach-of-contract claim, ruling that Bright Data hadn't breached Facebook and Instagram's Terms of Service because it scraped while logged out, and a logged-out visitor isn't contractually a "user" bound by those terms [Source: https://www.courthousenews.com/federal-judge-rules-against-meta-in-data-scraping-case/]. Meta dropped the suit entirely in February 2024 [Source: https://www.reddit.com/r/SaaS/comments/1ipjxkd/how_on_earth_are_major_companies_scraping_data/]. Don't over-read this one either: it turned specifically on logged-out versus logged-in scraping under California contract law, not a general ruling that scraping a platform is fine.

The practical takeaway: respect robots.txt, avoid scraping behind a login wall, and remember that a ToS violation is a civil/contract risk that runs on a completely separate track from criminal law, with outcomes depending on jurisdiction and the specific contract language at issue. Here's the bigger pattern: platforms that don't want to be scraped are increasingly closing off the official API path that used to make this whole question moot, so you never get far enough to need the legal answer.

DIY scraping vs. managed data APIs

Most explainers skip this part entirely, so let's state it plainly: official platform APIs are getting more expensive and more restricted, not less. That's the actual force pushing developers back toward HTML scraping and browser automation. Mordor Intelligence names "API deprecation on major platforms" as a direct driver of the scraping market's growth [Source: https://www.mordorintelligence.com/industry-reports/web-scraping-market].

Two examples make this concrete instead of abstract. As of February 2026, X closed free-tier and flat-fee self-serve access for new developers in favor of pay-per-use pricing: $0.005 per post read and $0.010 per profile read, charged per resource rather than per call, so a single call returning 100 posts costs $0.50 [Source: https://www.blotato.com/blog/twitter-api-pricing]. In April 2026, X added a "URL-post tax": a plain post write costs $0.015, but a post containing a URL costs $0.20, a roughly 1,233% premium [Source: https://www.blotato.com/blog/twitter-api-pricing]. Usage caps out at 2 million reads a month before you're negotiating a custom Enterprise contract starting around $42,000/month [Source: https://www.blotato.com/blog/twitter-api-pricing].

Reddit's 2023 API pricing changes are the other data point: $0.24 per 1,000 calls, which the developer of the third-party app Apollo estimated would cost roughly $20 million a year to keep running at his existing usage. He shut it down instead [Source: https://en.wikipedia.org/wiki/Reddit_API_controversy]. For a worked single-site example of the same build-vs-buy decision, see Naver crawling in 2026.

None of this means scraping is now mandatory, or that official APIs are always the wrong call. It means a developer building anything on social data today is choosing between three real costs: official API pricing and limits, DIY scraping's compounding maintenance burden (proxies, anti-bot tuning, layout-drift monitoring, and per-platform legal review, multiplied across every platform you need), or a managed API that has already absorbed that maintenance burden for you.

That third option is where SocialCrawl fits in. Instead of separately maintaining headless-browser orchestration, proxy rotation, CAPTCHA handling, and layout-drift monitoring for every social platform you need data from, you get a unified schema back from one API key, regardless of which platform the data came from. To be clear about scope: this doesn't replace scraping for everything. Product listings, real estate data, and general-purpose websites are still very much DIY-scraper territory. Social platform data specifically is the problem SocialCrawl actually solves.

Scraping responsibly: what to do before you write a line of code

Whichever approach you pick, a few habits keep you out of trouble and out of the way of the sites you're pulling from:

• Check and respect robots.txt before you scrape anything.
• Set a real, identifiable User-Agent instead of spoofing a browser string.
• Rate-limit your own requests; don't hit a server harder than a human user would.
• Cache what you've already fetched instead of re-scraping the same page.
• If an official API covers what you need, use it before reaching for a scraper.

What's next

If you want to try the Python example above, point it at a page you're allowed to scrape and watch what happens when you deliberately break something (kill your network mid-request, or point it at a JS-heavy page) to see the error handling do its job. If your target is social platform data instead of a generic website, take a look at the SocialCrawl API docs and see whether one API key covers what you were about to build a scraper for.

Frequently asked questions

What's the difference between web scraping and web crawling?

Crawling is discovering pages and links, either across a single site or the broader web, to build up a list of URLs worth visiting. Scraping is extracting structured data from a page you already have the URL for. A search engine indexer crawls; a script pulling prices off a product page scrapes. Real-world projects usually chain the two: crawl to find pages, then scrape each one.

How do you scrape multiple pages in Python?

The most common pattern is looping over a paginated URL structure (?page=1, ?page=2, and so on) or following "next page" links you find in each response's HTML, calling your fetch function on each one in turn. Add a short delay between requests so you're not hammering the server, and stop the loop once a page returns no more results or a "last page" signal.

How do you scrape a table from a webpage into Excel or CSV?

For simple, well-formed HTML tables, pandas' read_html() function will parse a page's tables directly into DataFrames you can export with .to_csv(). For anything messier or more custom, BeautifulSoup to extract the rows combined with Python's built-in csv module (or openpyxl for native .xlsx output) gives you full control over formatting.

Can you get blocked for web scraping?

Yes. Rate limits, IP bans, and CAPTCHA walls are the common technical responses, and depending on what you scrape and how, you can also create ToS-based contract liability separate from any criminal exposure, as covered above. Reasonable rate limiting, a legitimate User-Agent, and respecting robots.txt reduce the odds significantly, but they don't eliminate the risk entirely on sites actively trying to block automated traffic.

What is a crawl website tool?

"Crawl website tool" is generally used as a synonym for the broader category of web scraping tools covered above: frameworks like Scrapy for large-scale crawling, libraries like BeautifulSoup for lightweight extraction, headless browsers like Playwright for JavaScript-rendered sites, or no-code extensions like webscraper.io for point-and-click jobs. Which one counts as the right "crawl website tool" for you depends entirely on the target site and the scale of the job, per the breakdown above.