Scrape React, Vue & Angular Sites: A How-To

This guide shows you how to extract data from websites built with React, Vue, or Angular. These JavaScript frameworks render content dynamically, so standard scraping methods often fail. You’ll learn practical techniques to get the data you need, whether for analysis, research, or automation.

What You’ll Need

Basic knowledge of HTML, CSS, and JavaScript.
Familiarity with browser Developer Tools (right-click > Inspect).
Python (3.7+) or Node.js installed on your computer.
A code editor (VS Code, PyCharm, etc.).
For Python: requests, beautifulsoup4, and selenium or playwright packages.
For Node.js: axios, cheerio, and puppeteer or playwright packages.

Step-by-Step

Inspect the Page to Identify the Rendering Method
Open the target website in Chrome or Firefox. Right-click and select “View Page Source”. If you see mostly empty tags with JavaScript files at the bottom, it’s a client-side rendered app (React, Vue, Angular). If you see the actual content, it might be server-side rendered, but still use the same approach for safety.
Check the Network Tab for API Endpoints
Open Developer Tools (F12), go to the “Network” tab, and reload the page. Look for XHR or Fetch requests. These are API calls that return JSON data. Often, the data you see on the page comes directly from these endpoints. Find the one that returns the data you need (e.g., a list of products, user profiles, etc.).
Test the API Endpoint Directly
Right-click the API request and select “Copy as cURL”. Then, in your terminal, paste and run it. If you get a JSON response, you can skip DOM scraping entirely. Use Python’s requests or Node’s axios to replicate that request. Add headers like User-Agent and Accept to avoid blocks.
If No API, Use a Headless Browser
When the data is embedded in the JavaScript bundle or rendered after complex interactions, use a headless browser. With Python, install Playwright: pip install playwright and then playwright install. With Node.js: npm install playwright.

Write a Basic Scraper with Playwright
Here’s a Python example to load a page and extract text:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)  # Set to True for production
    page = browser.new_page()
    page.goto('https://example.com')
    # Wait for content to load
    page.wait_for_selector('div.product')
    # Extract data
    items = page.query_selector_all('div.product')
    for item in items:
        name = item.query_selector('h2').inner_text()
        price = item.query_selector('.price').inner_text()
        print(name, price)
    browser.close()

For Node.js, the equivalent uses puppeteer or Playwright’s async API.

Handle Infinite Scroll and Lazy Loading
Many React/Vue/Angular sites load more content as you scroll. In Playwright, use page.mouse.wheel(0, 10000) or page.evaluate('window.scrollTo(0, document.body.scrollHeight)') in a loop, waiting for new elements to appear. Alternatively, find the API call that fetches the next page of data and call it directly with a different page parameter.
Extract Data from the JavaScript State (Advanced)
If the data is not in the DOM but is in a JavaScript variable (e.g., window.__INITIAL_STATE__), you can evaluate JavaScript in the page context:
```
data = page.evaluate('() => window.__INITIAL_STATE__')
```
This returns a Python dict or Node object that you can parse.
Deal with Authentication and Sessions
If the data requires login, capture cookies from a logged-in browser session and pass them to your scraper. In Playwright, you can use context.add_cookies() to set them.
Respect Rate Limits and Robots.txt
Always check robots.txt (e.g., https://example.com/robots.txt) for allowed paths. Add delays between requests (time.sleep(2) in Python) to avoid overwhelming the server.
Save Your Data
Write the scraped data to a CSV or JSON file. Use Python’s csv module or Node’s fs module. Ensure you handle Unicode characters correctly (use encoding='utf-8' in Python).

Common Mistakes

Scraping the static HTML – You’ll get an empty page because the content is injected by JavaScript after load.
Not waiting for elements – Using page.goto() and immediately trying to select elements fails. Always use wait_for_selector() or similar.
Ignoring API calls – The easiest path is often the API, but many beginners go straight to DOM scraping.
Hard-coding selectors – Class names in React/Vue apps are often hashed (like _3xY7) and change between builds. Use stable attributes like data-testid or text content where possible.
Not handling pagination – You might only scrape the first page of results.
Breaking the site’s terms – Always review the website’s Terms of Service. Scraping may be prohibited.

Tips

Use the browser’s “Copy as cURL” – This gives you the exact headers and cookies needed to replicate an API request.
Inspect the JavaScript bundle – Look for strings like api/ or graphql in the source to discover hidden endpoints.
Use headless mode for production – Set headless=True to run without a visible browser, which is faster and uses fewer resources.
Cache your requests – If you’re scraping repeatedly, save responses locally to avoid hitting the server again.
Use a rotating User-Agent – Some sites block default headless browser user agents. Change them periodically.
Leverage CSS selectors over XPath – They are easier to read and maintain.
Use the page.locator() method in Playwright – It’s more robust than query_selector_all and supports auto-waiting.

Troubleshooting

Getting empty results – The page might be using Shadow DOM or iframes. Use page.frames to access iframes, and for Shadow DOM, use page.evaluate to pierce it.
Blocked by Cloudflare or similar – Use Playwright’s stealth mode or consider using a residential proxy. Avoid aggressive scraping patterns.
Data appears but not in the DOM – It might be in a canvas or WebGL. If so, use OCR or capture screenshots and parse them, but that’s very complex.
Element not found even after waiting – The selector might be wrong. Check if the element is inside a template tag or rendered conditionally. Use page.content() to see the final HTML.
API returns 403 or 401 – You need to include the correct headers (especially Authorization or X-CSRF-Token) or cookies. Re-copy the cURL from the browser after logging in.
Memory leaks – If you’re scraping many pages in a loop, close the browser after each batch and restart. Use browser.close() and start a new one.

Frequently Asked Questions

Is it legal to scrape React/Vue/Angular websites?

It depends on the website’s Terms of Service and your jurisdiction. Always check the terms and robots.txt. Public data is generally okay, but personal data or copyrighted content may be restricted.

What’s the best tool for scraping these frameworks?

Playwright (Python or Node) is the most robust because it handles dynamic rendering and offers fine control. Puppeteer is also good for Node users. For simple APIs, just use requests or axios.

Why do I get empty HTML when I use `requests`?

Because the content is loaded by JavaScript after the initial HTML is fetched. The requests library doesn’t execute JavaScript. You need a headless browser like Playwright.

Can I scrape data from a React app without a headless browser?

Yes, if the app fetches data from an API, you can call that API directly. Look at the Network tab to find the endpoints. This is often faster and more reliable.

How do I handle infinite scroll?

Either simulate scrolling with a headless browser or find the API endpoint that returns the next page of data. The latter is simpler and more efficient.

What if the website uses GraphQL?

GraphQL endpoints are just POST requests with a JSON body. You can replicate them with requests or axios. Inspect the Network tab to see the query and variables.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →