JS Crawler: How to Crawl Websites with Node.js

This guide shows you how to build a simple web crawler in Node.js that visits pages, extracts links, and follows them to discover more URLs. You’ll end with a reusable script that prints every page it finds, along with the links on each page. Before starting, you should have Node.js installed (version 14 or newer) and a basic understanding of JavaScript promises and async/await.

What You’ll Need

Node.js (v14+) installed on your machine
A terminal or command prompt
A text editor (VS Code, Sublime, etc.)
Basic familiarity with npm (Node package manager)

Step-by-Step

Set up a new project. Create a folder for your crawler, open a terminal inside it, and run npm init -y to create a package.json file.
Install the required packages. We’ll use axios for HTTP requests and cheerio for parsing HTML. Run: npm install axios cheerio.
Create the crawler file. Make a file named crawler.js in the project folder.

Import the modules. At the top of crawler.js, add:

const axios = require('axios');
const cheerio = require('cheerio');

Define the crawler function. Create an async function that takes a URL and a visited set. For example:

async function crawl(url, visited) {
  if (visited.has(url)) return;
  visited.add(url);
  try {
    const response = await axios.get(url);
    const $ = cheerio.load(response.data);
    console.log(`Crawled: ${url}`);
    // Extract all links
    const links = [];
    $('a').each((i, el) => {
      const href = $(el).attr('href');
      if (href) links.push(new URL(href, url).href);
    });
    console.log(`Found ${links.length} links on ${url}`);
    // Recursively crawl internal links
    for (const link of links) {
      if (link.startsWith('http') && new URL(link).hostname === new URL(url).hostname) {
        await crawl(link, visited);
      }
    }
  } catch (error) {
    console.error(`Error on ${url}: ${error.message}`);
  }
}

Start the crawl. Add a line to call the function with a starting URL:
```
crawl('https://example.com', new Set());
```
Run the script. In the terminal, execute node crawler.js. You’ll see the output in the console.
Add a delay (optional). To be polite to servers, add a small delay between requests using setTimeout or a package like p-limit to limit concurrency.
Save results to a file. Instead of just logging, you can collect all URLs in an array and write them to a text file using fs.writeFileSync.

Common Mistakes

Not checking the content type: Some URLs may return images or PDFs. Check response.headers['content-type'] to ensure it’s HTML before parsing.
Following external links: Without the hostname check, your crawler will go off-site and potentially crawl the entire internet. Always filter to the same domain (or a defined set of domains).
Infinite loops: If you don’t track visited URLs, you’ll loop forever on pages that link to each other. Always use a Set to record visited URLs.
Not handling relative URLs: Using new URL(href, url) correctly resolves relative paths like /about to absolute URLs.
Ignoring robots.txt: Ethical crawlers should respect robots.txt. Check it before crawling a site and skip disallowed paths.

Tips

Use a user-agent: Some sites block default axios user-agents. Set a custom one in the request headers, like 'Mozilla/5.0 (compatible; MyCrawler/1.0)'.
Limit concurrency: Crawling hundreds of pages simultaneously can overwhelm a server. Use a simple counter or a library like p-limit to keep requests to 2-5 at a time.
Handle pagination: Many sites have ‘next’ links. You can extract those and follow them explicitly.
Parse only what you need: If you only need certain data (like product prices), use cheerio selectors to extract just that, saving memory and time.
Use a database for large crawls: For thousands of pages, store results in a SQLite or MongoDB database instead of memory.

Troubleshooting

Getting 403 or 429 errors: The site is blocking you. Add a proper user-agent, slow down, or use a proxy. If it’s a 429, wait longer between requests.
Cheerio isn’t finding elements: The page might be rendered with JavaScript. In that case, you need a headless browser like Puppeteer, not just axios.
“TypeError: Invalid URL” happens when a link is malformed. Wrap the URL creation in a try/catch or check that the href starts with ‘http’ or ‘/’.
The script runs forever: There may be a loop or too many pages. Add a maximum page count or depth limit.
Memory usage grows too high: You might be storing too much data. Stream results to a file or database instead of keeping everything in memory.
Axios fails on HTTPS: If you get SSL errors, try adding httpsAgent with rejectUnauthorized: false (not recommended for production).

Frequently Asked Questions

Is web crawling legal?

It depends on the site’s terms of service and your jurisdiction. Always check robots.txt and the site’s terms. For personal, non-commercial use, it’s often fine, but avoid scraping personal data or copyrighted content.

How do I crawl a site that uses JavaScript to load content?

Use a headless browser like Puppeteer or Playwright. They execute JavaScript and give you the rendered HTML. The same logic applies, but you’ll use page.goto() instead of axios.

Can I crawl multiple domains at once?

Yes, but you need to adjust the hostname check to allow multiple domains. Be careful about the scale—crawling many domains can get your IP blocked.

How do I extract specific data like titles or prices?

Use cheerio selectors. For example, $('h1').text() gets the main heading. Inspect the site’s HTML to find the right selectors.

What’s the difference between a crawler and a scraper?

A crawler discovers and visits URLs, while a scraper extracts specific data from a page. This guide focuses on crawling; you can combine both by adding extraction logic inside the crawl function.

How do I stop the crawler after a certain number of pages?

Add a counter variable and check it before each recursive call. For example, if (count >= 100) return;.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →