keepcalculator

06 Numbers and Powers

Mirror a Website Locally: A Step-by-Step Guide

Card No. 06·84273 · Updated · Standard formula, shown below

Mirroring a website saves a local copy of pages and files so you can browse them offline, preserve a reference, or inspect a site you’re authorized to access. A mirroring tool can follow links and download resources automatically, but interactive features, private content, and some dynamically generated pages may not be reproduced. Before starting, check the site’s terms and access rules, and use a modest download rate.

What You’ll Need

  • A computer with enough free disk space for the site and a folder where you can store the copy.
  • A stable internet connection and the website’s starting URL.
  • A site-downloading tool such as HTTrack, or a command-line tool such as wget. The steps below use wget, available on many Linux systems and installable on macOS or Windows.
  • Permission to download the material. Do not try to defeat logins, paywalls, access controls, or download restrictions.

Step-by-Step

  1. Choose a safe, specific scope. Decide which public site or section you need. A whole large website can take hours and consume substantial storage. Check its terms, any published crawling guidance, and whether the content is copyrighted or personal. If you only need one section, mirror that section rather than the entire domain.
  2. Install wget if needed. On Linux, check whether it is already available by opening a terminal and entering wget --version. On macOS or Windows, install wget from a trusted package manager or distribution source, then open Terminal, PowerShell, or Command Prompt. Follow the installer’s instructions and confirm that wget --version displays a version. If you prefer a graphical interface, HTTrack offers a project-based way to set a URL, destination, and download options; the same scope and permission cautions apply.
  3. Create a destination folder. Make a new, empty folder for the mirror. For example, create site-copy in your Documents folder. Keeping the download separate prevents it from being mixed with other files and makes it easier to remove or update later.
  4. Run a conservative mirror command. In the terminal, move to the parent folder where you want the copy saved. Run this command, replacing the example address and folder name with your own:
    wget --mirror --page-requisites --convert-links --adjust-extension --no-parent --wait=1 --limit-rate=200k --directory-prefix=site-copy https://example.com/

    The options tell wget to recurse through links, fetch files needed to display pages, rewrite links for local browsing, add suitable HTML extensions, avoid crawling above the starting path, pause between requests, cap transfer speed, and save files in the chosen folder. The rate and pause are examples of conservative settings, not guarantees that a site’s server load will be low. Adjust them only with care and in accordance with the site’s guidance.

  5. Let the download finish and review the output. Keep the terminal window open. Wget reports downloaded files and errors as it works. Large sites may take a long time. Stop the process with Ctrl+C if you realize the scope is too broad, the server objects, or the download is causing problems. Check the final messages before assuming the mirror is complete.
  6. Open the local copy. Open the destination folder, find the saved site directory, and open its index.html file in a browser. If no index page is present, open the HTML file matching the starting page’s path. Follow internal links and check images, stylesheets, and other needed resources. A browser may restrict some functions when a page is opened directly from disk; that does not necessarily mean the files are missing.
  7. Use a local web server if direct opening fails. If Python is installed, open a terminal in the folder containing the downloaded index file and run python -m http.server 8000. On some systems, use python3 instead. Then visit http://localhost:8000/ in your browser. This serves the files from your own computer. Stop it with Ctrl+C when finished.
  8. Keep or refresh the mirror deliberately. Store a note with the starting URL and date so you know what the copy represents. To update it, run a suitable wget command again using the same destination, then check what changed. Do not assume an old copy is a complete or authoritative record; files may have been missed, updated, or removed.

Common Mistakes

  • Starting too high in the site hierarchy: Beginning at a home page may lead to thousands of pages or external links. Start at the narrowest useful URL and use a limit appropriate to the site.
  • Using the wrong path or folder: A typo in the URL or destination can produce an empty-looking copy or save files somewhere unexpected. Check the terminal’s current directory and the exact URL before running the command.
  • Expecting a perfect, interactive duplicate: A mirror usually saves retrievable files, not the server-side code, database, search, forms, or account functions. Some scripts rely on live services and will not work offline.
  • Ignoring external resources: The command limits crawling above the starting path, and resources on other domains may not be included. Do not remove scope limits casually; doing so can expand the download beyond the intended site.
  • Repeatedly restarting without checking: This can waste bandwidth and create confusion about which files are current. Inspect the folder and log output first, then decide whether to resume, update, or start over.

Tips

  • Try a small section or a single page first. Confirm that local links and images work before increasing the scope.
  • Use a separate folder for each site and date. This makes it easier to compare copies and delete an outdated one.
  • Respect site instructions, rate limits, and server responses. If access is denied or the site asks crawlers to stop, do not try to evade that restriction.
  • For a large site, check available disk space before beginning. A mirror can be much larger than the visible text because it may include images, documents, and other assets.
  • Use a local web server for testing when the browser blocks local scripts or paths. It serves the saved files without making them public on the internet.

Troubleshooting

The command is not recognized: wget may not be installed, or its location may not be on the system path. Reopen the terminal after installing it, check the installation instructions, and try wget --version. Alternatively, use a graphical mirroring tool.

The folder is empty or contains an error page: Verify the URL, internet connection, and destination path. A site may redirect to another address, require a browser session, block automated requests, or be unavailable. Do not attempt to bypass access controls; ask the site owner for an approved export if needed.

Pages load but images or styles are missing: Confirm that you used --page-requisites and inspect wget’s output for failed requests. Some resources may be hosted elsewhere, generated dynamically, or unavailable to automated downloads. A static mirror cannot recreate every live dependency.

Links lead back online or to missing files: Confirm --convert-links was included and that you are opening the downloaded copy, not the original site. Try serving the folder locally as described above. If a page was never downloaded, narrow the issue by checking its URL in the terminal output.

The download is too large or slow: Stop it, remove or keep the partial folder as appropriate, and restart with a narrower starting path. Keep a pause and rate limit in place. If the server returns errors or asks you to stop, stop rather than increasing the request rate.

Frequently Asked Questions

Can I mirror any website?
Technically, some public pages may be downloadable, but site terms, copyright, privacy, and access restrictions matter. Only mirror content you are allowed to copy, and do not bypass controls.

Will the local copy work without internet?
Saved HTML and downloaded resources can work offline, but features that call a live server or load remote files may fail.

Does a mirror copy login-protected pages?
Not with the example command, and this guide does not recommend bypassing authentication. For content you own or administer, use an authorized export or backup method.

How much disk space will I need?
It depends on the number and size of pages, images, downloads, and other assets. Start with a small test and check the folder size before mirroring a larger section.

How do I make the local copy searchable?
Some downloaded sites retain working internal navigation, but search often depends on a server-side service. You may need a separate local indexing tool; a mirror alone does not provide one.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →