Curl User Agent: What It Is & How to Set It

The curl user agent is the string curl sends in the User-Agent HTTP header to identify itself to web servers. By default, curl sends curl/8.0.1 (or your installed version), but you can change it with the -A or --user-agent option to mimic a browser or any other client.

What Is a User Agent?

A user agent is a string that a client (like a browser, bot, or command-line tool) sends in the HTTP request header to describe itself. The server can read this string to tailor the response—for example, serving a mobile version of a page to a phone browser, or blocking requests that look like automated scripts.

The format is not standardized, but browsers typically follow a pattern like:

• Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15
• curl/8.0.1

Notice that curl’s default is much shorter. That brevity is often enough for simple API calls, but many websites and CDNs treat any non-browser user agent differently—sometimes blocking it outright.

How to Set the User Agent in curl

You can override the default user agent with the -A (or --user-agent) flag followed by the string. Here are common examples:

• curl -A "Mozilla/5.0" https://example.com – sends a generic browser-like string.
• curl -A "MyCustomBot/1.0" https://api.example.com/data – identifies your own bot.
• curl --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" https://example.com – long form.

You can also set the header directly with -H "User-Agent: custom", which does the same thing. If you need to send an empty user agent (some servers treat that as a signal), use -A "".

Why Does the User Agent Matter?

Servers use the user agent for three main reasons:

1. Content negotiation – serving different HTML, CSS, or media to different devices.
2. Bot detection – blocking or rate-limiting requests that don’t look like a real browser.
3. Analytics – logging which clients are hitting the site.

If you are using curl to fetch a page that normally loads in a browser, you may get a stripped-down version, a CAPTCHA, or a 403 Forbidden error. Setting a browser user agent often fixes that—but be aware that some sites also check other headers (like Accept or Accept-Language) and may still block you.

How to Check What User Agent You’re Sending

To see exactly what curl sends, use the -v (verbose) flag. It prints the request headers to stderr:

curl -v https://example.com

In the output, look for a line like:

> User-Agent: curl/8.0.1

If you use -A, the line will reflect your custom string. This is the quickest way to debug why a server might be responding differently.

Common Mistakes and Misconceptions

• Mistake: Using a fake browser UA is always necessary. For APIs and public endpoints, the default curl UA is fine. Only change it when a site explicitly blocks non-browser clients.
• Mistake: Setting a UA but not other browser headers. Some sites check Accept, Accept-Language, and Referer too. Use -H to add them if needed.
• Misconception: The UA is a security feature. It’s just a string; anyone can set it to anything. Servers cannot trust it for authentication.
• Mistake: Forgetting to quote the UA. If your string contains spaces, you must wrap it in quotes in the shell, or curl will treat it as multiple arguments.

Comparing curl’s Default UA to Browsers

Here’s a side-by-side look at what different clients typically send:

• curl – curl/8.0.1 (short, no OS or browser info)
• Chrome on Windows – long string with Mozilla, Windows, AppleWebKit, Chrome, and Safari tokens
• Firefox on Linux – Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0
• Googlebot – Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The “Mozilla/5.0� prefix is a historical artifact—modern browsers all include it for compatibility, even though the original Mozilla browser is gone. Many servers look for that prefix to decide if a request comes from a real browser.

When to Use a Custom User Agent

• Web scraping – if you’re politely scraping a site, set a UA that identifies your bot and includes a contact URL, so the site owner can reach you.
• Testing – developers often set a mobile or desktop UA to test how a site responds to different devices.
• APIs – some APIs require a custom UA for rate limiting or tracking. Check the API documentation.
• Bypassing blocks – if a site blocks the default curl UA, setting a browser UA may help, but it’s not guaranteed and may violate the site’s terms of service.

Always respect robots.txt and the site’s terms. A custom UA does not give you permission to scrape aggressively or ignore rate limits.

Frequently Asked Questions

What is the default user agent for curl?

The default is curl/ followed by the version number, e.g., curl/8.0.1. You can see it by running curl --version.

How do I set a custom user agent in curl?

Use the -A flag: curl -A "MyAgent/1.0" https://example.com. Or use the long form --user-agent. For a string with spaces, always enclose it in quotes.

Why does a website block curl but allow browsers?

Many sites block requests that don’t look like a real browser, often to prevent scraping or automated abuse. The short curl/ UA is a clear signal. Setting a browser UA can sometimes bypass the block, but the site may also check other headers or use JavaScript challenges.

Can I remove the user agent header entirely?

Yes, use -H "User-Agent:" to send an empty header. Some servers treat that differently—some may reject it, others may accept it. It’s rarely useful, but it’s possible.

Does the user agent affect the response content?

Yes, for sites that do content negotiation. For example, a news site might serve a lightweight AMP version to a bot UA, or a mobile version to a phone UA. If you’re not getting the page you expect, try setting a different UA.

Is it illegal to fake a browser user agent?

No, it’s not illegal, but it may violate a website’s terms of service. Using a fake UA to bypass a block or access content you’re not authorized to see could lead to your IP being banned or legal action in extreme cases. Always use a UA that accurately represents your client.

Related

• Nano Delivery for Functional Foods: What It Can and Can’t Do
• Disable Apple Watch Turn-by-Turn Vibration While Driving
• Convert Play FLAC in iTunes: Full Guide

in t | Finish time | min | 50 | 0
in d | Distance | km | 10 | 0 | 100000
out pace | Average pace | min/km | t / d | 2
out speed | Average speed | km/h | d / (t / 60) | 2