Modify HTTP Request Headers with Selenium WebDriver in Java

This guide shows you how to intercept and modify HTTP request headers when using Selenium WebDriver in Java. This is useful for testing authentication, simulating different clients, or adding custom headers that your application expects. You’ll need a working Java project with Selenium WebDriver and the BrowserMob Proxy library, which acts as a local proxy to capture and alter requests.

What You’ll Need

Java Development Kit (JDK) 8 or higher
Maven or Gradle for dependency management
Selenium WebDriver Java bindings (latest stable version)
BrowserMob Proxy (com.github.detro.ghostdriver or net.lightbody.bmp) – we’ll use the net.lightbody.bmp artifact
A browser driver (e.g., ChromeDriver) matching your browser version
Basic familiarity with Java and Selenium

Step-by-Step

Add dependencies to your project. In your pom.xml (Maven) or build.gradle (Gradle), include Selenium and BrowserMob Proxy. For Maven, add:

<dependency>
  <groupId>org.seleniumhq.selenium</groupId>
  <artifactId>selenium-java</artifactId>
  <version>4.21.0</version>
</dependency>
<dependency>
  <groupId>net.lightbody.bmp</groupId>
  <artifactId>browsermob-core</artifactId>
  <version>2.1.5</version>
</dependency>

Start the BrowserMob Proxy. Create a new BrowserMobProxyServer instance and start it on a port (e.g., 8080). This proxy will listen for browser traffic.
```
BrowserMobProxyServer proxy = new BrowserMobProxyServer();
proxy.start(8080);
```
Add a request filter to modify headers. Use the addRequestFilter method to intercept requests and change or add headers. For example, to add an Authorization header:
```
proxy.addRequestFilter((request, contents, messageInfo) -> {
    request.headers().add("Authorization", "Bearer my-token");
    return null;
});
```
The filter is called for every request; you can conditionally modify based on URL, method, etc.

Configure Selenium to use the proxy. Get the proxy’s Selenium capabilities (Proxy object) and set it in your WebDriver options. For Chrome:

Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
ChromeOptions options = new ChromeOptions();
options.setProxy(seleniumProxy);
WebDriver driver = new ChromeDriver(options);

Run your test. With the driver, navigate to your target URL. The proxy will intercept all HTTP requests and apply your header modifications before they reach the server.
```
driver.get("https://example.com");
// ... your test assertions
```
Stop the proxy and quit the driver. After tests, clean up resources to avoid port leaks.
```
driver.quit();
proxy.stop();
```

Common Mistakes

Forgetting to set the proxy on the browser options. If you start the proxy but don’t configure the driver to use it, requests go direct and headers aren’t modified.
Using the wrong proxy object. BrowserMob’s Proxy class is different from Selenium’s Proxy. Use ClientUtil.createSeleniumProxy() to convert.
Not handling HTTPS traffic. If your site uses HTTPS, you need to install the proxy’s CA certificate in the browser or set up MITM (man-in-the-middle) properly. Otherwise, secure requests fail.
Modifying headers in the wrong filter. There are addRequestFilter and addResponseFilter; make sure you use the request one.
Not cleaning up resources. Leaving the proxy running can cause port conflicts in subsequent tests.

Tips

Use a unique port per test if running parallel tests to avoid conflicts.
Conditionally modify headers based on URL or request type to avoid unnecessary changes.
Log the requests to debug by adding a listener or printing in the filter.
For HTTPS, use proxy.setHarCaptureTypes() and ensure the browser trusts the proxy’s CA. You can download the CA from the proxy’s HTTP server (e.g., http://localhost:8080/CA).
If you need to remove a header, use request.headers().remove("Header-Name") in the filter.

Troubleshooting

Headers are not being modified. First, verify that the proxy is running and that the browser is actually using it. Check the proxy’s port and that the driver options are set correctly. Also, ensure your filter is added before the driver starts.

HTTPS errors or certificate warnings. This is common when intercepting secure traffic. Install the BrowserMob CA certificate into your browser’s trust store. Alternatively, you can set proxy.setTrustAllServers(true) for testing (not recommended for production).

Port already in use. If you get a BindException, choose a different port or free the existing one. You can also use port 0 to let the system pick a free port, then retrieve it with proxy.getPort().

Browser won’t start. Check that your ChromeDriver version matches your Chrome browser. Also, ensure the proxy is reachable from the browser’s perspective.

Selenium requests bypass the proxy. Some browsers have separate proxy settings for localhost. Use localhost or 127.0.0.1 explicitly. For Chrome, you may need to add --proxy-bypass-list=<-loopback> to avoid bypassing localhost.

Frequently Asked Questions

Can I modify headers for only specific URLs?

Yes, in the filter you can check the request URL using messageInfo.getUrl() and apply changes conditionally.

Does this work with all browsers?

It works with any browser that supports Selenium’s proxy capabilities, including Chrome, Firefox, and Edge. You just need to set the proxy in the respective options class.

How do I handle HTTPS requests?

You need to install the BrowserMob proxy’s CA certificate in your browser or set setTrustAllServers(true) (only for testing). Otherwise, the browser will block the connection.

Can I modify request headers after the driver is already running?

Yes, you can add filters at any time before the request is made. However, if the request is already in flight, it’s too late. Typically, you add filters before starting the browser.

Is there a way to modify headers without BrowserMob Proxy?

Yes, you can use browser-specific extensions like ModHeader, but that’s not pure Selenium. BrowserMob is the standard approach for Java.

What about performance impact?

Using a proxy adds a small overhead, but it’s usually negligible for testing. If you’re running many requests, consider reusing the proxy across tests.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →