This guide walks you through choosing the right fields when setting up a Browse AI data extraction robot. You’ll learn how to identify what data you need, map it to the page structure, and configure the fields so the robot captures clean, accurate results every time.
What You’ll Need
- A Browse AI account with an active plan (free or paid).
- The URL of the website you want to extract data from.
- Access to the page you’re targeting (make sure it’s not behind a login or paywall unless you have credentials).
- A clear idea of what data points matter for your project (e.g., product names, prices, dates).
Step-by-Step
- Open the Browse AI dashboard and click “New Robot.” Choose “Extract Data” as the robot type. You’ll be prompted to enter the URL of the page you want to scrape.
- Paste the URL and click “Continue.” Browse AI will load the page in its built-in browser. Wait for the page to fully render—this is critical because the field selector works on the live DOM.
- Click “Select Fields” to open the field mapping interface. You’ll see a preview of the webpage on the left and a field list on the right. This is where you define what to extract.
- Decide on the data structure you need. Ask yourself: Is this a single item (like one product detail page) or a list of items (like all products on a category page)? For lists, you’ll need to set up a repeating element (see Step 7). For single items, you can just add individual fields.
- Add a field by clicking “Add Field” and giving it a clear, descriptive name. Use names that match your use case, like “product_name” or “price”—not vague labels like “field1.” This makes the output CSV or JSON easier to work with later.
- Select the field on the page. After naming the field, click on the corresponding element in the page preview. Browse AI will highlight the element and suggest a selector. For example, if you want the product title, click on the title text. The tool will automatically generate a CSS selector or XPath.
- For repeating data (lists), set up a parent element. If you’re extracting multiple items from a list (e.g., all product cards on a category page), first click on one item to select it, then click “Select Parent” and choose the container that holds all the items. This tells the robot to loop through each item.
- Define fields inside the parent element. Once the parent is set, add child fields by clicking on elements within the selected item. For example, inside a product card, you might add fields for name, price, and rating. The robot will extract these for each item in the list.
- Use the “Custom” option for fields that need transformation. If you need to extract a value from an attribute (like a link from an href) or extract a part of the text, click the field’s settings and choose “Custom.” You can then specify the attribute or use regex to clean the data.
- Test each field as you go. After adding a field, click “Test” to see what the robot captures. Verify that the extracted value matches what you expect. If not, adjust the selector by clicking the element again or manually editing the selector.
- Set field types and options. For each field, you can set the data type (text, number, date, etc.) and options like “trim whitespace” or “extract from attribute.” Choose the appropriate type to ensure the data is formatted correctly in your output.
- Handle optional fields. If a field might be missing on some items (e.g., a discount price that doesn’t always appear), check the “Allow empty” option. This prevents the robot from failing when the element isn’t present.
- Review the field list and reorder if needed. The order of fields determines the column order in your CSV or JSON output. Drag and drop to arrange them logically.
- Click “Save Robot” and run a test extraction. Browse AI will run the robot on the provided URL and show you a sample of the extracted data. Review it carefully for accuracy.
- If the test looks good, set up a schedule (optional) and run the full extraction. You can choose to run it once or on a recurring basis (e.g., daily). The extracted data will be available for download in CSV, Excel, or JSON format.
Common Mistakes
- Selecting too broad or too narrow a selector. If you click on a container that includes extra elements, you’ll get unwanted data. If you click too specifically (like a single character), you’ll get partial data. Always test and refine.
- Not using a parent element for lists. If you just add a field for “price” without setting up a parent, the robot will only extract the first price on the page, not all prices.
- Ignoring dynamic content. Some pages load data via JavaScript after the initial load. If you don’t wait for the page to fully render (or use the “Wait for element” option), the robot may not find the fields.
- Using field names that are too generic. Names like “data” or “value” make it hard to understand your output. Use descriptive names that reflect the content.
- Forgetting to test on multiple pages. If the site has varying layouts (e.g., some products have a discount, others don’t), test on several URLs to ensure your selectors work universally.
Tips
- Start with a few fields and expand later. It’s easier to debug a robot with 3 fields than one with 20. Once you confirm the basic structure works, add more fields.
- Use the “Custom” option to extract from attributes. For example, to get the product URL, click on the link, then in the field settings choose “Custom” and set the attribute to “href.”
- Leverage regex for cleaning text. If the extracted text includes extra characters (like “$” or “USD”), use the regex option to strip them out. For example, replace “[^0-9.]” with “” to keep only numbers.
- Check the “Unique identifier” option for fields that are unique per item. This helps Browse AI avoid duplicates and makes the extraction more stable.
- Use the “Wait for element” setting if the page has lazy loading. This ensures the robot waits until the content is visible before extracting.
- Export a sample to see the exact output format. Before running a full extraction, export the test data and inspect it in a spreadsheet to catch any issues early.
Troubleshooting
- Robot can’t find the field. If the selector fails, try re-clicking the element. If the page uses dynamic classes, switch to a more stable selector like XPath based on the element’s text or position. You can also use the “Custom” option to manually enter a CSS selector.
- Extracted data is empty. This often happens when the element is not present on the page at the time of extraction. Check if the content requires scrolling or clicking. Use the “Wait for element” setting or add a step to scroll down.
- Data is duplicated. This can occur if the parent element is too broad and includes nested items. Try selecting a more specific parent that only contains the items you want.
- Fields are out of order in the output. Reorder the fields in the field list before running the robot. The order in the list is the order in the output.
- Robot works on one page but fails on another. The site may have different layouts for different categories. Create separate robots for each layout, or use conditional logic if Browse AI supports it (check the documentation).
- Data changes between runs. If the site updates dynamically (e.g., prices change), the robot will extract the current values. That’s expected. If you need historical data, schedule runs frequently.
Frequently Asked Questions
Can I extract data from a page that requires login?
Yes, but you’ll need to configure the robot to log in first. Browse AI allows you to set up login steps (enter username/password) before extraction. Make sure you comply with the site’s terms of service.
How do I extract data from a table?
For tables, you can either select the entire table as a parent and then define fields for each column, or use the “Table” extraction mode if available. Click on the table, and Browse AI should detect rows and columns automatically.
What is the difference between CSS selector and XPath?
CSS selectors are simpler and based on classes/IDs. XPath is more powerful and can navigate the DOM structure. If a CSS selector is too fragile, try XPath. Browse AI lets you switch between them in the field settings.
Can I extract images?
Yes. Click on the image element, then in the field settings choose “Custom” and set the attribute to “src” to get the image URL. You can also extract the alt text.
How do I handle pagination?
Browse AI has a “Pagination” option in the robot settings. You can specify the “Next” button selector, and the robot will click through pages automatically. Make sure to test on a few pages first.
Why does my extraction stop after a few items?
This could be due to rate limiting or the site blocking the bot. Try adding a delay between requests in the robot settings. Also, check if the site requires a captcha—Browse AI may handle it, but it’s not guaranteed.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
More cards in this drawer
- Tongue Escape Codes: What They Are0.20 movements/s
- Claude Code Workflow Habits Worth Trying
- HDMI 2.1 Explained: Why It Matters for Gaming
- Veeam Plugin for VMware Cloud Director: How It Works
- Best Path of Titans Server Hosting: Top Picks
- Oppo First Foldable: 120Hz Display & Snapdragon 888