keepcalculator

06 Numbers and Powers

Extract Invoice Data from PDFs with Python & Validate

Card No. 06·71436 · Updated · Standard formula, shown below

This guide shows you how to extract structured data—like invoice numbers, dates, line items, and totals—from PDF invoices using Python, then validate that extracted data for accuracy and completeness. You’ll learn a practical pipeline using popular libraries like pdfplumber for text extraction and pandas for validation, with clear steps you can adapt to your own invoice formats.

What You’ll Need

  • Python 3.8 or newer installed on your system.
  • A PDF invoice file (or several) to test with.
  • Install required libraries: pip install pdfplumber pandas
  • Basic familiarity with Python and command line.

Step-by-Step

  1. Set up your project folder. Create a directory, e.g., invoice_extractor/, and place your PDF invoices inside a subfolder like invoices/.
  2. Write a function to extract text from a PDF. Use pdfplumber to open the PDF and concatenate text from all pages. For example:
    import pdfplumber
    
    def extract_text(pdf_path):
        with pdfplumber.open(pdf_path) as pdf:
            return "\n".join(page.extract_text() or "" for page in pdf.pages)
  3. Define regex patterns for key fields. Common invoice fields include invoice number, date, total amount, and vendor name. For instance:
    import re
    
    patterns = {
        "invoice_number": r"Invoice\s*#?\s*([A-Z0-9-]+)",
        "date": r"Date\s*[:]?\s*(\d{1,2}/\d{1,2}/\d{2,4})",
        "total": r"Total\s*[:$]?\s*([\d,]+(?:\.\d{2})?)"
    }

    Adjust these to match your invoice layout.

  4. Extract data using the patterns. Write a function that applies the regex to the extracted text and returns a dictionary of field-value pairs. Handle cases where a field is missing by setting it to None.
  5. Handle line items. If your invoices contain a table of items, use pdfplumber‘s extract_table() method. For example:
    with pdfplumber.open(pdf_path) as pdf:
        page = pdf.pages[0]
        table = page.extract_table()

    Then convert the table into a list of dictionaries with column names.

  6. Store extracted data in a structured format. Use pandas to create a DataFrame for all invoices. Each row represents one invoice with columns for each field.
  7. Validate the extracted data. Write validation checks:
    • Required fields: Ensure invoice number, date, and total are not missing.
    • Date format: Check that dates parse correctly using datetime.strptime().
    • Numeric totals: Convert total strings to floats and check they are positive.
    • Consistency: If line items are extracted, verify that the sum of line item amounts matches the total (within a small tolerance).
  8. Output the results. Save the validated data to a CSV or Excel file using df.to_csv() or df.to_excel(). Include a column for validation status (e.g., “Pass” or “Fail”) and error messages.
  9. Test on multiple invoices. Run your script on a sample of invoices to see how well it handles variations. Adjust regex and table extraction as needed.

Common Mistakes

  • Hardcoding regex patterns that are too specific. Invoice formats vary; use flexible patterns and test on multiple samples.
  • Ignoring multi-page invoices. Some invoices span multiple pages; make sure to extract text from all pages, not just the first.
  • Assuming tables are always perfectly structured. PDF tables often have merged cells or missing borders; use extract_table() with caution and clean the output.
  • Not handling missing data gracefully. Your code should not crash when a field is absent; return None and flag it.
  • Forgetting to strip whitespace and special characters. Extracted text may contain extra spaces or newlines; clean strings before parsing.

Tips

  • Use pdfplumber‘s extract_words() to get word-level data if you need to locate fields by position.
  • For complex layouts, consider using camelot or tabula-py for table extraction, but they require Java or external dependencies.
  • Always keep a backup of the original PDFs; extraction is not lossless.
  • Use logging to track which invoices failed validation and why.
  • For recurring invoices from the same vendor, you can create a template with fixed coordinates for faster extraction.

Troubleshooting

  • No text extracted: The PDF may be scanned (image-based). Use OCR with pytesseract or pdf2image to convert pages to images first.
  • Regex not matching: Print the extracted text to see the exact format. Use regex101.com to test patterns.
  • Table extraction returns None: Try page.find_tables() to see if tables are detected. Adjust table settings like vertical_strategy.
  • Date parsing fails: Some invoices use formats like “2023-01-15” or “Jan 15, 2023”. Write a function that tries multiple formats.
  • Total mismatch with line items: This could be due to discounts, taxes, or rounding. Include a tolerance (e.g., 0.01) and flag only significant discrepancies.
  • Memory issues with many PDFs: Process invoices one by one and store results incrementally rather than loading all at once.

Frequently Asked Questions

Can I extract data from scanned invoices?

Yes, but you need OCR. Use pytesseract with Tesseract installed, or cloud OCR services. The accuracy depends on image quality.

How do I handle invoices with different layouts?

Create a set of regex patterns and table extraction strategies for each layout, then detect the layout based on keywords or structure. Alternatively, use a machine learning approach with training data.

What if the invoice has no line items, just a total?

That’s fine; your validation should only check required fields. Adjust your extraction to skip table parsing if no table is found.

Can I use this for currencies other than USD?

Yes, but adjust the regex to match currency symbols and decimal separators. For example, European formats use commas for decimals.

Is pdfplumber the best library for this?

It’s a good open-source choice for text-based PDFs. For complex tables, consider camelot. For scanned docs, use OCR tools.

How do I validate that the total is correct?

Sum the line item amounts and compare to the total field. Allow a small tolerance for rounding. If they don’t match, flag it for manual review.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →