Regex Data Extractor

A Streamlit application that applies named regex patterns to a text column in any CSV file and returns structured extraction results. Thirteen built-in patterns cover common document fields out of the box. Custom patterns can be added at runtime without editing any code. Extraction logic lives in pure Python modules, tested independently of the UI.

Business Problem

Contracts Specialists and Contracting Officer's Representatives (COR) who work with contract administration systems routinely store critical structured data — invoice numbers, contract identifiers, dollar amounts, contact information, and performance dates — embedded in free-text notes fields. A modification record may reference the original contract number, the revised amount, and the new completion date in a single prose sentence. A vendor note may contain an email address, a PO number, and a dollar figure with no consistent formatting.

Extracting those structured fields by hand — opening each record, reading the text, identifying the pattern, and copying the value into a separate column — is error-prone, inconsistent between analysts, and scales poorly across hundreds or thousands of records. The problem compounds during audits, when a Contracts Specialist must verify that all task order records cite a valid federal contract number in the prescribed format (GS-10F-XXXXXX), or that every invoice note references a matching INV-XXXXX identifier before the payment is approved.

This tool ingests any text-column CSV, runs a configurable library of named regex patterns against every row, computes per-pattern match statistics across the full dataset, and produces a structured export with one column per extracted field — turning a free-text audit into a structured data verification task in seconds, with no code required.

Key Features

Feature	Description
13 built-in patterns	Email, US phone, dollar amounts, ISO dates (YYYY-MM-DD), US dates (M/D/YYYY), ZIP codes, percentages, federal contract numbers (GS-10F-XXXXXX), invoice numbers (INV-XXXXX), PO numbers (PO-XXXXXXXX), HTTP/S URLs, IPv4 addresses, masked SSNs
Runtime custom patterns	Add any valid Python regex via the sidebar using name = pattern format; invalid syntax reported in sidebar without stopping the app
Three extraction modes	first — first match per row (default); all — list of all matches per row; count — match count per row
Coverage statistics	Per-pattern metrics: total rows, rows matched, match rate %, total matches, unique matches — sorted by match rate descending
Pattern drill-down	Select any pattern to see its regex, row hit count, top-15 most frequent matches (bar chart), and sample matching rows
Extracted fields table	Source column + one column per active pattern showing first match per row; matched cells highlighted
CSV and JSON export	Full dataset with extracted columns (extracted_data.csv), coverage report (coverage_report.csv), active pattern config (patterns.json)
Caching	@st.cache_data on data loading and extraction; re-selecting patterns does not reprocess the full file

Skills Demonstrated

Skill	Implementation
Named pattern registry	src/patterns.py: BUILTIN_PATTERNS dict — 13 entries; PATTERN_DESCRIPTIONS for display; compile() wrapper defaults to re.IGNORECASE
Runtime pattern validation	src/patterns.py: validate_pattern() — re.compile() probe; returns (is_valid, error_message) tuple; used in app sidebar before adding pattern to active set
Multi-pattern extraction	src/extractor.py: extract_patterns() — iterates pattern dict; dispatches to extract_first, extract_all, or extract_count per mode; raises ValueError on invalid mode or missing column
Coverage analysis	src/extractor.py: coverage_stats() — per-pattern extract_all, flattens to match list, computes rows_matched via boolean sum, sorts by match_rate_% descending
Frequency ranking	src/extractor.py: top_matches() — flattens all matches, pd.Series.value_counts().head(n), lowercase-normalised before counting
Export pipeline	src/reporter.py: build_export_df() — original DataFrame + extracted_<name> column per pattern; no-match rows filled with empty string (not NaN) for clean CSV export
Multi-tab Streamlit UI	app/main.py — four tabs: Pattern Coverage (Plotly horizontal bar), Drill-Down (px.bar frequency), Extracted Data (styled DataFrame), Export (st.download_button ×3)
@st.cache_data composition	load_csv() cached by path/bytes; run_extraction() cached by JSON-serialized DataFrame + source_col + patterns; two-level cache prevents redundant I/O and computation

Architecture

data/sample_notes.csv ──► load_csv()
       or                      │
uploaded CSV bytes ────────────┤
                               ▼
                   extract_patterns(mode="first") ──► Extracted Data tab
                               │
                         coverage_stats()
                               │
                    ┌──────────┴───────────┐
                    ▼                      ▼
          extraction_summary()      top_matches()
                    │                      │
              Header metrics         Drill-Down tab
                               │
                         build_export_df()
                               │
                         Export tab (3 downloads)

The extraction engine (src/extractor.py) operates on pandas Series — each row's text field is processed independently by re.compile(pattern, re.IGNORECASE).search() or .findall(). The UI layer (app/main.py) composes these functions behind @st.cache_data barriers keyed on the JSON-serialized DataFrame and pattern dict, preventing full re-extraction on sidebar interactions that don't change the data.

Three extraction modes — extract_patterns() dispatches to three helpers based on mode:

Mode	Column Naming	Return Type per Row	Use Case
first	<pattern_name>	str or None	Extracted fields table; CSV export
all	<pattern_name>_all	list of str	Multi-match rows; complete capture
count	<pattern_name>_count	int	Coverage analysis; filter rows by hit count

Built-in pattern library — 13 named patterns targeting federal contracting document fields:

Pattern	Matches	Example
email	Standard email addresses	alice@agency.gov
phone	US phone (3 formats)	(555) 123-4567
dollar_amount	Dollar values with optional cents	$9,165.00
date_iso	ISO 8601 dates	2024-01-15
date_us	US short dates	1/15/2024
zip_code	5 or 9-digit ZIP	20001-1234
percent	Numeric percentages	12.5%
contract_number	Federal contract format	GS-10F-719570
invoice_number	Invoice identifiers	INV-12345
po_number	Purchase order numbers	PO-12345678
url	HTTP/HTTPS URLs	https://sam.gov/...
ipv4	IPv4 addresses (shape match)	192.168.1.1
ssn_masked	Masked SSNs	***-**-1234

See docs/ARCHITECTURE.md for full pattern documentation, extraction mode semantics, output format reference, and engineering design decisions.

Project Structure

regex-data-extractor/
├── .github/
│   └── workflows/
│       └── tests.yml             # CI: Python 3.11, ubuntu-latest, pytest
├── .streamlit/
│   └── config.toml               # Light theme, primary color #2E5FA3
├── app/
│   └── main.py                   # Streamlit dashboard — 4 tabs, sidebar controls, 3 exports
├── src/
│   ├── __init__.py               # Public API (14 exports: 12 functions + 2 constants)
│   ├── patterns.py               # BUILTIN_PATTERNS, PATTERN_DESCRIPTIONS, validate_pattern(), compile()
│   ├── extractor.py              # extract_first/all/count, extract_patterns, coverage_stats, top_matches
│   └── reporter.py               # extraction_summary, per_row_match_count, build_export_df
├── data/
│   └── sample_notes.csv          # 200 synthetic contract administration notes (5 record types)
├── docs/
│   ├── ARCHITECTURE.md           # System design, pattern library reference, output format schemas
│   ├── DATA_DICTIONARY.md        # Input schemas, API reference, export file schemas
│   ├── ENGINEERING_DECISIONS.md  # Six annotated architecture decisions with trade-offs
│   └── TESTING.md                # Test suite structure, per-class breakdown, known gaps
├── screenshots/
│   ├── 01_overview.png           # Application overview with pattern coverage chart
│   ├── 02_core_feature.png       # Pattern drill-down and match frequency analysis
│   └── 03_results.png            # Extracted data table and export panel
├── scripts/
│   └── generate_sample_data.py   # Script that produced sample_notes.csv
├── tests/
│   ├── test_patterns.py          # 22 tests — BUILTIN_PATTERNS, validate_pattern, compile
│   ├── test_extractor.py         # 33 tests — extract_first/all/count/patterns, coverage_stats, top_matches
│   └── test_reporter.py          # 15 tests — extraction_summary, per_row_match_count, build_export_df
├── CHANGELOG.md
├── LICENSE
├── README.md
└── requirements.txt

Installation

git clone https://github.com/RichieGarafola/regex-data-extractor.git
cd regex-data-extractor

python -m venv .venv

# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

pip install -r requirements.txt
streamlit run app/main.py

The app opens at http://localhost:8501.

Usage

1. Open the app. Sample data loads automatically via the Use sample data toggle in the sidebar.
2. To analyze your own data: toggle off Use sample data and upload a CSV file.
3. Select the text column to extract from using the Text column dropdown.
4. Select built-in patterns using the multiselect (default: email, phone, dollar_amount, contract_number, invoice_number, date_iso, date_us).
5. Add custom patterns in the text area using name = pattern format, one per line:

order_id = ORD-\d{6}
ticket = TKT-[A-Z0-9]{8}

6. Navigate tabs to analyze results:
   • Pattern Coverage — horizontal bar chart of match rates (green ≥ 50%, orange ≥ 20%, red < 20%); coverage detail table with all stats
   • Drill-Down — select one pattern to see its regex string, row hit count, top-15 match frequency bar chart, and sample matching rows
   • Extracted Data — source column plus one extracted-value column per active pattern; first match per row; matched cells highlighted in blue
   • Export — download extracted_data.csv (full dataset + extracted columns), coverage_report.csv (per-pattern stats), patterns.json (active pattern configuration)

For programmatic use, the full extraction pipeline is available via the public API:

import pandas as pd
from src import BUILTIN_PATTERNS, coverage_stats, build_export_df

df = pd.read_csv("data/sample_notes.csv")
patterns = {k: BUILTIN_PATTERNS[k]
            for k in ["email", "contract_number", "invoice_number", "dollar_amount"]}

stats = coverage_stats(df, "notes", patterns)
print(stats[["pattern", "rows_matched", "match_rate_%"]])

export = build_export_df(df, "notes", patterns)
export.to_csv("extracted_output.csv", index=False)

Sample Data

data/sample_notes.csv — 200 synthetic contract administration notes across 5 record types: Contract, Invoice, Task Order, Modification, and Vendor. Each note embeds realistic structured fields — email addresses, contract numbers in the GS-10F-XXXXXX federal format, invoice references (INV-XXXXX), PO numbers, dollar amounts, ISO and US-format dates, and phone numbers — to provide realistic match coverage across the 13 built-in patterns.

Column	Type	Description
record_id	str	Unique identifier (REC-XXXX)
category	str	Record type: Contract, Invoice, Task Order, Modification, Vendor
notes	str	Free-text field with embedded structured data
reviewed	str	Review status: Yes / No / Pending

Regenerate with: python scripts/generate_sample_data.py

Testing

pytest tests/ -v

70 tests across 13 test classes in 3 files. All tests pass in under 4 seconds with no external dependencies beyond pip install -r requirements.txt. Tests do not mock any internal library functions — all assertions run against real regex matching logic with inline-constructed inputs.

tests/test_patterns.py    22 tests  — BUILTIN_PATTERNS compilation, match/no-match cases for all 13 patterns, validate_pattern, compile()
tests/test_extractor.py   33 tests  — extract_first/all/count helpers, extract_patterns (3 modes + error cases), extract_to_columns, coverage_stats, top_matches
tests/test_reporter.py    15 tests  — extraction_summary dict, per_row_match_count, build_export_df column naming and fill behavior

Pattern tests verify both positive (should match) and negative (should not match) cases for every built-in pattern — including boundary cases like PO numbers that are too short, contract numbers that are incomplete, and phone numbers that fail to form a valid 10-digit sequence.

See docs/TESTING.md for per-class test breakdown, assertion detail, and known gaps.

Limitations

• Pattern precision is not validated — the IPv4 pattern matches the shape d.d.d.d but does not reject out-of-range octets (999.0.0.1 will match)
• First-match-only in the exported CSV — build_export_df uses mode="first", so rows with multiple matches export only the first value
• No multi-column extraction in one session — one source column is processed at a time; scanning multiple text columns requires separate passes
• No regex flag controls in the UI — all patterns run case-insensitive; multiline or dotall behavior cannot be enabled without editing src/patterns.py
• Phone number false positives — the phone pattern matches any 10-digit sequence and may match serial numbers or other numeric strings of the same shape
• Large files — @st.cache_data serializes to JSON before caching; files above ~100 MB may be slow to process in-browser
• Custom pattern names must not collide with built-in names — a custom pattern with the same name as a built-in pattern silently overrides it

Roadmap

• Export all matches per row (not just the first) as a long-format CSV
• Regex flag toggles in the UI (case sensitivity, multiline)
• Side-by-side pattern comparison to see where two patterns overlap on the same column
• Saved pattern sets loadable from a JSON file
• Batch processing across multiple text columns in a single session
• Match highlighting within the source text (show which substring was captured)
• Custom pattern name collision detection with warning

Screenshots

Overview	Core Feature	Results
Application overview with pattern coverage chart	Pattern drill-down and match frequency analysis	Extracted data table and export panel

---

Documentation

Document	Description
Architecture and Design Decisions	System design, pattern library reference, three extraction modes, and output format schemas
Engineering Decisions	Six annotated architecture decisions with alternatives and trade-off rationale
Test Inventory and Coverage	Test suite structure, per-class breakdown, known gaps, and testing philosophy
Data Dictionary and API Reference	Input column schemas, constants, public API function reference, and export file schemas

License

MIT — see LICENSE