Skip to content

[ADD] ai_document_extraction - #96

Open
volkantasci wants to merge 58 commits into
OCA:19.0from
volkantasci:19.0-ai-document-extraction
Open

[ADD] ai_document_extraction#96
volkantasci wants to merge 58 commits into
OCA:19.0from
volkantasci:19.0-ai-document-extraction

Conversation

@volkantasci

Copy link
Copy Markdown

Add the ai_document_extraction module: local OCR (PaddleOCR) + LLM (Ollama) invoice data extraction into draft account.move records.

Features

  • OpenCV image pre-processing (grayscale, CLAHE, denoise, Otsu, resize)
  • PaddleOCR with layout tags ([HEADER]/[BODY]/[FOOTER] from bounding boxes)
  • OpenAI-compatible LLM extraction (qwen3:4b via Ollama) with a strict JSON schema, logo/header guard and hallucination controls (no math, null for unknown)
  • Async processing through queue_job (non-blocking UI)
  • rapidfuzz fuzzy partner matching (threshold configurable)
  • Extraction wizard for the partner-fallback flow
  • Configurable via res.config.settings (API base URL, model, OCR language, match threshold)
  • Full i18n (POT) and OCA scaffolding (readme fragments, license, security)

Notes

  • External deps (paddleocr, rapidfuzz, etc.) are declared in the manifest / repo requirements.txt; cv2 is imported lazily and documented in readme/INSTALL.md since cv2 is not a valid PyPI distribution name for CI.
  • Tests mock the OCR/LLM at the service boundary, so CI needs no model downloads.
  • Depends on queue_job (OCA/queue, 19.0).

@OCA-git-bot OCA-git-bot added series:19.0 mod:ai_document_extraction Module ai_document_extraction labels Aug 6, 2026
@volkantasci

Copy link
Copy Markdown
Author

Updated the documentation (commit 1ee7c6e) to match the vision-LLM pipeline already implemented in the code: INSTALL.md now lists Pillow/pdf2image/requests instead of the removed PaddleOCR/OpenCV dependencies, and the readme no longer claims OCR is used. Describes running a vision model via Ollama (qwen3-vl) or a cloud provider (OpenRouter).

@volkantasci

Copy link
Copy Markdown
Author

Hi @etobella and @angelmoya 👋

Just a friendly ping — when you have a moment, could you take a look at this PR whenever convenient? All CI checks are green (pre-commit, Odoo/OCB tests, codecov, runboat).

Happy to adjust anything based on your feedback. Thanks so much for your time and all the maintenance work!

@etobella

Copy link
Copy Markdown
Member

Interesting, however, why don't you use https://github.com/OCA/ai/tree/18.0/ai_connection ? This way we can reuse code and so on.

@volkantasci
volkantasci force-pushed the 19.0-ai-document-extraction branch from 3f64a21 to 0c759e6 Compare August 13, 2026 15:23
@volkantasci

Copy link
Copy Markdown
Author

Hi @etobella, thanks for the suggestion! Following up on your feedback:

We ported ai_connection (and its dependency ai_tool) to 19.0 in a separate PR — #100 — and refactored ai_document_extraction to use it:

  • Provider config (URL / model / API key) now lives on ai.connection records via a new openai_compatible kind, with an AiOpenAICompatibleClient (OpenAI-compatible /chat/completions, Ollama num_ctx/keep_alive auto-detected from the URL).
  • res.config.settings now links an AI Connection instead of storing API fields; extraction-only settings (fuzzy_match_threshold) remain.
  • The extraction job routes through connection._run(...); retry behavior on empty/unparseable responses is preserved.

All 85 tests pass across ai_tool, ai_connection, and ai_document_extraction; pre-commit and CI are green. Happy to adjust anything else.

@OCA-git-bot OCA-git-bot added mod:ai_tool Module ai_tool mod:ai_connection Module ai_connection labels Aug 13, 2026
@volkantasci

Copy link
Copy Markdown
Author

Quick update on the final UX since my previous comment (relevant for review):

  • The AI configuration no longer lives in General Settings: it is a small settings form under Settings > Technical > AI > AI Document Extraction, right next to AI Connection. It links the connection to use and the partner-match threshold, and keeps General Settings clean.
  • Fixed an Odoo 19 edge case: get_values() returning integer 0 for the unset ai_connection_id Many2one created a phantom record that crashed the settings onchange with 'int' object has no attribute 'origin'. It now returns False and is covered by a regression test.
  • The settings transient follows the OCA convention and lives in wizards/.

All green: 78 local tests, CI (pre-commit, Odoo/OCB, codecov, runboat).

@volkantasci

Copy link
Copy Markdown
Author

Hi @etobella 👋 gentle follow-up — whenever you have a moment.

Where things stand: per your suggestion, ai_connection (and ai_tool) has been ported to 19.0 in #100, and this PR now builds on top of it — provider config on ai.connection records, the extraction job routed through connection._run(...), settings linked to an ai.connection. All CI is green.

This module is the foundation of a broader AI-integration effort we're preparing: several more OCA modules will consume the same connection layer, with new features arriving as separate PRs and branches. Your review direction here would be especially valuable — the sooner the design is aligned, the easier those follow-ups will be to review.

Happy to adjust anything. Thanks for your time!

@volkantasci

Copy link
Copy Markdown
Author

Hi @etobella 👋

Per your suggestion, ai_connection/ai_tool are now in #100 and this PR has been refactored to use connection._run(...) (provider config on ai.connection, settings moved to Technical > AI). All 85 tests green.

#100 is the blocker for a clean rebase here - once it is merged I will rebase this one to remove the duplicated commits. Happy to adjust anything else in the meantime.

Thanks!

@volkantasci volkantasci reopened this Aug 30, 2026
@volkantasci

Copy link
Copy Markdown
Author

Note: #100 was restructured per the OCA migration guidelines into two per-module migration PRs: #110 (ai_tool) and #111 (ai_connection, stacked). I will rebase this PR once both merge.

…ted files, external deps)

- Drop cv2 from external_dependencies.python (not a valid PyPI name and not
  in manifestoo EXTERNAL_DEPENDENCIES_MAP; would break OCA CI pip install).
  Documented in readme/INSTALL.md instead.
- Replace per-module requirements.txt with readme/INSTALL.md (19.0 CI
  generates a repo-level requirements.txt from external_dependencies).
- Commit files generated by pre-commit: pyproject.toml (whool), README.rst,
  static/description/index.html, repo requirements.txt.
- Prettier reformat of empty view XMLs; add development_status Alpha.
- _parse_json_response uses raw_decode to find the first valid JSON object,
  ignoring trailing prose with extra braces and markdown code fences.
- Add tests for extract_invoice_data (mocked requests) covering the
  Authorization header behavior.
- _ai_prepare_image cleans up the temp file on render failure
- _ai_set_untaxed_line scopes account lookup to the move's company (company_ids)
- Add tests for action_extract_with_ai guards (non-draft, wrong move_type,
  missing attachment) and the enqueue path
@volkantasci
volkantasci force-pushed the 19.0-ai-document-extraction branch from 512d23f to 985f034 Compare September 4, 2026 14:20
@OCA-git-bot OCA-git-bot removed mod:ai_tool Module ai_tool mod:ai_connection Module ai_connection labels Sep 4, 2026
@volkantasci

Copy link
Copy Markdown
Author

Conflict resolved.

Rebased onto the current 19.0 tip and removed two commits that did not belong to this PR:

The branch now contains only ai_document_extraction (plus the repo-level requirements.txt / checklog-odoo.cfg / eslint.config.cjs adjustments it needs).

Note on CI: the test jobs currently fail only because ai_connection 19.0 has not been released yet (Could not find a version that satisfies odoo-addon-ai_connection==19.0.*). This PR depends on #111 — once that merges and ai_connection is available, the install/tests here should go green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mod:ai_document_extraction Module ai_document_extraction series:19.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants