AI document insights

The final evaluation reads the PDF's digital text first and uses Tesseract OCR when a page has too little text. Extracted text is split into overlapping chunks, embedded with BGE-small, and stored in a LlamaIndex in-memory index. Retrieval selects context before Mistral generates the answer.

That separation matters when investigating an error. A missing field might originate in extraction. An incomplete answer might result from retrieval discarding a passage. A wrong identifier can still be a generation error even when the correct passage was found.

I compared Tesseract, PaddleOCR, and EasyOCR on one scanned certificate. I also compared MiniLM, E5, and BGE embeddings on one document and three queries. In the embedding comparison, the language model was disabled so I could examine the retrieved chunks directly instead of confusing retrieval quality with the model's writing.

A later experiment compared a baseline retriever, a similarity threshold, and cross-encoder reranking on the same query. These were small, targeted comparisons. They helped expose behavior worth investigating rather than establish a generally superior configuration.

All three OCR engines captured the core fields on the tested certificate. What differed was how they grouped labels with values, handled the table, and covered the surrounding text. PaddleOCR kept a label and its value together; Tesseract captured more text and handled the table row.

The report records 220 words for Tesseract and 36 regions for PaddleOCR. Those are different units, so the counts are not a ranking of accuracy. The engines' confidence scores are also not a shared accuracy scale. The useful finding was a tradeoff between text coverage, grouping, and setup effort on that particular page.

I recommended combining extraction strategies because the tools handled different parts of this page well. Degraded scans and unfamiliar layouts would need their own tests before carrying that recommendation further.

The baseline answer included all four quality-control methods expected for the tested query. Adding the similarity threshold produced the same answer. Adding the reranker reduced five retrieved chunks to three, and the final answer omitted three of the four methods.

I would check answer completeness before keeping that extra stage. Retaining fewer passages looked cleaner, but in this one-query comparison it meant losing three expected methods. A broader test set would be needed to decide whether reranking helped overall.

The evaluation PDF contains two certificates of the same type. A generic question about a lot number returned a real value from the wrong certificate. A source citation can therefore be genuine while the answer still addresses the wrong document.

That failure points to a requirement for the next version: establish which document the user means before returning a value. A document identifier, page range, or clarifying question could resolve the ambiguity. Filtering only by document type would leave the same problem.

MiniLM was fastest in the recorded embedding comparison, while the capstone and final evaluation used BGE-small. The reason for that change was not recorded. I can describe the discrepancy, but I cannot reconstruct a decision that was never documented.

A useful next experiment would compare the models against the final pipeline's own input and chunking settings. The earlier comparison used a different, narrowly scoped setup.

The first evaluation reported 80% Exact Match and 60% Recall. Investigating the apparent misses exposed a wrong page label and an ambiguous lot-number question in the manually prepared answer key. After correcting the key against the PDF, the reported scores were 100% and 100% on the same five-question test set.

The correction changed how the result was scored. It did not demonstrate an improvement to the model. Keeping that distinction visible is more informative than presenting the final percentages alone.

Recall@4 was 100%, while Precision@4 was 45%. The expected evidence was found, but the selected context also contained substantial irrelevant material. A high recall result did not mean that retrieval was clean or efficient.

The reported average response time was about 292 seconds per query, compared with 88 milliseconds for retrieval. Those measurements describe different parts of the workflow. The response time was a practical limitation for interactive use, even on questions the pipeline answered correctly.