The idea in brief.
A fluent answer can conceal a missing table row. Amlok’s FinQA experiment isolates the evidence-retrieval stage and shows why finding support and generating an answer need separate evaluations. [1]
- BM25’s Recall@5 is 73.90%; its complete-evidence rate is 61.81%. These measure different things. [1]
- A missing evidence item is a retrieval failure; it is not automatically a wrong final answer.
- A review interface should expose the source, calculation, and unresolved evidence gaps.
Measurements are reproduced from Amlok research release 0.3. They concern lexical retrieval on FinQA, not an end-to-end language model, a lending decision, or a comparison with frontier models.
Ask a narrower question first
The study uses all 1,147 FinQA test questions across 380 source filenames. Candidate evidence consists of the supplied report’s sentences and table rows, with column headings retained. Rankers receive the question and evidence, while reference answers and gold evidence identifiers are withheld until scoring. [1][2]
This design asks whether the necessary support appears near the top of the retrieved set. It does not ask whether a language model can compute the answer, interpret the accounting treatment, or make a credit decision. Keeping the task narrow makes the result easier to reproduce and harder to overclaim.
Two metrics expose two failure modes
Recall@5 averages the fraction of annotated evidence retrieved for each question. Complete evidence@5 asks whether every annotated item is present. A method can perform reasonably on average recall while still leaving many questions with at least one missing input.
BM25 reaches 73.90% mean Recall@5 and 61.81% complete evidence@5. The paired Recall@5 gain over TF-IDF is 4.31 percentage points, with a 95% source-cluster bootstrap interval of 2.37–6.22 points. The uncertainty describes these fixed methods on this dataset, not every financial document a production system might encounter. [3]
Average coverage versus complete support
View figure data
| Measure | Value |
|---|---|
| Mean Recall@5 | 73.90% |
| Complete evidence@5 | 61.81% |
Make the missing input visible
Imagine a calculation needs operating income and interest expense. If retrieval finds the former but not the latter, a reviewable system should identify the missing input rather than quietly substitute an unrelated figure. The source period and unit matter too: a correct number from the wrong year can produce a convincing but invalid ratio.
The interface should show the evidence used, the formula applied, and any unresolved assumption. A reviewer should be able to open the original context, challenge a classification, and preserve a correction. This turns review from an impression of whether the prose sounds plausible into an inspection of a claim. [4]
Evaluate each stage and the whole workflow
A practical test plan separates extraction, retrieval, calculation, explanation, and the institutional response. Each stage needs an error definition. End-to-end evaluation still matters, but a single aggregate score cannot explain which component failed or how to fix it.
Amlok’s model card explicitly distinguishes completed baselines from proposed specialist-model and frontier-model evaluations. A stronger future result should preserve that separation: publish the task, source construction, comparator, uncertainty, and failure examples. Reviewability is a property of the evidence trail, not a synonym for confident language. [5]
Sources & methodology.
Figures identify their population, period, and units. Scenarios marked illustrative are Amlok calculations using stated assumptions, not observed customer outcomes.
- 1Real-data training and benchmark resultsAmlok research and product documentation
Research release 0.3. Separate SBA charge-off and FinQA evidence-retrieval experiments; self-reported reproducible research, not independent product validation.
- 2FinQA: A Dataset of Numerical Reasoning over Financial DataChen et al. · EMNLP 2021
Original benchmark paper. Amlok’s local evidence construction differs from other published evaluation pipelines.
- 3FinQA retrieval result manifestAmlok · downloadable JSON
Machine-readable lexical retrieval results; paired uncertainty uses source-filename clusters.
- 4Financial analysis and evidenceAmlok research and product documentation
Definitions, reconciliation, and analytical methods; implementation guidance rather than measured portfolio performance.
- 5Research model cardAmlok research and product documentation
Intended uses, exclusions, status of model development, and limits to generalization.
