Browse resources
Evaluating Synthesis Accuracy
prepared by Albert Carter & Dominik Łasiński, 10 August 2026
The Synthesis Methodology explains how Watchtower turns a bank policy document into a structured list of commitments. This evaluation methodology explains how Watchtower verifies whether that structured list is actually correct: how a synthesis is measured against a reference list, and what those measurements do and do not tell us.
When evaluating a synthesis, Watchtower compares it against a list of commitments generated by human bank analysts, following the same scope and splitting rules as synthesis itself. Data scientists call these human-generated references ground truth. Ground truth belongs to a specific document or document set, not to a bank in general. It can only be used to check a synthesis of the same source material.
How a comparison works
A synthesis is evaluated by comparing its list of commitments against the ground-truth list, not by re-reading the source document. The comparison happens in two stages.
In the first stage, we ask a large language model (often referred to as an LLM, or chatbot) to align the two lists: for each ground-truth commitment, is there a predicted commitment describing the same rule? The LLM sees only the supporting quotation for each commitment, tagged with its value-chain segment (upstream, midstream, downstream). It does not see thresholds, exceptions, or any other structured field at this stage. The alignment question is deliberately narrow: does this predicted rule describe the same restriction as this ground-truth rule, regardless of how the wording or caveats differ?
The LLM returns one decision per ground-truth commitment: matched to a specific predicted commitment, or missed. Any predicted commitment left over is an extra. This is a full match-or-no-match decision, not a graded one. A low-confidence pairing counts exactly the same as a high-confidence one, with no threshold either has to clear.
In the second stage, deterministic rules, not an LLM, compare the structured fields on each matched pair, using exact equality after light normalization (case and whitespace differences are ignored; single item lists are treated the same as the equivalent plain value).
Twelve scope fields are checked this way for every matched pair, covering the same categories described in the Commitment fields table: industry (theme, coal or oil/gas subtype, value chain, activity stage), geography (global or not), counterparty (level and relationship stage), and product (financing type, capital markets inclusion, transaction stage).
Thresholds and exceptions are checked more narrowly. For thresholds, we compare which metrics appear (for example, a revenue-share threshold versus a production threshold) and how many of each, not the specific numeric value or inequality attached to them. For exceptions, we compare how many non-empty exceptions each side records, not what the exceptions actually say. This is a deliberate scope decision, not an oversight: threshold values and exception language are highly varied in ways that resist fixed rules. As a result, we check that the LLM found the right number of the right type of thing, and rely on human review for correctness.
What the resulting numbers mean
Two commitment-discovery metrics come directly out of the alignment stage:
- Recall: of everything in the ground truth, how much did the synthesis find? Low recall means the synthesis missed commitments the policy actually contains.
- Precision: of everything the synthesis produced, how much was expected? Low precision can mean genuine over-extraction, but can equally mean the synthesis split a commitment differently than the ground truth did, or that the ground truth itself is incomplete for that document.
A further set of metrics apply only to matched pairs, so they say nothing about commitments that were missed entirely:
- Field accuracy: across the twelve scope fields, what share of matched pairs have the same value? This can be high even when recall is low, because it is calculated only over the pairs that were successfully matched in the first place.
- Threshold recall and precision and exception-count accuracy: the same idea applied to thresholds and exceptions, checking presence and count rather than content.
None of these numbers is meaningful in isolation, because each depends on the ones before it. High field accuracy scores indicate well matched commitments, not missed ones. High recall scores indicate the alignment step found a partner for most ground-truth commitments, not that the pairing was correct. The alignment step is asked to produce a one-to-one mapping, but nothing in the process enforces this, so the same predicted commitment can occasionally be matched to two different ground-truth commitments. When that happens, recall and precision figures can look artificially strong (in one case, apparent precision exceeded 100%), even though the underlying alignment is invalid. For this reason, we treat the audit output, which shows every individual pairing and not just the summary numbers, as the primary evidence, and the aggregate percentages as a summary of it.
Two worked examples
Here are the real results of two synthesis evaluations: a Barclays Climate Change Statement (37 ground-truth commitments) and a Santander Environmental, Social & Climate Change Risk Management Policy (15 ground-truth commitments). Both used our standard alignment LLM and the full audit.
| Result | Barclays | Santander |
|---|---|---|
| Recall | 100.0% | 100.0% |
| Precision | 108.8% | 93.8% |
| Overall field accuracy | 88.3% | 87.2% |
| Threshold recall | 60.0% | 85.7% |
| Exception-count accuracy | 45.9% | 73.3% |
A score that exceeds 100% indicates that the number of matched commitments has to be higher than the number of predicted commitments. This happens when the LLM assigns the same predicted commitment to more than one ground-truth commitment.
Examining the Santander result indicated a smaller version of the same problem: one predicted commitment was assigned to two different ground-truth commitments, despite an otherwise plausible-looking set of numbers. While both matches are individually reasonable, the duplication makes the alignment invalid. Both cases show why we read the underlying pairings before treating the summary percentages as a finding, and why match confidence alone is not a reliable signal that a pairing is correct.
Taken together with the recall and accuracy figures reported in the Synthesis Methodology, this evaluation process is how Watchtower verifies those figures rather than simply asserting them. It is why the detailed audit, not the summary percentage, is treated as the primary output of any single evaluation run.
What this evaluation does not tell us
Consistent with how we describe synthesis itself, we want to be precise about the boundaries of what this checking process measures.
The automated checks cover an explicit, limited set of fields. Beyond the twelve scope fields, threshold-metric presence, and exception counts, several fields recorded in a synthesis are not automatically scored: the exact wording of threshold values and inequalities, the substance of exception language, certainty ratings the LLM attaches to its own extractions, and the specific text supporting each field. These are shown in the audit, but graded correctness is left to human review. Most of this is because these fields contain free text that does not reduce well to exact-match comparison, or because they were added to the synthesis schema after this comparison structure was built, a known gap we expect to narrow over time, not a permanent design choice.
Alignment is not stored or reused. Every evaluation run asks the LLM to align the two lists again, so the same synthesis and ground truth can receive a different pairing on a different run. Field comparison is deterministic once a pairing exists, but the pairing itself is not guaranteed to be stable or, as the worked examples above show, guaranteed to be valid.
We are pleased with the results of our synthesis. However, at the time of this writing, there is ongoing work to improve synthesis scoring, particularly around exception accuracy and taxonomy. While the above example is real, it will likely have become out of date by the time you read it.