Case study

Continental’s climate targets were in the report. Embedding search still missed them.

Under OSA's default top_k = 5, the stock TCFD targets question ranked Conti’s intensity numbers outside top-5. Raising top_k, rephrasing the question, and LLM scoring each recover it.

Continental AG GB 2025 TCFD tcfd_11

1 · Miss

Same PDF, same TCFD question, different evidence

Asked against Continental’s 2025 annual report, OSA cited management / IRO text and scored the disclosure poorly. A full-document chat on the same PDF found the public intensity targets: 0.13 tCO₂e/t by 2035, interim 0.17 by 2030, 2025 performance 0.20.

OSA · embedding only

tcfd_11

Score 3.0

The report does not provide quantifiable targets for understanding climate-related risks and opportunities. It describes approaches and identified IROs, but no numerical targets or performance tracking.

Cited

  • Management approach for Scope 1 / 2
  • Identified impacts, risks and opportunities

Gaps

  • Numerical emissions targets
  • Performance tracking metrics

Langdock reference

A full-document chat tool (Langdock) on the same PDF surfaces the intensity targets directly on the first try. It just works here, but there's no way to see what it skipped to get there, or to adjust its search. More on that below.

Langdock reference: chat tool table showing Continental Scope 1 and 2 intensity targets

2 · Why

OSA never saw the right page

OSA does not read the whole PDF for every question. It cuts the report into roughly 500-character chunks, and for each question it only looks at the 5 chunks (top_k = 5) whose wording is most similar to the question. It never sees the rest of the report. That is what keeps it fast and affordable on a 300-page annual report instead of re-reading everything every time.

“Most similar” is decided by comparing the question and every chunk as vectors (an embedding). This favours chunks that share vocabulary with the question. The stock TCFD question asks about targets to understand and manage climate-related risks and opportunities, which is abstract, compliance-style language. Conti’s actual numbers sit in a table about Scope 1/2 intensity targets, base year, interim milestones, which is concrete, metric-style language. The two don’t share much vocabulary, so the embedding model ranked that table around position 18 to 52 out of hundreds of chunks, not in the top 5.

The report was read, chunked, and embedded correctly. top_k = 5 is just OSA's default setting, a fixed number of chunks to look at, picked to keep the tool fast and cheap on typical reports and questions. It is not tuned per report or per question, so on a report this dense and a question this abstract, 5 is too few. Fixing it means turning two existing dials, top_k and the question wording, or adding a second pass that re-checks a wider set for usefulness.

OSA defaults: top K 5, chunk size 500, optional LLM scoring
Defaults that produced the miss. LLM scoring is the second stage after retrieval.
Old prompt, stock TCFD wording, ranked the target chunk 52nd. New prompt, emissions-targets wording, ranked the same chunk 1st, on the same report and the same top_k
Same report, same top_k. The old prompt (stock TCFD wording) ranked the target chunk 52nd. The new prompt (emissions-targets wording) ranked it 1st. Question phrasing still matters. The scorer only reorders what retrieval already returned.

3 · First fix

Raise top_k before anything else

Since the intensity-target chunk was ranked around position 18 to 52, the simplest fix is to look further down the list: raise top_k from 5 to 15 to 20 for dense annual reports. That alone gets the right chunk into the pool OSA actually reads, on this report and the others we tested below. It costs a few more tokens per question. It does not require touching the question wording or the LLM scorer.

Raising top_k buys recall. More of the relevant passages make it into the pool. It also costs precision. More noise comes along with them. That is the classic precision-recall trade-off from ClimRetrieve. Across prompt variants, Recall@K roughly doubles from top_k = 5 to top_k = 15, while Precision@K falls over the same range. Conti's intensity targets sitting at rank 18 are exactly that pattern.

ClimRetrieve figures: Recall@K rises from top_k 5 to 15 while Precision@K falls over the same range
ClimRetrieve (text-embedding-3-large). Top row: Recall@K rises as K goes from 5 to 15. Bottom row: Precision@K falls over the same range. Raising top_k recovers more evidence and dilutes the pool.

That drop in precision is the reason OSA has an LLM scorer. Once you look at 15 or 20 chunks instead of 5, somebody has to decide which of those chunks are actually useful. The built-in scorer does that by default. If your analysts use a different definition of useful evidence, that is also why we offer a fine-tuned scorer trained on your labels. Rephrasing the question can shrink the gap before scoring. Scoring and fine-tuning are how you keep the recall you just bought without drowning in noise.

3 · Second fix

Match the question's vocabulary to the report's vocabulary

If you keep top_k low, rephrasing the question toward the report's own wording moves the answer up the ranking without reading more chunks. We tested four phrasings of the same TCFD question on Continental and four more reports, and tracked where the correct passage first appeared. Lower is better. At top_k = 5, the passage needs to rank between 1 and 5.

Original (stock TCFD wording)
“What targets does the organisation use to understand/quantify/benchmark climate-related risks and opportunities? How is the organization performing against these targets?”
Emissions targets
“What greenhouse gas emissions targets has the organisation set for Scope 1, Scope 2 and Scope 3? Include absolute and intensity targets, base year, interim milestones and target years.”
CO2 intensity
“What CO2 or GHG intensity and absolute reduction targets does the organisation disclose for Scope-1 and Scope-2 emissions, including interim targets and base year?”
Short targets
“What are the organisation's climate change mitigation targets and how ambitious are they?”
Report Original Emissions targets CO2 intensity Short targets
Continental 52 1 1 7
Siemens 21 5 7 3
Befesa 3 3 1 1
Kyocera 2 2 1 2
PORR 5 1 2 3

Best embedding rank of the correct passage per question phrasing, top_k = 20 pool. Red marks ranks outside top-15 at stock wording. The table shows the stock wording was the one consistent problem. No single rephrase is safe for every report. Continental's own best phrasing is Siemens's second-worst.

3 · Third fix

Add LLM scoring to clean up the wider pool

The ClimRetrieve trade-off above is why the scorer exists. Raising top_k and rephrasing get the right chunk into the candidate pool. The wider pool also brings noise. LLM scoring is the pass that buys precision back: score each of the (say) 20 retrieved passages against the question, re-order by score, then answer only on the top of that list. On Conti with the stock question and a top-20 pool, this moved the intensity-target passage from embedding #18 to LLM #1, without changing the question or the top_k setting.

Before: top-5 embedding ranking misses Conti intensity targets, which actually sit at rank 18. After: LLM scoring promotes them to rank 1 in the same pool
Left: what top-5 saw without scoring. Right: same top-20 pool after the scorer restores precision.

Fine-tune the scorer on your labels

The built-in scorer recovers precision with a generic definition of useful evidence. If your rating methodology, audit checklist, or sector vocabulary differs, a generic scorer will keep promoting the wrong kind of noise. That is when you book a fine-tuning engagement: we train an open-source scorer on your labelled passages and plug it into OSA, so the precision you buy back matches how your team actually judges evidence. See DIRAS / UsefulBench for the research behind company-aligned scoring. Get in touch below to start.

Get in touch

A note on chat tools

Why not just paste the PDF into Langdock or Claude?

Sometimes they just work. The Langdock reference above found Conti's intensity targets on the first try, no configuration needed. But for a sustainability analysis you need to be able to defend later, that first-try success is not enough on its own:

  • You cannot measure what it skipped. A general chat tool doesn't tell you it read page 131 and ignored page 212. You only see the answer it chose to give you.
  • You cannot adjust how it searches. There is no top_k, no chunk size, no scorer to tune when the first answer is wrong or incomplete on a harder report.
  • You cannot see the code. For an audit trail, a rating methodology, or a regulatory filing, "a closed model said so" is not a reproducible method.

OSA is slower to set up for exactly this reason: top_k, chunk size, question wording and scoring are all visible and adjustable, and every answer comes with the source chunks it was built from. That is what lets you measure a miss like this one and fix it, instead of hoping the next report behaves.

4 · Takeaways

  1. Be careful with top_k. The default is a guess, not a guarantee, and it is often too shallow for dense reports and abstract TCFD wording.
  2. Check more than one report. Look for structural differences, like numbers sitting in tables instead of prose, that change where retrieval needs to look.
  3. Adjust your question set to your set of reports. Match the report's own vocabulary instead of standardizing one generic phrasing across every company. See customize the framework.
  4. Once retrieval is solid, use LLM scoring to buy precision back on the wider pool. Scoring cannot recover a chunk that retrieval never returned.
  5. Book a fine-tuning engagement if you want a scorer trained on your own definition of useful evidence, so the precision you recover matches how your team judges passages.

Try it or talk to us

Deploy OSA with LLM scoring, or get in touch to talk through fine-tuning. Background and related research live on the Climate+Tech site.