OSA does not read the whole PDF for every question. It cuts the report into roughly 500-character
chunks, and for each question it only looks at the 5 chunks (top_k = 5) whose wording
is most similar to the question. It never sees the rest of the report. That is what keeps it fast and
affordable on a 300-page annual report instead of re-reading everything every time.
“Most similar” is decided by comparing the question and every chunk as vectors (an embedding). This
favours chunks that share vocabulary with the question. The stock TCFD question asks about
targets to understand and manage climate-related risks and opportunities, which is abstract,
compliance-style language. Conti’s actual numbers sit in a table about Scope 1/2
intensity targets, base year, interim milestones, which is concrete, metric-style language.
The two don’t share much vocabulary, so the embedding model ranked that table around position
18 to 52 out of hundreds of chunks, not in the top 5.
The report was read, chunked, and embedded correctly. top_k = 5 is just OSA's default
setting, a fixed number of chunks to look at, picked to keep the tool fast and cheap on typical reports
and questions. It is not tuned per report or per question, so on a report this dense and a question
this abstract, 5 is too few. Fixing it means turning two existing dials, top_k and the question
wording, or adding a second pass that re-checks a wider set for usefulness.