Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

How does “Citation Coverage” actually work in GenAI Observability? The LLM judge prompt does NOT include citations.

0

I’m trying to understand how Citation Coverage is actually computed in GenAI Observability.

The documentation says citation coverage measures how much of the answer is supported by cited passages, which implies the evaluator needs:

  1. Citations inside the candidate answer (e.g., [1], [2]) OR
  2. A mapping between citations and retrieved passages

But when I inspect the actual LLM-judge prompt used by Observability, it only includes:

  • Related Passages (retrieved KB text)
  • Candidate Response

There are no citation markers, no citation list, and no passage IDs. Yet the evaluator still produces outputs like “Citation coverage score: 0” or “1,” and the explanations refer to “cited passages,” even though none were provided.

Also, the UI shows:

“Knowledge Base did not retrieve passages for this input. This prompt/response pair will not be included in Citation coverage scoring.”

So my questions are:

  1. How is citation coverage computed when no citations are passed to the model?
  2. Are we expected to make the model output citations even though the evaluation prompt doesn’t reference them?
  3. Should both citations and retrieved passages be passed, and is this a missing piece in the default evaluation prompt?
  4. Is this a design issue or am I misunderstanding the intended workflow?

I need to know how Citation Coverage is supposed to work, given the current prompt structure.

Topics
Machine Learning & AIGenerative AI on AWS
Tags
Knowledge Bases for Amazon BedrockAmazon Bedrock
Language
English

asked 10 months ago202 views

1 Answer
0

Citation Coverage in Amazon Bedrock Evaluations measures how well a response's content is supported by its citations, focusing on whether information derived from retrieved passages has been properly cited.

The metric works by comparing the faithfulness of the response to both the cited passages and the full set of retrieved passages, then calculating their ratio. It provides a score from 0-1, where scores closer to 0 indicate that the model failed to cite relevant supporting passages, and scores closer to 1 suggest proper citation of information from the retrieved passages.

When evaluating Citation Coverage, the system expects the candidate response to include citations or references to the source material. The evaluation prompt you're seeing (with just "Related Passages" and "Candidate Response") is designed to assess how faithful the response is to the available context, which is a component of calculating citation coverage.

For Citation Coverage to work properly:

  1. Your RAG system should generate responses that include citations or references to the source passages
  2. These citations should be included in the "Candidate Response" part of the evaluation
  3. The retrieved passages should be included in the "Related Passages" section

If your responses don't include citations, the system will likely give low Citation Coverage scores, as it can't identify which parts of the response are attributed to which sources.

The message "Knowledge Base did not retrieve passages for this input" indicates that for that particular query, no passages were retrieved from the knowledge base, so Citation Coverage can't be calculated (since there's nothing to cite).

A perfect score of 1 doesn't necessarily mean all information in the response is cited—rather, it indicates that all information that could have been cited (based on the retrieved passages) was properly cited. This helps identify when RAG systems are missing opportunities to cite relevant source material.
Sources
Evaluate models or RAG systems using Amazon Bedrock Evaluations – Now generally available | AWS Machine Learning Blog
Mistral Large 1 (24.02) - Amazon Bedrock
Anthropic Claude 3.5 Haiku - Amazon Bedrock
Anthropic Claude 3.5 Sonnet - Amazon Bedrock
MetaLlama 3.1 70B Instruct - Amazon Bedrock

answered 10 months ago

AWS
SUPPORT ENGINEER

reviewed 10 months ago

  • Safant
    10 months ago

    Here is an example, where citation coverage is one: And i had 4 citations, so can i assume each response part corresponds to citations not retrived documents right? Citation coverage score: 1 To assess the citation coverage (how much of the information in the generated response that can be supported by the context has been properly cited), we need to do the following for each generated response segment that contains citations: (1) determine whether the cited passages support all the information present in the generated response, and (2) determine whether the context supports all the information present in the generated response. Response part 1: Information coverage based on citations:The candidate response closely……..ages. Response part 2: Information …… passages. Response part 3: Information …..e bucket. Response part 4: Information coverage based on citations:The candidate response accurately reflects the information provided in the related passages…… in the available context passages. Information coverage based on context: The candidate response accurately reflects the information provided in the context passages regarding the. It mentions the system's purpose, the use of...... All these points are covered in the passages.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content