Knowledge-driven conversational solutions are essential in modern digital workflows. They allow large language models to interact with curated knowledge bases for a diverse range of tasks including customer support, search and analytics. However, evaluating them poses some challenges — they’re typically composed of multiple interdependent components, continuously evolving sources of data and feature opaque, black box mechanisms.

In this post, we detail how we evaluated an enterprise AI conversational system powered by AWS Bedrock Agents and Amazon Kendra, using Weights & Biases (W&B) Weave.

Before this, our evaluation relied on ad hoc manual testing. Team members submitted queries to the system, manually inspected the generated responses, and subjectively judged quality. This approach was inconsistent across evaluators, couldn't scale as the knowledge base grew, lacked reproducibility and provided no way to isolate whether failures originated from retrieval, prompting, or generation.

By replacing this fragmented process with a structured, metrics-driven evaluation framework, we achieved significant improvements in system performance, accuracy and stakeholder alignment.

The challenges of evaluating AI systems

Although traditional metrics like accuracy or F1 score are applicable to algorithmically-generated text data, they’re insufficient for assessing a complex conversational system. This isn't because we can't calculate precision/recall for text, but because binary classifications miss the multiple dimensions of retrieval systems. Our evaluation identified several critical challenges:

  • Black-box evaluation limitations. Bedrock Agents concealed retrieval mechanisms, which made it difficult to directly calculate retrieval metrics.

  • Inconsistent source attribution. Generated responses were often missing citations linked to knowledge base documents.

  • An evolving knowledge base. As the knowledge base expanded from object storage to enterprise search, the evaluation criteria needed to evolve.

  • Multi-stakeholder alignment. Business users, technical teams and compliance officers all required different evaluation perspectives.

How can we define system accuracy holistically?

In most established forms of machine learning, accuracy is typically a single measure of correct predictions. However, for knowledge-driven conversational solutions, accuracy is multi-dimensional; it needs to be assessed across three interdependent components:

  1. A retrieval component (RAG): Did the system retrieve relevant and correct documents based on the input query?

  2. A prompt engineering component: Did the system effectively guide the LLM to use retrieved context?

  3. Language model component (the LLM): Did the LLM generate a factually correct and coherent answer?

All three need to work in harmony for accuracy. Perfect retrieval with poor prompting, for example, still produces irrelevant answers, while strong prompting with bad retrieval leads to hallucinations.

A structured evaluation framework

To address these challenges, we implemented Weights & Biases Weave as a unified evaluation platform. The system under evaluation is a customer-facing conversational AI assistant: users submit natural language queries, Amazon Kendra retrieves relevant documents from a knowledge base and AWS Bedrock Agents generates a response using those documents as context.

Because failures can occur at any stage of this pipeline — wrong documents retrieved, poor prompt construction, or unfaithful generation — a single accuracy score isn’t sufficient. Our framework consolidated assessment across five critical dimensions: retrieval quality, answer faithfulness, answer relevance, context precision and system performance. Each dimension targets a specific stage of the pipeline, enabling us to pinpoint exactly where quality degrades.

The components of the solution included:

  • A unified evaluation project: Using W&B Weave's project structure, we centralized all test cases, metrics and production data in a single workspace — ensuring every team member worked from the same source of truth.

  • A structured test dataset: We curated a set of representative queries, each paired with ground truth answers and expected source documents, to measure system performance consistently across iterations.

  • Multi-dimensional scoring: More than 25 granular metrics were implemented, including precision@k, recall@k, semantic similarity and hallucination detection — each targeting a specific stage of the RAG pipeline.

  • Traceability: Each evaluation run captured full traces of prompts, retrieved documents and generated responses. This made it possible to debug failures at any stage.

  • Visualization: Custom dashboards provided actionable insights for both technical teams (who needed component-level diagnostics) and business stakeholders (who needed system-level quality trends).

Implementing the evaluation process

We developed a systematic analysis and improvement workflow, which enabled consistent progress tracking and targeted enhancements.

Our initial evaluation runs provided a strong starting point across multiple dimensions:

  • The overall RAG score was 0.8626, providing a reliable benchmark for system performance.

  • Answer similarity began at 0.4444, reflecting early alignment with ground truth answers.

  • Topic coverage was 0.6667, showing partial but consistent coverage of expected topics.

  • Response confidence started at 0.85, representing a solid foundation for reliable responses.