Token count in Evaluation Pipeline in Bedrock

While Amazon Bedrock does not provide a direct way to track token counts or costs per question specifically within the evaluation pipeline, there are some related capabilities and workarounds you can consider:

1. Token Usage Tracking: For individual model invocations, Bedrock provides token usage information in the response headers. This includes input tokens, output tokens, and total tokens used. You can capture these headers to track token usage on a per-request basis.

2. Evaluation Metrics: The evaluation pipeline in Bedrock offers various built-in metrics for different types of evaluation jobs. While these don't directly measure token counts, they do provide insights into model performance. For example, metrics like "Builtin.Correctness", "Builtin.Completeness", and "Builtin.Helpfulness" are available for knowledge base evaluation jobs with response generation.

3. Custom Logging: To track token usage or costs per question in the evaluation pipeline, you might need to implement a custom logging solution. This could involve using AWS services like CloudWatch, Lambda, or a database to store and analyze the token usage data you collect from individual model invocations.

4. APIs: While there isn't a specific API for tracking tokens in the evaluation pipeline, you can use the InvokeModel API to make individual requests and extract token usage from the response headers. The EvaluationJob API can be used to manage and retrieve information about evaluation jobs, though it doesn't provide direct token count information.

To get a more precise cost allocation or token tracking system for the evaluation pipeline, you may need to combine these approaches or work with AWS support to develop a custom solution that meets your specific requirements.

Sources
The number of output and input tokens in Bedrock responses | AWS re:Post
Accurate Method for Calculating Token Consumption for AI Usage Across Various Business Units | AWS re:Post
EvaluationDatasetMetricConfig - Amazon Bedrock

You can enable Bedrock Model Invocation Logging (https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html). You can select to save the logs in CloudWatch and/or S3. It'll then log all model invocations requests, where you can go and inspect token usage for each request.