Accurate Method for Calculating Token Consumption for AI Usage Across Various Business Units

While Amazon SageMaker JumpStart provides easy access to pre-trained models, including LLMs, it does not have built-in mechanisms specifically for tracking token consumption in the same way that Amazon Bedrock does. However, there are alternative approaches we can recommend to address the customer's requirements for cost allocation and usage monitoring:

1. Token Consumption Tracking: SageMaker JumpStart itself doesn't have a native token counting feature. However, you can implement custom logging within your application to track the number of tokens used in each request and response. This would involve using the tokenizer associated with the specific LLM to count tokens before sending requests and after receiving responses.

2. Usage-Based Cost Allocation: For granular cost allocation, you can leverage several AWS features and best practices:

a) Use separate AWS accounts for different business units, linked under an AWS Organizations structure. This allows for clear separation of costs at the account level.

b) Implement detailed AWS tagging strategies. Apply tags to all relevant resources (e.g., SageMaker endpoints, EC2 instances) to categorize usage by department, project, or other business dimensions.

c) Utilize AWS Cost Explorer and AWS Cost and Usage Reports to analyze and allocate costs based on the tags and account structure.

d) Set up Amazon CloudWatch custom metrics to log relevant usage data (such as number of requests, processing time, or custom token counts). These metrics can be used for more detailed cost allocation and monitoring.

e) Consider using Amazon SageMaker Model Monitor to track model performance and usage patterns, which can indirectly help with cost allocation.

f) Implement a custom solution using AWS Lambda functions to log detailed usage information to Amazon S3 or Amazon DynamoDB, which can then be analyzed for cost allocation purposes.

While these solutions don't provide the exact same functionality as Amazon Bedrock's token tracking, they can be combined to create a comprehensive system for monitoring usage and allocating costs across business units.

For implementation, it's recommended to work with AWS Professional Services or an AWS Partner to design and implement a solution that meets the specific needs of the enterprise customer in the UAE. This would ensure that the solution is optimized for their particular use case and complies with any local regulations or requirements.

Remember that while SageMaker JumpStart itself is free to use, costs will be incurred for the underlying AWS resources used, such as EC2 instances for model hosting and S3 for storage. Accurate tracking and allocation of these costs will be crucial for the customer's requirements.

Sources
Unified data, analytics and AI

Hi,

You can track LLM input and out token consumption on a per invocation basis: they are returned by the Bedrock in the http headers of the Bedrock response.

So, if you capture those headers and know who made this request, you can have a fully precise accounting the costs.

The headers are processed like this (Python)

RESPONSE_METADATA = "ResponseMetadata"
HTTP_HEADERS = "HTTPHeaders"

LATENCY = "x-amzn-bedrock-invocation-latency"
OUTPUT_TOKENS = "x-amzn-bedrock-output-token-count"
INPUT_TOKENS = "x-amzn-bedrock-input-token-count"

response = boto3.client("bedrock-runtime").invoke_model(body=body,
                                                                    modelId=self.model.value,
                                                                    accept="application/json",
                                                                    contentType="application/json")

headers = response[RESPONSE_METADATA][HTTP_HEADERS]

# write to CW as metrics

        if INPUT_TOKENS in headers:
            metric = CloudwatchMetric(namespace=namespace, name=INPUT_TOKENS.removeprefix("x-amzn-"))
            metric.put(cw_value=int(headers[INPUT_TOKENS]), cw_dimensions=[{"model": model.value}])
        if OUTPUT_TOKENS in headers:
            metric = CloudwatchMetric(namespace=namespace, name=OUTPUT_TOKENS.removeprefix("x-amzn-"))
            metric.put(cw_value=int(headers[OUTPUT_TOKENS]), cw_dimensions=[{"model": model.value}])
        if LATENCY in headers:
            metric = CloudwatchMetric(namespace=namespace, name=LATENCY.removeprefix("x-amzn-"))
            metric.put(cw_value=int(headers[LATENCY]) / 1000, cw_unit=CW_UNIT_SECONDS,
                       cw_dimensions=[{"model": model.value}])

Best,

Didier