Amazon-Titan Express huge execution time for custom parameters

The significant increase in execution time you're experiencing with Amazon Titan Express is likely not directly related to the specific custom values you've set for the inference parameters. However, changing these parameters from their default values can indeed affect the model's behavior and potentially its execution time.

Let's break down the changes you've made:

1. Temperature: You increased it from 0.7 to 0.8. A higher temperature generally increases randomness in the output, which might lead to slightly longer processing times as the model explores more diverse options.

2. Top P: You decreased it from 0.9 to 0.3. A lower top P value restricts the model to consider only the most likely tokens, which could potentially speed up the process rather than slow it down.

3. MaxTokenCount: You decreased it from 8,192 to 8,000. This small reduction should not cause a significant increase in execution time. In fact, it might slightly reduce the time if the model was consistently generating near the maximum token count.

Given these changes, the dramatic increase from 5 seconds to 30 seconds is unexpected and likely not solely due to these parameter adjustments. Such a large difference suggests there might be other factors at play:

1. Network latency or connectivity issues
2. Backend service load at the time of your requests
3. The specific content of your prompts or the complexity of the generated responses
4. Potential issues with the Java SDK implementation or your application's configuration

To troubleshoot this issue:

1. Try reverting to the default values and see if the execution time returns to normal.
2. Test with different sets of custom parameters to see if the increased time persists.
3. Monitor your network connection and ensure there are no connectivity issues.
4. Check if the issue occurs consistently across different times of day or with different input prompts.
5. Consider reaching out to AWS support if the issue persists, as they may be able to provide more specific insights or identify any potential service-related issues.

Remember, while custom parameters can affect model behavior and potentially execution time, such a drastic increase is unusual and warrants further investigation beyond just the parameter settings.
Sources
Amazon Titan Text models - Amazon Bedrock
Knowledge Bases for Amazon Bedrock now lets you configure inference parameters