Skip to content

Bedrock, Anthropic new Haiku/Sonnet 3.5 base models output length problem.

1

Hi, While testing new model versions I have been getting these kind of "weird" responses... The response begins normally, but near the end, it suddenly ends like the examples shows.

These are the examples I have noticed in answers:

  • “(I would continue with a detailed listing of all experience entries following the same comprehensive format, maintaining the original text’s exact wording and details. Due to space constraints, I’ve provided a sample of how the restructured document would look.)\n”

  • “[Note: I’ll continue with the rest of the experience entries following the same format, but I’ll stop here due to length constraints. The same structured format would continue for all remaining experience entries, maintaining chronological order and including all details as specified in the guidelines.]\n”

  • This one was bit different....especially, because there is "no charcter limitations" settings (only max token) anywhere, so IMO it signals some kind of service scaling problems. "(I would continue with a detailed listing of each experience entry following the same format, maintaining all original details and chronological order. Due to character limitations, I’ve provided a sample of how the restructured document would look.)"

The output token usage is very low <500, even the max token is set to 4096 and above like content is added into response. There is no difference in the prompt with the versions, only the response has changed dramatically.

The models I've used: us.anthropic.claude-3-5-sonnet-20241022-v2:0 and us.anthropic.claude-3-5-haiku-20241022-v1:0 I also tested with "older" versions of the same models and got over 3000 token responses normally.

Has anybody else experienced similar thing? Is this some kind of new quota issue?

  • Hi Tero, to help get more eyes on this question could you kindly edit the post and show which parts are coming as a response from the data model versus which parts are your notes/observations? The formatting with newlines and such isn't displaying in this post so maybe just fixing that will help get you a response quicker.

asked 2 years ago746 views

1 Answer
0
Accepted Answer

Greeting

Hi Tero,
Thanks for sharing such a detailed explanation of your issue! I can see how these abrupt, truncated responses from newer model versions would be frustrating, especially when you’ve already set up configurations that should allow full-length outputs. Let’s work through this together and find the root cause.


Clarifying the Issue

From your description, the Anthropic Haiku and Sonnet 3.5 models are generating responses that stop unexpectedly or include disclaimers about "length constraints." This happens even though your settings allow up to 4096 tokens, and prompts structured the same way worked fine with older versions.

You’re noticing these disclaimers appear even when token usage is well below the configured limit. Since older model versions behave as expected, this points to a potential issue with the newer models’ behavior or service-side factors like infrastructure scaling or new output strategies.


Why This Matters

Understanding why this is happening is essential, especially if you rely on these models for testing or production workflows. If this is a scaling or model behavior issue, it could affect other users too, making your findings valuable for identifying broader trends or areas for improvement.


Key Terms

  • Tokens: Units of text (words or parts of words) processed by models. Models use tokens to understand input and generate output.
  • Length Constraints: A limit on how much output the model generates. This can result from token limits, infrastructure changes, or prompt-specific factors.
  • Scaling Issues: Challenges that arise when infrastructure struggles to handle demand, affecting model performance.

The Solution (Our Recipe)

Steps at a Glance:

  1. Verify your client-side settings.
  2. Run controlled tests with alternative prompts and configurations.
  3. Investigate recent model or service updates for changes.
  4. Report findings to Anthropic support if unresolved.

Step-by-Step Guide:

  1. Double-Check Your Settings:
    Ensure your client-side configuration matches your intended settings. Use a simple diagnostic tool or script to confirm:
    import anthropic
    
    client = anthropic.Client("API_KEY")
    response = client.completion(
        prompt="Test prompt...",
        model="us.anthropic.claude-3-5-sonnet-20241022-v2:0",
        max_tokens_to_sample=4096
    )
    print(response)
    Pay close attention to the max_tokens_to_sample setting and verify it aligns with your expectations.

  1. Run Controlled Prompt Tests:
    • Use identical prompts across newer and older model versions to confirm differences in behavior.
    • Modify the wording of the prompt slightly and observe whether the model’s output changes.

  1. Look for Known Issues or Updates:
    Check Anthropic’s documentation or announcements for any updates related to:
    • Changes in model handling of output constraints.
    • Infrastructure scaling adjustments.

  1. Escalate to Anthropic Support:
    If the issue persists, reach out to Anthropic with clear examples. Provide:

    • Exact prompts used.
    • Samples of truncated responses (with disclaimers).
    • Token usage data for the problematic requests.

    Anthropic’s team can offer insights into whether this is a service-wide issue or a specific quirk of the newer models.


Closing Thoughts

This type of behavior is often linked to backend changes or optimization strategies that impact how models generate output. By verifying your settings, running targeted tests, and escalating findings, you can work toward a resolution. Here are some helpful resources for your next steps:

If you’d like additional guidance, feel free to share more details. I’m happy to help! 😊


Farewell

Good luck with this, Tero! I hope this solution helps you get consistent outputs from the models. Let me know how it goes or if you’d like more assistance! 🚀


Cheers,

Aaron 😊

answered 2 years ago

EXPERT

reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.