llama3 adds additional characters in generated response

Question

All,
I am experimenting with llama3 model and I see very strange behavior.

Simplified code is below:

*MAX_TOKENS = 1000
bedrock_client = bedrock.get_bedrock_client()

try:

appendString = "?"  * 8128

customer_input = "I am a …?"

prompt = """

Please list different style options for:
    """ + customer_input +  """

"""

prompt_input = {
   ** # "prompt":  prompt + appendString ,
    "prompt": "[INST]" + prompt + appendString + "[/INST]",**
    "max_gen_len": MAX_TOKENS,
    "temperature":0.5,
    'top_p': 0.5
    }

body = json.dumps(prompt_input)

modelId = "meta.llama3-8b-instruct-v1:0"  # change this to use a different version from the model provider
    accept = "application/json"
    contentType = "application/json"
    claudeResponse = ""
    response = bedrock_client.invoke_model(
        body=body, modelId=modelId, accept=accept, contentType=contentType, trace= "ENABLED"
    )
finally:
    bedrock_client.close()

In many cases after several executions output grow (both length and number of generated tokens). Sometimes additional characters are seemingly from input, sometimes they are not even related.
If word "[INST] is included in the prompt, issue almost never happens.

Does this behavior related to the fact that session is not being closed (documentation is not clear on close() method)? If yes, how to close session properly?

Input is always the same (in accordance with the code and logs).

Please let me know what the issue can be.
Any help would be greatly appreciated

Thank you
YK

re:Post Agent · Answer

The behavior you're experiencing with the Llama 3 model in Amazon Bedrock is unusual and not expected. However, there are a few points to consider:

1. Session handling: The issue is likely not related to session closure. The `bedrock_client.close()` method in your `finally` block should be sufficient to properly close the session after each invocation.

2. Prompt formatting: Your use of "[INST]" and "[/INST]" tags is correct for Llama 3 models. These tags help the model understand the beginning and end of the instruction, which can improve response consistency.

3. Appending characters: The `appendString = "?" * 8128` you're adding to the prompt is unusual and may be causing unexpected behavior. Large amounts of repeated characters could potentially confuse the model or cause it to behave unpredictably.

4. Token limit: Ensure that your total prompt (including the appended string) doesn't exceed the model's maximum token limit. If it does, it could lead to unexpected truncation or behavior.

5. Response parsing: Make sure you're correctly parsing the response from the model. The generated text should be in the 'generation' field of the response body.

6. Model behavior: Large language models can sometimes exhibit unexpected behaviors, especially when given unusual inputs. The repeated characters in your prompt might be triggering such behavior.

To troubleshoot:

1. Remove the `appendString` and test with just your base prompt.
2. Double-check that you're correctly parsing the response and not including any artifacts from previous runs.
3. Try adjusting the `temperature` and `top_p` parameters. Lower values (closer to 0) will make the output more deterministic.
4. Monitor the `prompt_token_count` and `generation_token_count` in the response to ensure you're not exceeding token limits.
5. If the issue persists, you may want to contact AWS support for further assistance, as this could potentially be a model-specific issue.

Remember, while the Llama 3 models are powerful, they can sometimes produce unexpected results, especially with non-standard inputs. Always validate and sanitize both inputs and outputs when working with these models in production environments.
      
**Sources**  
[Meta Llama models - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-meta.html)  
[Meta Llama 3 models are now available in Amazon Bedrock | AWS News Blog](https://aws.amazon.com/blogs/aws/metas-llama-3-models-are-now-available-in-amazon-bedrock/)  
[Community | AWS BedRock - Boto3 Demo - LLAMA2 Model](https://community.aws/content/2dVRLD3pySeHF1H6hXVZ9p36By6/aws-bedrock---boto3-demo---llama2-model)

llama3 adds additional characters in generated response

Add your answer

Relevant content