BEDROCK LLAMA 3.3 70B keeps returning response in markdown although it is specified to be in JSON

The issue you're experiencing with Llama 3 70B on AWS Bedrock, where responses are inconsistently formatted between JSON and Markdown, is a known problem that others have encountered as well. This inconsistency can indeed cause issues with code execution when expecting a specific format.

To address this problem, consider the following approaches:

1. Double-check your prompt format: Ensure that you're using the correct prompt structure for Llama 3 models. The prompt should start with "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n", followed by your actual prompt content, and end with "<|start_header_id|>assistant<|end_header_id|>\n\n".

2. Be explicit in your instructions: In your prompt, be very specific about the desired output format. For example, you could include instructions like "Respond only in valid JSON format. Do not use Markdown or any other formatting."

3. Use a wrapper or post-processing: Implement a wrapper function or post-processing step that attempts to parse the response as JSON. If parsing fails, you could either retry the request or have a fallback mechanism to handle non-JSON responses.

4. Adjust model parameters: Experiment with different temperature and top_p settings. Lower values might lead to more consistent outputs, though at the cost of reduced creativity.

5. Consider using a different model: If the inconsistency persists and is causing significant issues in your application, you might want to explore other models available in Bedrock that may provide more consistent JSON outputs.

6. Implement retry logic: When you receive a non-JSON response, you could implement logic to retry the request, potentially with a slightly modified prompt emphasizing the need for JSON output.

If these approaches don't resolve the issue, it may be worth reaching out to AWS support for further assistance, as they might have additional insights or be able to investigate any potential issues with the model's implementation in Bedrock.
Sources
Bedrock Llama 3 empty string response problem | AWS re:Post
Llama 3 providing empty response and string response instead of JSON | AWS re:Post