1 Answer
- Newest
- Most votes
- Most comments
0
For Cohere this post would help: https://repost.aws/questions/QU4x7qfzsnSZmGG4xVyvSx0Q/why-streaming-is-not-supported-for-titan-and-cohere-model-on-bedrock.
In case of Titan I am not sure if response stream is supported. In any case I am pasting how response stream can be parsed as example, I hope this is something like this of help?
payload = {
"prompt": prompt,
"max_tokens_to_sample": 1000,
"top_k": 50,
"temperature": 0.1
}
response_stream = self.bedrock_runtime.invoke_model_with_response_stream(
accept="application/json",
body=bytes(json.dumps(payload), "utf-8"),
contentType="application/json",
modelId=self.model_id,
)
stream = response_stream.get('body')
output = []
if stream:
for event in stream:
chunk = event.get('chunk')
if chunk:
chunk_obj = json.loads(chunk.get('bytes').decode())
text = chunk_obj['completion']
output.append(text)
complete_output = ''.join(output)
Relevant content
- asked 6 months ago
- asked 7 months ago
- asked 5 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 8 months ago
- AWS OFFICIALUpdated 3 months ago
- AWS OFFICIALUpdated 2 years ago
Thanks for the reply reply, however this is using the library. If you call the endpoint directly, the response isn’t as expected