A customer in need of configuring an API Gateway fronting a Kinesis Stream. We're currently working through the docs to Create an API Gateway API as an Amazon Kinesis Proxy.
For the PutRecord
and PutRecords
APIs, customer wants to support the following payload structure (note the data-elements are passed in as json):
{
"records": [
{
"data": <json>,
"PartitionKey": <String>
},
{
"data": <json>,
"PartitionKey": <String>
}
]
}
e.g. like the following:
{
"records": [
{
"data": {
"Events": [
{"DSType":"RTDS","DSInstance":"Unit 1","DSPoint":"Tag 1","TimeStamp":"2017-10-10T11:10:00.0000000+02:00","Value":8.0,"Quality":192},
{"DSType":"RTDS","DSInstance":"Unit 2","DSPoint":"Tag 2","TimeStamp":"2017-10-10T11:10:00.0000000+02:00","Value":7.0,"Quality":193}
],
"Project":"EventAcquisitionStream",
"Plant":"Plant1"
},
"partition-key": "some key"
},
{
"data": {
"Events": [
{"DSType":"RTDS","DSInstance":"Unit 3","DSPoint":"Tag 3","TimeStamp":"2017-10-11T11:10:00.0000000+02:00","Value":8.0,"Quality":192},
{"DSType":"RTDS","DSInstance":"Unit 4","DSPoint":"Tag 4","TimeStamp":"2017-10-10T11:10:00.0000000+02:00","Value":7.0,"Quality":193}
],
"Project":"EventAcquisitionStream",
"Plant":"Plant2"
},
"partition-key": "some key"
}
]
}
Per AWS documentation referenced above, we have configured the following model and body template mapping for the PutRecords
API:
Model:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "PutRecords proxy payload data model",
"type": "object",
"properties": {
"records": {
"type": "array",
"items": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"events": {
"type": "array",
"items": {
"type": "object",
"properties": {
"dstype": { "type": "string" },
"dsinstance": { "type": "string" },
"dspoint": { "type": "string" },
"timeStamp": { "type": "string" },
"value": { "type": "number" },
"quality": { "type": "integer" }
}
}
},
"project": { "type": "string" },
"plant": { "type": "string" }
}
},
"partition-key": { "type": "string" }
}
}
}
}
}
Body template mapping:
{
"StreamName": "$input.params('stream-name')",
"Records": [
#foreach($elem in $input.path('$.records'))
{
"Data": "$util.base64Encode($elem.data)",
"PartitionKey": "$elem.partition-key"
}#if($foreach.hasNext),#end
#end
]
}
Problem is, when this is decoded on the Kinesis consumer-side, the contents of $elem.data
has been transformed into invalid json, e.g. replacing :
with =
(note for Events
, Project
and Plant
elements):
Decoded payload: {Events=[{"DSType":"RTDS","DSInstance":"Unit 1","DSPoint":"Tag 1","TimeStamp":"2017-10-10T11:10:00.0000000+02:00","Value":8.0,"Quality":192},{"DSType":"RTDS","DSInstance":"Unit 2","DSPoint":"Tag 2","TimeStamp":"2017-10-10T11:10:00.0000000+02:00","Value":7.0,"Quality":193}], Project=EventAcquisitionStream, Plant=Plant1}
What is going on here, how can I configure my body template mapping to retain the json data-structure within the data elements of the request, while still allowing for base64-encoding?