JSON formatting issue ( api gateway -> firehose )

The issue you're experiencing is related to JSON formatting when sending data from API Gateway to Firehose. To resolve this problem, you need to ensure that the JSON data is properly formatted before it reaches Firehose. Here are a few approaches you can consider:

1. API Gateway Configuration: Make sure your API Gateway is configured to handle JSON payloads correctly. In the integration request settings, you can set up a mapping template to format the incoming data properly.

2. Use JSON.stringify(): If you're constructing the JSON in your code before sending it to API Gateway, use JSON.stringify() to properly format the object as a JSON string. This will ensure that the quotes and structure are correct.

3. Escape Special Characters: If you're manually constructing the JSON string, make sure to escape any special characters, especially double quotes, within the string values.

4. Firehose Record Format Conversion: Firehose has built-in capabilities to convert record formats. You can enable record format conversion in your Firehose delivery stream settings. This can help standardize the format of incoming data.

5. Data Transformation in Firehose: If the JSON formatting issue persists, you can use Firehose's data transformation features to correct the format before it's delivered to S3. This can be done using Firehose's built-in transformation capabilities without requiring a Lambda function.

6. Verify JSON Structure: Double-check that your JSON structure is valid. Each key-value pair should be enclosed in double quotes, and the entire object should be enclosed in curly braces.

By implementing these steps, you should be able to resolve the JSON formatting issue in your API Gateway to Firehose to S3 pipeline. Remember to test your solution thoroughly to ensure that the data is being correctly formatted and processed throughout the entire flow.

Sources
Firehose: Dynamic partitioning | AWS re:Post
Convert input data format in Amazon Data Firehose - Amazon Data Firehose
HIVE_CURSOR_ERROR: Row is not a valid JSON Object - JSONException: A JSONObject text must end with '}' at 2 [character 3 line 1] | AWS re:Post

Hi, here's how to ensure proper JSON formatting:

Solution

1. In API Gateway, set up a mapping template in the integration request:

#set($inputRoot = $input.path('$'))
{
    "mediaId": "$inputRoot.mediaId",
    "ctrId": $inputRoot.ctrId,
    "matId": $inputRoot.matId,
    "timestamp": $inputRoot.timestamp
}

2. Configure the content-type header:

Content-Type: application/json

3. If you're using direct service integration, modify the Method Request to:

{
    "mediaId": "$input.path('$.mediaId')",
    "ctrId": $input.path('$.ctrId'),
    "matId": $input.path('$.matId'),
    "timestamp": $input.path('$.timestamp')
}

4. For proper string escaping, use the $util.escapeJavaScript() utility:

{
    "Data": "$util.escapeJavaScript($input.body)\n"
}

Testing

Test your API with this sample payload:

{
    "mediaId": "MPLT000015",
    "ctrId": 42,
    "matId": 1317,
    "timestamp": 1731048538
}

The output in Firehose should match this format exactly.

For more information, see:

• API Gateway Mapping Templates
• Kinesis Firehose Data Transformation

Let me know if you need any clarification!