I have been trying to fetch data using a solar data logger using an api gateway that sends data to my api gateway which is integrated to my lamda function but the problem is that api gateway even after adding binary support for text/csv and application/x-www-form-urlencoded and i have been trying to parse the data into json in lamda function and api request using mapping template for data transformation and i've been successful recently only problem now is the request payload is accepted only via api gateway and is rejected if i used postman or if any other source sends the request the logs show that data transformation failed but i am confused why would this issue occur as its being inserted into mongodb after being sent to lambda function
import json
import pymongo
def lambda_handler(event, context):
data = json.loads(event['body']) # Parse the JSON payload
data_fields = data['data'].split("\n")[1:] # The data fields, starting from the second row
client = pymongo.MongoClient("mongodb+srv://******.mongodb.net/?retryWrites=true&w=majority")
db = client["******"]
collection = db["****"]
headers = data_fields[0].split(",") # The headers
data_dict = {
"values": {header: value for header, value in zip(headers[1:], data_fields[1:])}
}
# Insert the data into the collection
collection.insert_one(data_dict)
client.close()
return {
"statusCode": 200,
"body": json.dumps("Data inserted into MongoDB")
}
here is the mapping template for x-www-form-url encoded
{
#set ($body = $input.body.toString())
#set ($myArray1 = $body.split("&"))
#set ($key = $myArray1[0])
#set ($key = $key.replace("key=", ""))
"key": "$key",
#set ($checksum = $myArray1[1])
#set ($checksum = $checksum.replace("md5sum=", ""))
"checksum": "$checksum",
#set ($data = $myArray1[2])
#set ($data = $data.replace("data=", ""))
#set ($data = $util.escapeJavaScript($data))
"data": "$data"
}
here's the test data
key=9C-95-6E-7B-C9-11&md5sum=b156d0c8ada364f1a7141eac396510d4&data=ts,inv_2_AC_Active_Power,inv_2_AC_Reactive_Power,inv_2_AC_Frequency,inv_2_AC_Voltage_AN,inv_2_AC_Current_A,inv_2_AC_Active_Power_A,inv_2_AC_PF_A,inv_2_AC_Voltage_BN,inv_2_AC_Current_B,inv_2_AC_Active_Power_B,inv_2_AC_PF_B,inv_2_AC_Voltage_CN,inv_2_AC_Current_C,inv_2_AC_Active_Power_C,inv_2_AC_PF_C,inv_2_DC_Voltage_1,inv_2_DC_Current_1,inv_2_DC_Power_1,inv_2_DC_Voltage_2,inv_2_DC_Current_2,inv_2_DC_Power_2,inv_2_DC_Voltage_3,inv_2_DC_Current_3,inv_2_DC_Power_3,inv_2_DC_Voltage_4,inv_2_DC_Current_4,inv_2_DC_Power_4,inv_2_DC_Voltage_5,inv_2_DC_Current_5,inv_2_DC_Power_5,inv_2_DC_Voltage_6,inv_2_DC_Current_6,inv_2_DC_Power_6,inv_2_DC_Voltage_7,inv_2_DC_Current_7,inv_2_DC_Power_7,inv_2_DC_Voltage_8,inv_2_DC_Current_8,inv_2_DC_Power_8,inv_2_DC_Voltage_9,inv_2_DC_Current_9,inv_2_DC_Power_9,inv_2_DC_Voltage_10,inv_2_DC_Current_10,inv_2_DC_Power_10,inv_2_DC_Voltage_11,inv_2_DC_Current_11,inv_2_DC_Power_11,inv_2_DC_Voltage_12,inv_2_DC_Current_12,inv_2_DC_Power_12,inv_2_kWh_Day_Active,inv_2_kWh_Total_Active,inv_2_Status_Code,inv_2_Event_Code,inv_2_percent,inv_1_AC_Active_Power,inv_1_AC_Reactive_Power,inv_1_AC_Frequency,inv_1_kWh_Total_Active,inv_1_kWh_Day_Active,inv_1_Status_Code,inv_1_Event_Code,inv_1_Event_Code_2,inv_1_Event_Code_3,inv_1_Event_Code_4,inv_1_Event_Code_5,inv_1_DC_Voltage_1,inv_1_DC_Current_1,inv_1_DC_Voltage_2,inv_1_DC_Current_2,inv_1_DC_Voltage_3,inv_1_DC_Current_3,inv_1_DC_Power_1,inv_1_DC_Power_2,inv_1_DC_Power_3,inv_1_AC_Voltage_AN,inv_1_AC_Current_A,inv_1_AC_Voltage_BN,inv_1_AC_Current_B,inv_1_AC_Voltage_CN,inv_1_AC_Current_C,inv_1_Temperature_Internal,inv_1_Current_String_1,inv_1_Current_String_2,inv_1_Current_String_3,inv_1_Current_String_4,inv_1_Current_String_5,inv_1_Current_String_6,inv_1_Current_String_7,inv_1_Current_String_8,inv_1_Current_String_9,inv_1_Current_String_10,inv_1_Current_String_11,inv_1_Current_String_12,inv_1_percent,meter_grid_AC_Active_Power,meter_grid_AC_Reactive_Power,meter_grid_AC_Apparent_Power,meter_grid_kWh_Total_Import,meter_grid_AC_Voltage_AN,meter_grid_AC_Voltage_BN,meter_grid_AC_Voltage_CN,meter_grid_AC_Voltage_AB,meter_grid_AC_Voltage_BC,meter_grid_AC_Voltage_CA,meter_grid_AC_Current_A,meter_grid_AC_Current_B,meter_grid_AC_Current_C,meter_grid_AC_Active_Power_A,meter_grid_AC_Active_Power_B,meter_grid_AC_Active_Power_C,meter_grid_AC_Apparent_Power_A,meter_grid_AC_Apparent_Power_B,meter_grid_AC_Apparent_Power_C,meter_grid_AC_Reactive_Power_A,meter_grid_AC_Reactive_Power_B,meter_grid_AC_Reactive_Power_C,meter_grid_AC_PF,meter_grid_AC_Frequency,meter_grid_kVAh_Total_Import,meter_grid_kVARh_Total_Import,meter_grid_kVARh_Total_Export,meter_DG125_AC_Active_Power,meter_DG125_AC_Reactive_Power,meter_DG125_AC_Apparent_Power,meter_DG125_kWh_Total_Import,meter_DG125_AC_Voltage_AN,meter_DG125_AC_Voltage_BN,meter_DG125_AC_Voltage_CN,meter_DG125_AC_Voltage_AB,meter_DG125_AC_Voltage_BC,meter_DG125_AC_Voltage_CA,meter_DG125_AC_Current_A,meter_DG125_AC_Current_B,meter_DG125_AC_Current_C,meter_DG125_AC_Active_Power_A,meter_DG125_AC_Active_Power_B,meter_DG125_AC_Active_Power_C,meter_DG125_AC_Apparent_Power_A,meter_DG125_AC_Apparent_Power_B,meter_DG125_AC_Apparent_Power_C,meter_DG125_AC_Reactive_Power_A,meter_DG125_AC_Reactive_Power_B,meter_DG125_AC_Reactive_Power_C,meter_DG125_AC_PF,meter_DG125_AC_Frequency,meter_DG125_kVAh_Total_Import,meter_DG125_kVARh_Total_Import,meter_DG125_kVARh_Total_Export,meter_DG250_AC_Active_Power,meter_DG250_AC_Reactive_Power,meter_DG250_AC_Apparent_Power,meter_DG250_kWh_Total_Import,meter_DG250_AC_Voltage_AN,meter_DG250_AC_Voltage_BN,meter_DG250_AC_Voltage_CN,meter_DG250_AC_Voltage_AB,meter_DG250_AC_Voltage_BC,meter_DG250_AC_Voltage_CA,meter_DG250_AC_Current_A,meter_DG250_AC_Current_B,meter_DG250_AC_Current_C,meter_DG250_AC_Active_Power_A,meter_DG250_AC_Active_Power_B,meter_DG250_AC_Active_Power_C,meter_DG250_AC_Apparent_Power_A,meter_DG250_AC_Apparent_Power_B,meter_DG250_AC_Apparent_Power_C,meter_DG250_AC_Reactive_Power_A,meter_DG250_AC_Reactive_Power_B,meter_DG250_AC_Reactive_Power_C,meter_DG250_AC_PF,meter_DG250_AC_Frequency,meter_DG250_kVAh_Total_Import,meter_DG250_kVARh_Total_Import,meter_DG250_kVARh_Total_Export,eb_state,kWh_Curtailment_Day,kWh_Curtailment_Lifetime,kWh_Curtailment_DG_Day,kWh_Curtailment_Grid_Day
1691214121,0,0,49.940,232.100,0.120,0,0,233.100,0.100,0,0,235.400,0.120,0,0,490.100,0,0,490.200,0,0,276.200,0,0,490,0,0,490,0,0,490.100,0,0,490.400,0,0,490.100,0,0,489.900,0,0,489.600,0,0,96.600,0,0,96,0,0,0,26852.400,0,0,0,0,0,0,16095,170.800,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,60185.984,-14305.585,61893.344,11868.400,234.840,233.440,232.690,405.390,403.900,404.800,87.664,103.856,72.824,20249.582,23392.458,16985.238,20642.376,24265.726,16985.238,-4007.763,-6451.228,-3846.593,-0.972,49.940,12596.900,645.800,2867,0,0,0,350.200,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,357.800,1.900,66.700,0,0,0,883.300,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,918.800,122.800,94.300,0,0,0,0,0
1691214158,0,0,49.920,232.300,0.110,0,0,232.800,0.100,0,0,235,0.130,0,0,493.400,0,0,493.600,0,0,277.200,0,0,493.300,0,0,493.300,0,0,493.400,0,0,493.800,0,0,493.300,0,0,493.300,0,0,493,0,0,97.200,0,0,96.700,0,0,0,26852.400,0,0,0,0,0,0,16095,170.800,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,61599.024,-14434.688,63296.924,11869,234.630,233.250,232.460,405.020,403.540,404.420,89.472,105.704,75.400,20582.072,23784.232,17527.482,20992.814,24655.458,17527.482,-4132.382,-6496.302,-3915.754,-0.973,49.923,12597.600,645.800,2867.200,0,0,0,350.200,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,357.800,1.900,66.700,0,0,0,883.300,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,918.800,122.800,94.300,0,0,0,0,0
here's error from postman
here's the cloud watch log
(c18d0a4c-6df1-488e-a070-838e905ded4b) Extended Request Id: LGAWZF3DhcwFllg=
(c18d0a4c-6df1-488e-a070-838e905ded4b) Verifying Usage Plan for request: c18d0a4c-6df1-488e-a070-838e905ded4b. API Key: API Stage: 6lgxld85fh/Dev
(c18d0a4c-6df1-488e-a070-838e905ded4b) API Key authorized because method 'POST /solar' does not require API Key. Request will not contribute to throttle or quota limits
(c18d0a4c-6df1-488e-a070-838e905ded4b) Usage Plan check succeeded for API Key and API Stage 6lgxld85fh/Dev
(c18d0a4c-6df1-488e-a070-838e905ded4b) Starting execution for request: c18d0a4c-6df1-488e-a070-838e905ded4b
(c18d0a4c-6df1-488e-a070-838e905ded4b) HTTP Method: POST, Resource Path: /solar
(c18d0a4c-6df1-488e-a070-838e905ded4b) Execution failed due to configuration error: Unable to transform request
(c18d0a4c-6df1-488e-a070-838e905ded4b) Method completed with status: 500
(c18d0a4c-6df1-488e-a070-838e905ded4b) X-ray Tracing ID : Root=1-64ff1428-51a3e0e671b26992330675e8
Not sure if you resolved the issue, but the $data does not include a JSON so this may be the reason for it failing.
well issue was with this line #set ($body = $input.body.toString()) removing this results in successful insertion although the value of keys end up empty