Hi Team,
I am looking forward to a solution that can help in fetching the JSON objects in batches using S3 select query.
For example : I have a JSON file with 50 records present in it.
with S3 select query , I could retrieve first 10 records completely and then next 10 and so on.
I had tried scanRange before , but that works on bytes by providing start and end range , which doesn't ensure full records to be fetched .
as the object size can vary for a single record.
{ { status: "Success", created_at: "19 AUG 2019", employees:[ {"name":"name1", "id":"1"}, {"name":"name2", "id":"2"}, {"name":"name3", "id":"3"} ], contacts: [ {"a": "123"}, {"b": "456"} ] }, { status: "Success", created_at: "19 AUG 2019", employees:[ {"name":"name1", "id":"1"}, {"name":"name2", "id":"2"}, {"name":"name3", "id":"3"} ] }, { status: "In_progress", created_at: "19 AUG 2019", employees:[ {"name":"name1", "id":"1"}, {"name":"name2", "id":"2"}, {"name":"name3", "id":"3"} ] }, { status: "Failed", created_at: "19 AUG 2019", employees:[ {"name":"name1", "id":"1"}, {"name":"name2", "id":"2"}, {"name":"name3", "id":"3"} ] } }
I could get any 2 complete record in one batch & next 2 records next.
Batch 1 :
{
status: "In_progress",
created_at: "19 AUG 2019",
employees:[
{"name":"name1", "id":"1"},
{"name":"name2", "id":"2"},
{"name":"name3", "id":"3"}
]
},
{
status: "Failed",
created_at: "19 AUG 2019",
employees:[
{"name":"name1", "id":"1"},
{"name":"name2", "id":"2"},
{"name":"name3", "id":"3"}
]
}
batch 2 :
{
status: "Success",
created_at: "19 AUG 2019",
employees:[
{"name":"name1", "id":"1"},
{"name":"name2", "id":"2"},
{"name":"name3", "id":"3"}
],
contacts: [
{"a": "123"},
{"b": "456"}
]
},
{
status: "Success",
created_at: "19 AUG 2019",
employees:[
{"name":"name1", "id":"1"},
{"name":"name2", "id":"2"},
{"name":"name3", "id":"3"}
]
}