Solutions Architecture Diagram

0

If we have a batch or some program to call 300k records one by one and to send it to REST endpoint one at a time to pull the json response in 500 lines so in that flow what could be the best architecture considering the compute, performance and time to complete and preventing data loss ? And this happens once a month, what are the points to consider and what solution is best considering the OCP and S3 resources are availble and we could add pulsar or Kafka for the second flow.

When receiving the response from rest endpoint we need to be make sure the response is received and synchronously handshake from previous step and the rest response should be pushed to s3 and if any failures that call has to be retried with some mechanism to store and record the processing status.

I need the architecture diagram for the batch program to call for 50k records one by one -> rest endpoint-> receive the rest response in json with 500 lines -> store them in s3

Consider no messages are lost and even when restendpoint is down it should record the status and retry and get the response and pushed to s3 so all this calls are synchronous

What should be the technology stack used in this workflow.

Thanks

No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions