Solutions Architecture Diagram

0

If we have a batch or some program to call 300k records one by one and to send it to REST endpoint one at a time to pull the json response in 500 lines so in that flow what could be the best architecture considering the compute, performance and time to complete and preventing data loss ? And this happens once a month, what are the points to consider and what solution is best considering the OCP and S3 resources are availble and we could add pulsar or Kafka for the second flow.

When receiving the response from rest endpoint we need to be make sure the response is received and synchronously handshake from previous step and the rest response should be pushed to s3 and if any failures that call has to be retried with some mechanism to store and record the processing status.

I need the architecture diagram for the batch program to call for 50k records one by one -> rest endpoint-> receive the rest response in json with 500 lines -> store them in s3

Consider no messages are lost and even when restendpoint is down it should record the status and retry and get the response and pushed to s3 so all this calls are synchronous

What should be the technology stack used in this workflow.

Thanks

답변 없음

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠