Skip to content

Running multi-node parallel job in AWS Batch using R

1

Hi, I am trying to build a multi-node parallel job in AWS Batch running an R script. My R script runs independently few statistical models for several users. Hence, I want to split and distribute this job running on parallel on a cluster of several servers for faster execution. My question is to better understand the architecture of it. My understanding is that at some point I have to prepare a containerized version of my R-application code using a Docker image pushed to ECR. My question is:

The parallel logic should be placed inside the R code, while using same image? If yes, how does Batch know how to split my job (in how many chunks) ?? Is the for-loop in the Rcode enough? Or I should define the parallel logic somewhere in the Dockerfile saying that: container1 run the models for user1-5, container2 run the models for user6-10, etc.. ??

Could you please share some ideas on that topic for better understanding? Much appreciated.

No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.