AWS Builder Center: Learn, Build and Connect with builders in the AWS community
AWS Builder Center is the official home for builders on AWS. Share and read what others are working on, follow people who inspire you, explore training and workshops, and find tools to support what you're building.
Running multi-node parallel job in AWS Batch using R
1
Hi,
I am trying to build a multi-node parallel job in AWS Batch running an R script. My R script runs independently few statistical models for several users. Hence, I want to split and distribute this job running on parallel on a cluster of several servers for faster execution.
My question is to better understand the architecture of it. My understanding is that at some point I have to prepare a containerized version of my R-application code using a Docker image pushed to ECR. My question is:
The parallel logic should be placed inside the R code, while using same image? If yes, how does Batch know how to split my job (in how many chunks) ?? Is the for-loop in the Rcode enough?
Or I should define the parallel logic somewhere in the Dockerfile saying that: container1 run the models for user1-5, container2 run the models for user6-10, etc.. ??
Could you please share some ideas on that topic for better understanding? Much appreciated.