By using AWS re:Post, you agree to the Terms of Use

Master and Slave Instances Architecture

0

Hi Dears,

I have a qustion on how to build master EC2 that can communicate with other EC2s based on need and send tasks to them based on a certian code/ group of tasks. Where Master EC2 will take data from RDS db and send data with task to each EC2 based on scudualing and aknowledgamet from each one.

I appriclate your help dears! Thanks Basem

2 Answers
2
Accepted Answer

I would probably put the tasks on SQS, then have the "Slaves" (worker nodes) pull the messages from SQS.

If tasks contain a lot of data, save it to S3, and have the SQS message include a path (key) to the object so that the worker can find it.

This will be more robust (retry etc.), you can scale your EC2s based on SQS queue length, and you won't have to deal with service discovery (CloudMap/ALB) and firewalls while staying secure.

Good luck!

//Carl

profile picture
answered a month ago
1

Do you want to think of using AWS Batch or serverless compute (StepFunction or a series of Lambda functions) for such use-case?

https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html

https://aws.amazon.com/step-functions/

An example serverless batch processing architecture (may or may not be suitable for your use-case, but just an fyi) - https://aws.amazon.com/blogs/compute/creating-aws-serverless-batch-processing-architectures/

profile picture
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions