I have a django app which uses celery beat to scan the DB and trigger tasks accordingly. I want to deploy this to elastic beanstalk, but simply applying leader_only to my celery beat invocation won't be enough as we need a way to implement protection such that the beat instance is not killed during autoscaling events.
So far I've found the following options online
Run a separate ec2 that runs celery beat - not ideal but I could make this a cheap instance since the functionality required is so simple and lightweight. I assume that if I point this at an SQS queue and have my workers pulling from that queue everything will work fine. However, it's not clear to me how to have this instance discover the tasks from my Django app short of deploying it again to the second instance and then having that beat instance interact with my queue.
Use some sort of leader selection lambda like described here (https://ajbrown.org/2017/02/10/leader-election-with-aws-auto-scaling-groups.html) for my EB autoscaling group. This seems a bit extra complicated, in order to implement this I'm guessing the idea is to have a script in my container commands that checks if it is the leader instance (as assigned by the leader tag in the above tutorial) and only execute celery beat if this is the case.
Ditch SQS and use an Elasticache Redis instance as my broker, then install the redbeat scheduler (https://github.com/sibson/redbeat) to prevent multiple instances of a beat service from running. I assume this wouldn't affect the tasks it spawns though correct? My beat tasks spawn several tasks of the same 'type' with different arguments (would like an idiot check on this if possible).
My question is, can anyone help me assess the pros and cons of these implementations in terms of cost and functionality? Is there a better, more seamless way to ensure that celery beat simply runs on one instance alone, while my celery workers scale with my autoscaling infrastructure? AWS newbie so would greatly appreciate any help!
