While I was writing this the other answers popped up - but adding detail on how to use SQS and auto-scaling.
- Set up a trigger so that a SQS message is sent when an image is uploaded to S3. https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html
- Configure an auto-scaling group based on queue depth. https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-using-sqs-queue.html
- The auto-scaling group should launch the appropriate AMI (that contains your code) on a GPU-enabled instance.
- Have your code retrieve the SQS message, process the image then delete the message from the SQS queue.
- Auto-scaling will then terminate the instance if there are no more messages in the queue.
At step 5 your code could check to see if other messages have arrived and if so, process those in a loop.
The maximum number of instances in step 2 is up to you - do you need to process in parallel? Then have more than one. Do you need to process the images one at a time? Then have a maximum of 1.
Why not do all the preprocessing and the deep learning model in the Lambda function and skip EC2 entirely?
If this isn't an option for some reason - for example, if the run time is longer than the max execution limit of 15 minutes - then you could consider using Lambda to invoke an SSM Run command. The Lambda function would be invoked when an image is uploaded to the S3 bucket. The Lambda would send a run command to the EC2 instance, which could then pick up the image, perform the preprocessing and deep learning model, and return the resulting image to the other S3 bucket.
I would look at S3 Event Notification - When you upload a new image to S3 you can use this as the trigger to your Lambda function. Event notification can send events to:
Amazon Simple Notification Service (Amazon SNS) topics.
Amazon Simple Queue Service (Amazon SQS) queues.
More on S3 Event Notification . . . https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html
Expanding on tedrent's answer, it's pretty easy to write a python/boto3 app to run on your EC2 which will pull S3 Event Notifications off an SQS queue and get the corresponding files from S3.
EC2 https연결을 위한 로드밸런스 타겟 그룹에서 unhealthy 로 나옵니다.질문됨 6일 전
S3 Cross Account 사용 시 ACL 설정 관련 문제수락된 답변질문됨 7일 전
elastic beanstalk 배포는 항상 ec2가 새로 생성되어 교체 되나요?질문됨 4일 전
왜 계속 인스턴스랑 DB가 삭제되지않고 돈이나가나요?질문됨 2일 전
키를 통한 S3 업로드시 access denied 되는 이유를 알고싶습니다.수락된 답변질문됨 21일 전
SSL인증서의 도메인ip 변경하는방법질문됨 13일 전
S3 - Cloudfront 사용시 GET 파라미터로 넘겨 다운로드 파일명 변경 방법.질문됨 2년 전
RDS Aurora MySQL 로그 관련질문됨 5년 전
aws cloudsearch suggest processing time질문됨 23일 전
EC2 의 네트워크 성능에 관한 문의질문됨 6년 전