Spot Instance scheduling

For your daily task automation with spot instances, here's a comprehensive solution:

1. Setting up a scheduler: You have several options to schedule your daily task:

• EventBridge Scheduler: Create a recurring schedule using a cron expression to run every morning at your desired time. This gives you flexibility with scheduling options using either cron-based schedules (specific times) or rate-based schedules (regular intervals).

• AWS Systems Manager's Quick Setup: This allows you to specify tag keys and values for instances, set schedule options including time zone, days, and times for starting instances.

• AWS Instance Scheduler: This solution provides more flexibility for complex scheduling needs and can be deployed via CloudFormation.

2. Configuring spot instances:

• Create an AMI with your Python script and all necessary dependencies
• Ensure the instance has the proper IAM role to access S3
• Configure your Python script to run at startup via user data
• Include shutdown commands at the end of your script (e.g., sudo shutdown -h now)
• When using EventBridge Scheduler, you can configure a flexible time window which helps optimize costs with spot instances

3. Handling spot instance termination: If your spot instance is terminated by AWS in the middle of your task:

• Your task will be interrupted and incomplete
• You'll need to implement checkpointing in your Python script to track progress
• Consider saving intermediate results to S3 so the next instance can resume where the previous one left off

4. Automatic continuation: By default, if a spot instance is terminated, a new one won't automatically start and continue the task. You'll need to:

• Implement logic in your script to check for incomplete tasks from previous runs
• Store task state in a persistent storage like S3 or DynamoDB
• Have your script detect and resume incomplete work
• Configure your scheduler to start a new instance the next day regardless of previous day's outcome

Complete solution approach:

1. Create an AMI with your Python script and dependencies
2. Set up your script to:

• Run automatically on instance startup
• Check for and resume incomplete work
• Save progress periodically to S3
• Shut down the instance upon completion

3. Configure EventBridge Scheduler to start your spot instance daily
4. Ensure proper IAM permissions for S3 access and instance management
5. Implement logging to CloudWatch for monitoring task execution
6. Consider setting up SNS notifications for task failures

This serverless approach to scheduling will ensure your task runs daily, processes the S3 objects, and then terminates the instance automatically to minimize costs.
Sources
Scheduled task automation | AWS re:Post
Stop and start EC2 instances automatically on a schedule using Quick Setup - AWS Systems Manager
How to schedule automatic instance start and stop (QuickSetup) | AWS re:Post

I would do this a different way.

• Run a ECS Task instead of EC2.
• Schedule ECS task to run daily on Spot on Fargate
• As its DOCKER, updating the image and pre-requisites should be easier