Troubleshooting EC2 Schedule Issue

Question

Hi Aaron Rose

Thanks for your feedback on the thread titled: "(SO0030) instance-scheduler-on-aws v3.0.7 failed"

In this regards, please find below few comments:

1. Verify the time zone in the Instance Scheduler configuration:
scheduler-cli describe-configuration  
scheduler-cli update-configuration --timezone "Your_Timezone"

It seems that describe-configuration is not a valid subcommand. I have therefore used

`scheduler-cli describe-schedules --stack ec2FlaskInstanceScheduler`
 
**Output**

{
   "Schedules": [
      {
         "Type": "schedule",
         "Name": "running",
         "Periods": [],
         "Description": "Instances running",
         "OverrideStatus": "running"
      },
      {
         "Type": "schedule",
         "Name": "scale-up-down",
         "Periods": [
            "working-days...",
            "weekends..."
         ],
         "Timezone": "UTC",
         "Description": "Vertical scaling on weekdays, based on UTC time"
      },
      {
         "Type": "schedule",
         "Name": "seattle-office-hours",
         "Periods": [
            "office-hours"
         ],
         "Timezone": "Pacific",
         "Description": "Office hours in Seattle (Pacific)"
      },
      {
         "Type": "schedule",
         "Name": "stopped",
         "Periods": [],
         "Description": "Instances stopped",
         "OverrideStatus": "stopped"
      },
      {
         "Type": "schedule",
         "Name": "..-office-hours",
         "Periods": [
            "office-hours"
         ],
         "Timezone": ",
         "Description": "Office hours in ..."
      }
   ]
}

2. Ensure the EC2 instance has the correct Schedule tag (mon-fri-7-9):
aws ec2 describe-tags --filters "Name=resource-id,Values=INSTANCE_ID"

Please find below the keys retrieved:

"Key": "aws:autoscaling:groupName",
            "ResourceId": "i-03f564b2f1721f297",
            "ResourceType": "instance",
            "Value": "awseb-e-anyy7wbtnj-stack-AWSEBAutoScalingGroup-b1hEocHGvM2N"
        },
        {
            "Key": "aws:cloudformation:logical-id",
            "ResourceId": "i-03f564b2f1721f297",
            "ResourceType": "instance",
            "Value": "AWSEBAutoScalingGroup"
        },
        {
            "Key": "aws:cloudformation:stack-id",
            "ResourceId": "i-03f564b2f1721f297",
            "ResourceType": "instance",
            "Value": "arn:aws:cloudformation:eu-central-1:014430281164:stack/awseb-e-anyy7wbtnj-stack/b09ae9e0-3c40-11ee-8644-02bb263e2194"
        },
        {
            "Key": "aws:cloudformation:stack-name",
           ....
        },
        {
            "Key": "aws:ec2launchtemplate:id",
            ...
        },
        {
            "Key": "aws:ec2launchtemplate:version",
            ...
        },
        {
            "Key": "elasticbeanstalk:environment-id",
            ...
        },
        {
            "Key": "elasticbeanstalk:environment-name",
            ...
}

I could not find any key related to the stackname ec2FlaskInstanceScheduler.
Could it be due to the fact that the instanceid lies in  --region eu-central-1b ?

In any case while trying to re-create the scheduler with 
`scheduler-cli create-period --stack ec2FlaskInstanceScheduler --region eu-central-1b --name mon-fri-7-9 --begintime 7:00 --endtime 20:59`

the resulting error: Could not connect to the endpoint URL: "https://cloudformation.eu-central-1b.amazonaws.com/"

As suggested:
`aws ec2 create-tags --resources INSTANCE_ID --tags Key=Schedule,Value=mon-fri-7-9`
has not improved the situation yet.

3.Check CloudWatch Logs for the scheduler Lambda to confirm stop actions.
Nothing related to stop actions have been displayed to the logs

Force a scheduler sync:
`scheduler-cli sync-schedules`
As evoked previously `sync-schedules` is not a valid cli sub-command. Hence the resulting error message:
usage: scheduler-cli [-h] [--version]
                     {create-period,create-schedule,delete-period,delete-schedule,describe-periods,describe-schedule-usage,describe-schedules,update-period,update-schedule} 
  
As suggested, running
`aws ec2 stop-instances --instance-ids INSTANCE_ID`
results in :

{
    "StoppingInstances": [
        {
            "CurrentState": {
                "Code": 64,
                "Name": "stopping"
            },
            "InstanceId": "i-03f564b2f1721f297",
            "PreviousState": {
                "Code": 16,
                "Name": "running"
            }
        }
    ]
}

In other terms it cannot be stopped for some reasons: could a team member investigate further i-03f564b2f1721f297 as there might surely be something wrong with it, please? I cannot even stop that instance manually as it would re-run by itself few seconds later.

Looking forward to hearing from you.

Thanks and best regards
ITF

re:Post Agent · Answer

1. Timezone Configuration:
The output from `scheduler-cli describe-schedules` shows that you have multiple schedules with different timezones. The "scale-up-down" schedule is set to UTC, while "seattle-office-hours" is set to Pacific. Ensure that the schedule you're using for your instance has the correct timezone.

2. Instance Tags:
The tags you've listed for your instance (i-03f564b2f1721f297) don't include a "Schedule" tag. This is crucial for the Instance Scheduler to work. You need to add a tag with the key "Schedule" and the value of your schedule name (e.g., "mon-fri-7-9").

3. Region Issues:
The error you encountered when trying to create a period suggests you're using an incorrect region format. AWS regions don't include availability zones (like eu-central-1b). Use just "eu-central-1" as the region.

4. Instance Stopping Issue:
The fact that your instance can't be stopped manually and restarts on its own suggests it might be part of an Auto Scaling group or managed by Elastic Beanstalk. This could interfere with the Instance Scheduler's ability to stop the instance.

5. CloudWatch Logs:
If you're not seeing any stop actions in the CloudWatch Logs for the scheduler Lambda, it might indicate that the scheduler isn't attempting to stop the instance at all, possibly due to the missing Schedule tag.

Recommendations:
1. Add the correct Schedule tag to your instance:
   `aws ec2 create-tags --resources i-03f564b2f1721f297 --tags Key=Schedule,Value=mon-fri-7-9 --region eu-central-1`

2. Verify your schedule configuration:
   `scheduler-cli describe-schedules --stack ec2FlaskInstanceScheduler --region eu-central-1`

3. Check if your instance is part of an Auto Scaling group or Elastic Beanstalk environment. If so, you may need to adjust your approach to scheduling.

4. Review the Instance Scheduler's IAM roles and policies to ensure it has the necessary permissions to stop your instance.

5. Check the CloudWatch Logs for the Instance Scheduler Lambda function to see if there are any error messages or indications of why it's not stopping the instance.

If these steps don't resolve the issue, you may need to involve AWS support to investigate further, especially regarding the instance that can't be stopped manually.
      
**Sources**  
[Scheduled Start and Stop of EC2 Instance: Start Works, but Stop Does Not | AWS re:Post](https://repost.aws/questions/QUI3Pfs-2CRRyFzyp15vyk5w/scheduled-start-and-stop-of-ec2-instance-start-works-but-stop-does-not)  
[Troubleshooting - Instance Scheduler on AWS](https://docs.aws.amazon.com/solutions/latest/instance-scheduler-on-aws/troubleshooting.html)  
[Sample schedules - Instance Scheduler on AWS](https://docs.aws.amazon.com/solutions/latest/instance-scheduler-on-aws/sample-schedules.html)

Troubleshooting EC2 Schedule Issue

Add your answer

Relevant content