Elastic Beanstalk breaking changes today with Launch Config?

Apologies for the difficulties with Elastic Beanstalk. We are aware of Elastic Beanstalk issues affecting some customers. Affected customers are receiving regular updates via Personal Health Dashboard. I'm copying our latest update here, however please check your dashboard for any updates:

We continue to work on the fix for the issue identified in the additional batch deployments between March 31 at 7:00 PM and April 4 at 9:18 PM PDT. We have made significant progress in implementing this fix and are working to finalize the implementation of this over the next few hours. We will provide the next update by April 8 at 2:00 PM PDT.

Note that this issue is only affecting a small percentage of customers, and that rolling with additional batch is otherwise working as expected. If you don't have a Personal Health Dashboard notice, you are likely unaffected by the ongoing known issue, and maybe facing a different issue.

Other than the ongoing issue, please also be aware that using newer instance types and newer platforms will require Launch Template, and Elastic Beanstalk automatically will use Launch Templates when required. For this reason, I recommend to add "ec2:CreateLaunchTemplate" permissions anywhere you use Elastic Beanstalk to avoid permission errors, even after resolution of the ongoing issues.

Please watch your Personal Health Dashboard for further updates. If you require urgent assistance, I recommend opening a support case.

Thanks for the reply, but this doesn't answer my question and no issues appeared in Health Dashboard.

As I wrote, the exact same CF & EB templates worked on Thu Apr 4 and then suddenly stopped working on Fri Apr 5 with "ERROR InvalidParameterValue: Unknown template resource: AWSEBAutoScalingLaunchConfiguration". No changes at all. No new instance types or features are being used (these are old t3). Why would EB suddenly no longer recognize resource "AWSEBAutoScalingLaunchConfiguration" which has been used for a long time and still documented?

As for the other EB stack with "ec2:CreateLaunchTemplate" permissions errors, I did add that permission to resolve that one, as noted. However, that was never needed until Apr 5. This same stack has been in use for a long time without that permission until suddenly on Apr 5 it became a requirement. I mentioned this simply as further evidence that something seems to have changed with EB/CF on Apr 4-5.

Please advise how I can continue using AWSEBAutoScalingLaunchConfiguration in Elastic Beanstalk, as still specified in the documentation. Thanks.

Hello, while I'm awaiting more details I have some additional details on the specific error you are seeing. Apologies for not including the details of why the error you see is related to the ongoing issue. Note I recommend opening a support case so we can take a closer look. A support case will allow you to safely and securely share details to my team. If you share a case ID, I can verify my team takes a look. Until we can review your specific account details here's what's likely occurring, based also on other reports we've seen so far.

Part of the ongoing issues with rolling with additional batch deployments often (but does not always) include symptoms of the Launch Configuration resource "AWSEBAutoScalingLaunchConfiguration" being replaced by a Launch Template resource, "AWSEBEC2LaunchTemplate". When this occurs, this can break references to the AWSEBAutoScalingLaunchConfiguration logical resource ID as defined in the Elastic Beanstalk managed CloudFormation stack. This is why you are seeing a "unknown template resource" error. Normally, we only replace the launch configuration with a launch template when necessary, such as when using modern auto scaling group features. However, during this issue, some customers saw the launch configuration replaced incorrectly, even when not using features that required replacement.

For more specific root cause details for your specific account and environments, I recommend opening a support case. This will also allow my team to review your environments and make sure the issue seen is the same as other reports.

While we have a fix in the works, note a workaround to this issue is to temporarily set rolling updates to "deactivated" (by console UI) or value "false" if using the API. This will prevent the rolling configuration update which prevents the erroneous Launch Configuration replacement. However, to do this, you will still need to contact support (or await the fix), as affected environments often end up in an "invalid" or "not ready" stuck status that requires support to reset the status of the environment back to ready, and only then will you be able to try the workaround.

Please open a support case if you require further assistance at https://aws.amazon.com/contact-us/

Thanks for the response and I appreciate the offer, but I don't have access to support right now. Since I could see that EB was suddenly creating Launch Templates instead of Launch Configurations, I went ahead with reconfiguring yaml to support Launch Templates. I had this mostly working as of yesterday.

But then today it's stopped working again! Because I see it trying to create Launch Configs instead of Templates -- the same yaml which yesterday created Launch Templates. Wow. Can it be the CloudFront/EB teams are actively tinkering with this all in production, making breaking changes while trying to fix the issue?

So I've reverted my changes back to support Launch Configurations and it partly worked, except now it's not honoring DisableIMDSv1=false to make IMDSv2 optional, even though it's in the yaml. It's requiring IMDSv2 for which our code hasn't yet been updated. Another breaking change?

Hello,

All AWS accounts have "Basic" support at a minimum, so even if you don't have a paid technical support plan, I'd be happy to take a closer look at your account if you open any support case and share the ID. I ask for a case ID because I cannot ask for any account nor resource details here over re:Post, and a case will allow me to securely review your account.

Here is the latest update from our teams, as sent to all affected accounts via AWS Health Dashboard at https://health.aws.amazon.com/health/home#/account/dashboard/open-issues, please stay tuned to your health dashboard for any further updates.

[April 10 8:17 AM PDT] We are still working on the fix for the issue identified in the additional batch deployments between March 31 at 7:00 PM and April 4 at 9:18 PM PDT. The root cause has been identified and a fix has been deployed and recovery has been confirmed in the US-WEST-1, AP-NORTHEAST-1 and EU-WEST-1 regions and currently deploying through AP-SOUTHEAST-1, EU-CENTRAL-1, SA-EAST-1 and US-EAST-1. All other impacted regions will be updated by April 12. As a workaround, please change the rolling update type to either “Deactivated” or “Immutable”. We are aware that some customers who were migrated to use launch templates are experiencing IAM related errors. For these cases, we would advise to wait for fix to rollout, alternatively customers can either add missing IAM permissions or clone the environment with the help of customer support. We will provide an update by April 11 9:00 AM PDT, or sooner if we have additional information to share.

I have an update from our Elastic Beanstalk team:

[April 11 1:32 PM PDT] We have fully resolved the issue that affected additional batch deployments between March 31 at 7:00 PM and April 4 at 9:18 PM PDT. You can now proceed with your deployments as normal. The service is now operating normally.

Note affected customers may still need to contact support to have any environments stuck in a "not ready" status be set to "ready", as well as to verify that the underlying CloudFormation stack isn't stuck in "UPDATE_ROLLBACK_FAILURE", which will require a continue update rollback operation as well. We're currently in progress setting affected environments to ready state, although customers may still need to perform their own continue update rollback operation per https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-continueupdaterollback.html

If you have further questions, or require assistance to final recovery per above, please reply to support case 171286870500640 which I had our internal teams open against your account, and I'll be happy to help further. I'm reviewing your account now, and will continue communications in the support case.