Elastic Beanstalk creation AWSEBInstanceLaunchWaitCondition issue

Hi,

I have been trying to create a new Beanstalk environment in a private VPC that does have a NAT gateway configured, but it is failing with AWSEBInstanceLaunchWaitCondition. On logging into the instance which is created I can access the internet and see in the /var/log/eb-commandprocessor.log it seems to be failing on the bootstrapping of the environment.

It is a Node.js 10.16.0 app running on 64bit Amazon Linux/4.9.2

It seems like it cannot access an S3 endpoint to download a manifest. The environment is configured with the following for the instances:
IAM instance profile: aws-elasticbeanstalk-ec2-role
Service role: aws-elasticbeanstalk-service-role

Is this an AWS issue, or possibly some configuration on my side?

I've tried to recreate this multiple times. I seem to have the same issue whether it is in a private VPC or with the defaults set.

Any assistance would be appreciated.
Philip

[2019-07-31T20:16:37.228Z] DEBUG [2458]  : Reading config file: /etc/elasticbeanstalk/.aws-eb-stack.properties
[2019-07-31T20:16:37.229Z] DEBUG [2458]  : Checking if the command processor should execute...
[2019-07-31T20:16:37.231Z] DEBUG [2458]  : Checking whether the command is applicable to instance (i-0571565fa8axxxxxx)..
[2019-07-31T20:16:37.231Z] INFO  [2458]  : Command is applicable to this instance (i-0571565fa8axxxxxx)..
[2019-07-31T20:16:37.231Z] DEBUG [2458]  : Checking if the received command stage is valid..
[2019-07-31T20:16:37.231Z] INFO  [2458]  : No stage_num in command. Valid stage..
[2019-07-31T20:16:37.231Z] INFO  [2458]  : Received command CMD-PreInit: {"api_version"=>"1.0", "request_id"=>"0", "command_name"=>"CMD-PreInit"}
[2019-07-31T20:16:37.231Z] INFO  [2458]  : Command processor should execute command.
[2019-07-31T20:16:37.231Z] DEBUG [2458]  : Storing current stage..
[2019-07-31T20:16:37.231Z] DEBUG [2458]  : Stage_num does not exist. Not saving null stage. Returning..
[2019-07-31T20:16:37.231Z] DEBUG [2458]  : Reading config file: /etc/elasticbeanstalk/.aws-eb-stack.properties
[2019-07-31T20:16:37.231Z] DEBUG [2458]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_ContainerConfigFileContent||commands..
[2019-07-31T20:16:37.232Z] DEBUG [2458]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_API||_Commands..
[2019-07-31T20:16:37.232Z] INFO  [2458]  : Found enabled addons: ["logstreaming", "logpublish"].
[2019-07-31T20:16:37.233Z] INFO  [2458]  : Updating Command definition of addon logstreaming.
[2019-07-31T20:16:37.233Z] INFO  [2458]  : Updating Command definition of addon logpublish.
[2019-07-31T20:16:37.233Z] DEBUG [2458]  : Refreshing metadata...
[2019-07-31T20:16:37.234Z] DEBUG [2458]  : Using cached environment metadata.
[2019-07-31T20:16:37.234Z] INFO  [2458]  : Recreated directory /opt/elasticbeanstalk/deploy/configuration/.
[2019-07-31T20:16:37.234Z] DEBUG [2458]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_AppSourceUrlFileContent||url..
[2019-07-31T20:16:37.235Z] INFO  [2458]  : Created appsource url file at /opt/elasticbeanstalk/deploy/configuration/appsourceurl.
[2019-07-31T20:16:37.235Z] DEBUG [2458]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_ContainerConfigFileContent..
[2019-07-31T20:16:37.236Z] INFO  [2458]  : Created container config file at /opt/elasticbeanstalk/deploy/configuration/containerconfiguration.
[2019-07-31T20:16:37.236Z] DEBUG [2458]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_ContainerConfigFileContent||commands..
[2019-07-31T20:16:37.237Z] DEBUG [2458]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_API||_Commands..
[2019-07-31T20:16:37.237Z] INFO  [2458]  : Found enabled addons: ["logstreaming", "logpublish"].
[2019-07-31T20:16:37.238Z] INFO  [2458]  : Updating Command definition of addon logstreaming.
[2019-07-31T20:16:37.238Z] INFO  [2458]  : Updating Command definition of addon logpublish.
[2019-07-31T20:16:37.238Z] DEBUG [2458]  : Retrieving metadata for key: AWS::CloudFormation::Init||Infra-WriteApplication2||files..
[2019-07-31T20:16:37.239Z] DEBUG [2458]  : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||ManifestFileS3Key..
[2019-07-31T20:16:37.353Z] INFO  [2458]  : Finding latest manifest from bucket 'elasticbeanstalk-eu-west-1-913248xxxxxx' with prefix 'resources/environments/e-xxxxxx/_runtime/versions/manifest_'.
[2019-07-31T20:16:37.500Z] INFO  [2458]  : Found manifest with key 'resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx'.
[2019-07-31T20:16:37.548Z] WARN  [2458]  : Failed to fetch manifest with key resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx from elasticbeanstalk-eu-west-1-913248xxxxxx, reason: Access Denied. Retrying after 0.2198677603215735 seconds.
[2019-07-31T20:16:37.768Z] INFO  [2458]  : Finding latest manifest from bucket 'elasticbeanstalk-eu-west-1-913248xxxxxx' with prefix 'resources/environments/e-mevwvg3g77/_runtime/versions/manifest_'.
[2019-07-31T20:16:37.783Z] INFO  [2458]  : Found manifest with key 'resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx'.
[2019-07-31T20:16:37.824Z] WARN  [2458]  : Failed to fetch manifest with key resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx from elasticbeanstalk-eu-west-1-913248xxxxxx, reason: Access Denied. Retrying after 3.7115773669033487 seconds.
[2019-07-31T20:16:41.537Z] INFO  [2458]  : Finding latest manifest from bucket 'elasticbeanstalk-eu-west-1-913248xxxxxx' with prefix 'resources/environments/e-xxxxxx/_runtime/versions/manifest_'.
[2019-07-31T20:16:41.671Z] INFO  [2458]  : Found manifest with key 'resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx'.
[2019-07-31T20:16:41.747Z] WARN  [2458]  : Failed to fetch manifest with key resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx from elasticbeanstalk-eu-west-1-913248xxxxxx, reason: Access Denied. Retrying after 7.0076288331419 seconds.
[2019-07-31T20:16:48.755Z] INFO  [2458]  : Finding latest manifest from bucket 'elasticbeanstalk-eu-west-1-913248xxxxxx' with prefix 'resources/environments/e-xxxxxxx/_runtime/versions/manifest_'.
[2019-07-31T20:16:48.878Z] INFO  [2458]  : Found manifest with key 'resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx'.
[2019-07-31T20:16:48.922Z] WARN  [2458]  : Failed to fetch manifest with key resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx from elasticbeanstalk-eu-west-1-913248xxxxxx, reason: Access Denied. Retrying after 2.605142037623594 seconds.
[2019-07-31T20:16:51.528Z] INFO  [2458]  : Finding latest manifest from bucket 'elasticbeanstalk-eu-west-1-913248xxxxxx' with prefix 'resources/environments/e-xxxxxx/_runtime/versions/manifest_'.
[2019-07-31T20:16:51.653Z] INFO  [2458]  : Found manifest with key 'resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx'.
[2019-07-31T20:16:51.699Z] WARN  [2458]  : Failed to fetch manifest with key resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx from elasticbeanstalk-eu-west-1-913248xxxxxx, reason: Access Denied.
[2019-07-31T20:16:51.699Z] ERROR [2458]  : Failed to fetch manifest with key resources/environments/e-xxxxxx/_runtime/versions/manifest_1564597xxxxxx from elasticbeanstalk-eu-west-1-913248xxxxxx, reason: Access Denied. (ElasticBeanstalk::ManifestDownloadError)
	at /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/manifest.rb:147:in `rescue in fetch'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/manifest.rb:100:in `fetch'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/manifest.rb:57:in `update_cache'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command.rb:58:in `load'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command-processor.rb:134:in `load_command'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command-processor.rb:66:in `execute!'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/bin/command-processor:45:in `<top (required)>'
	from /opt/elasticbeanstalk/lib/ruby/bin/command-processor:23:in `load'
	from /opt/elasticbeanstalk/lib/ruby/bin/command-processor:23:in `<main>'
caused by: Access Denied (Aws::S3::Errors::AccessDenied)
	at /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.3.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.3.0/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.3.0/lib/aws-sdk-core/plugins/s3_accelerate.rb:33:in `call'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.3.0/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.3.0/lib/seahorse/client/plugins/response_target.rb:21:in `call'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.3.0/lib/seahorse/client/request.rb:70:in `send_request'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-core-2.3.0/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-resources-2.3.0/lib/aws-sdk-resources/request.rb:24:in `call'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-resources-2.3.0/lib/aws-sdk-resources/operations.rb:41:in `call'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sdk-resources-2.3.0/lib/aws-sdk-resources/operation_methods.rb:19:in `block in add_operation'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/manifest.rb:126:in `fetch'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/manifest.rb:57:in `update_cache'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command.rb:58:in `load'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command-processor.rb:134:in `load_command'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command-processor.rb:66:in `execute!'
	from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/bin/command-processor:45:in `<top (required)>'
	from /opt/elasticbeanstalk/lib/ruby/bin/command-processor:23:in `load'
	from /opt/elasticbeanstalk/lib/ruby/bin/command-processor:23:in `<main>'

Edited by: PhilipCorreia on Jul 31, 2019 1:29 PM