ECS and CDK: Trouble placing tasks on the correct nodes

0

I have a CDK process that creates a cluster with 1 t3a.medium and 3 t3a.micro instances running. The process creates those instances and registers them with the cluster. I can also click on the cluster in the AWS console and see instances under the infrastructure tab. So I know that there are instances registered with the cluster. I can also look at instances in the CLI doing:

aws ecs list-container-instances --filter "attribute:ecs.instance-type == t3a.micro" --cluster Cluster0F4AFB82-9XhtVavw8Lz1
{
    "containerInstanceArns": [
        "arn:aws:ecs:us-east-1:xxx:container-instance/Cluster0F4AFB82-9XhtVavw8Lz1/9c016f70e7b74b09a9ece93c7d84b465",
        "arn:aws:ecs:us-east-1:xxx:container-instance/Cluster0F4AFB82-9XhtVavw8Lz1/ffcbd6f70e7b74b09a9ece93c7d846754",
        "arn:aws:ecs:us-east-1:xxx:container-instance/Cluster0F4AFB82-9XhtVavw8Lz1/e4ce4a7378bf42a0bcac8a2c5463e34a"
    ]
}

Or looking at all of the instances:

[cloudshell-user@ip-10-2-36-180 ~]$ aws ecs list-container-instances --cluster Cluster0F4AFB82-9XhtVavw8Lz1
{
    "containerInstanceArns": [
        "arn:aws:ecs:us-east-1:xxx:container-instance/Cluster0F4AFB82-9XhtVavw8Lz1/60dff172513d4e609a2ad0f9f8ba6abd",
        "arn:aws:ecs:us-east-1:xxx:container-instance/Cluster0F4AFB82-9XhtVavw8Lz1/ffcbd6f70e7b74b09a9ece93c7d846754",
        "arn:aws:ecs:us-east-1:xxx:container-instance/Cluster0F4AFB82-9XhtVavw8Lz1/9c016f70e7b74b09a9ece93c7d84b465",
        "arn:aws:ecs:us-east-1:xxx:container-instance/Cluster0F4AFB82-9XhtVavw8Lz1/e4ce4a7378bf42a0bcac8a2c5463e34a"
    ]
}

However, when I run my tasks I get Reason: No Container Instances were found in your cluster. for the all tasks. I do have placement constraints on the tasks I'm attempting to run on the t3a.micros. But you can see above from the CLI --filter AWS returns the instances that I expect.

If I run ecs-cli check-attributes I get:

ecs-cli check-attributes --task-def solrNode --container-instances abe6a20b4a5543a784a4ae36cfddcb1a  --cluster Cluster0F4AFB82-OaQPQblRfm8D
Container Instance                                                    Missing Attributes
Cluster0F4AFB82-OaQPQblRfm8D  None

Here is my cluster code:

        Cluster solrCluster = Cluster.Builder.create(this, appId("Solr:Cluster"))
                .containerInsights(true)
                .vpc(vpc)
                .capacity(AddCapacityOptions.builder()
                        .autoScalingGroupName("solr")
                        .vpcSubnets(getSubnetSelection(NetworkStack.STORAGE_SUBNET))
                        .minCapacity(1)
                        .maxCapacity(4)
                        .instanceType(settings.getInstanceType("solr.image.type"))  // settings returns t3a.medium
                        .machineImage(EcsOptimizedImage.amazonLinux2())
                        .updatePolicy(UpdatePolicy.rollingUpdate())
                        .keyName(settings.get("ssh.key.name"))
                        .blockDevices(listOf(BlockDevice.builder()
                                .deviceName("/dev/xvda")
                                .volume(BlockDeviceVolume.ebs(settings.getInt("solr.hdd.gb"), EbsDeviceOptions.builder()
                                        .deleteOnTermination(false)
                                        .volumeType(EbsDeviceVolumeType.GP2)
                                        .encrypted(true)
                                        .build()))
                                .build()
                        ))
                        .build()
                )
                .build();
        solrCluster.addCapacity(appId("Zookeeper"), AddCapacityOptions.builder()
                .autoScalingGroupName("zookeeper")
                .vpcSubnets(getSubnetSelection(NetworkStack.STORAGE_SUBNET))
                .minCapacity(3)
                .maxCapacity(3)
                .instanceType(InstanceType.of(InstanceClass.T3A, InstanceSize.MICRO))
                .machineImage(EcsOptimizedImage.amazonLinux2())
                .updatePolicy(UpdatePolicy.rollingUpdate())
                .keyName(settings.get("ssh.key.name"))
                .build()
        );

My task is:

        ContainerDefinition container = solrTask.addContainer( "Solr:Container", ContainerDefinitionOptions.builder()
                .image(ContainerImage.fromRegistry("docker.io/solr:9.2"))
                .containerName("Solr9")
                .essential(true)
                .cpu(2048)
                .memoryReservationMiB(settings.getInt("solr.memory.reserve")) // returns 2048
                .environment(mapOf(
                        "SOLR_HEAP", settings.get("solr.jvm.max"),
                        "ZK_HOST", "zoo1.solr.cloud:2181,zoo2.solr.cloud:2181,zoo3.solr.cloud:2181"
                ))
                .portMappings(listOf(
                        portMapping(8983,8983)
                ))
                .ulimits(listOf(Ulimit.builder().name(UlimitName.NOFILE).softLimit(65000).hardLimit(65000).build()))
                .logging(LogDriver.awsLogs(AwsLogDriverProps.builder()
                        .logGroup(appLogGroup)
                        .streamPrefix("solr-" + settings.getEnv() + "-")
                        .build()))
        container.addMountPoints(MountPoint.builder()
                .sourceVolume(efsVolumeName)
                .readOnly(false)
                .containerPath("/opt/solr/backup")
                .build());

And for the other tasks

                    ContainerDefinition container = pair.left.addContainer( appId("Zookeeper:Container:id"+pair.right), ContainerDefinitionOptions.builder()
                            .image(ContainerImage.fromRegistry("docker.io/zookeeper:3.8"))
                            .containerName("zookeeper_" + pair.right)
                            .essential(true)
                            .cpu(2048)
                            .memoryReservationMiB(384)
                            .portMappings(listOf(
                                    portMapping(2181, 2181),
                                    portMapping(2888, 2888),
                                    portMapping(3888, 3888),
                                    portMapping(8080, 8080)
                            ))
                            .environment(mapOf(
                                    "ZOO_MY_ID", pair.right.toString(),
                                    "ZOO_SERVERS", "server.1=zoo1.solr.cloud:2888:3888" +
                                            " server.2=zoo2.solr.cloud:2888:3888" +
                                            " server.3=zoo3.solr.cloud:2888:3888"
                            ))
                            .logging(LogDriver.awsLogs(AwsLogDriverProps.builder()
                                    .streamPrefix("zookeeper-" + settings.getEnv())
                                    .logGroup(appLogGroup)
                                    .build()))
                            .build()
                    );
                    Ec2Service service = Ec2Service.Builder.create(this, "Zookeeper:Service:" + pair.right)
                            .taskDefinition(pair.left)
                            .vpcSubnets(getSubnetSelection(NetworkStack.STORAGE_SUBNET))
                            .placementConstraints(listOf(PlacementConstraint.memberOf("attribute:ecs.instance-type == t3a.micro")))
                            .cluster(solrCluster)
                            .securityGroups(listOf(storageAccessGroup))
                            .cloudMapOptions(CloudMapOptions.builder()
                                    .name("zoo" + pair.right)
                                    .cloudMapNamespace(privateDns)
                                    .dnsRecordType(DnsRecordType.A)
                                    .container(container)
                                    .dnsTtl(Duration.seconds(10))
                                    .build())
                            .build();

So I clearly have instances attached to my cluster. And those instance stats match. And my placement constraints show that there are instances that match those as well for those services. So why is it not working?!

1 Answer
0

Thank you for posting the query.

To answer your question, we require details that are non-public information. Please open a support case with AWS using the following link

https://console.aws.amazon.com/support/home#/case/create

AWS
SUPPORT ENGINEER
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions