By using AWS re:Post, you agree to the Terms of Use
/Step Function manage/trigger Fargate task container override storage size?/

Step Function manage/trigger Fargate task container override storage size?

0

Hi, We successfully trigger fargate task through step function, Something like below:

  • Is there anyway to define the ephemeral storage size, cpu and memory within the container override section below?
  • Now, when we create fargate task, we set desire_count=0, min_count=0 and we have 0 instance always on (which is good, we don't want to always on instance). But how will max_count be treated? When we have more than max_count of concurrent step function running, will some step just hanged and waiting for the finishment of certain fargate task? Or throwing error?

Thank you , in advance.

"UncompressGranuleECS": { "Parameters": { "LaunchType": "FARGATE", "Cluster": "${module.cluster.ecs_cluster_name}", "TaskDefinition": "${local.fargateTaskARN}", "NetworkConfiguration": { "AwsvpcConfiguration": { "Subnets": ${jsonencode(var.subnet_ids)}, "AssignPublicIp": "DISABLED" } }, "Overrides": { "ContainerOverrides": [ { "Name":"dyen-cumulus-uncompress_granule-fargate",

              "Command": ["java",
                   "-Xms7500M", 
                   "-Xmx7500M",
                   "-jar",
                   "/home/dockeruser/build/libs/my-jarfile.jar"],
              "Environment":[  
                {  
                    "Name":"region",
                    "Value":"us-west-2"
                },
                {  
                    "Name":"TASK_TOKEN",
                    "Value.$":"$$.Task.Token"
                },
                {  
                    "Name":"CMA_MESSAGE",
                    "Value":"myjson:[]"
                }
              ]
            }
        ]
      }
    },
    "Type": "Task",
    "HeartbeatSeconds": 1800,
    "Resource": "arn:aws:states:::ecs:runTask.waitForTaskToken",
    "Retry": [
      {
        "ErrorEquals": [
          "States.ALL"
        ],
        "IntervalSeconds": 10,
        "MaxAttempts": 0
      }
    ],
    "Catch": [
      {
        "ErrorEquals": [
          "States.ALL"
        ],
        "ResultPath": "$.exception",
        "Next": "failureStep"
      }
    ],
    "Next": "MissingStep"
  },
2 Answers
1
  1. Based on the documentation all 3 of them should be configurable in the Overrides section of the RunTask API, not in the ContainerOverrides.

  2. If you try to invoke more tasks than what is configured, you should receive an error back in StepFunctions. You should use some retry mechanism, maybe with an additional Wait state to wait for other tasks to complete.

EXPERT
answered 3 months ago
0

Agree with @Uri that you need to implement error handling for any throttling. Wanted to confirm you initially can have up to 1 million concurrent step executions open and you can start up to 10 fargate tasks via RunTask API concurrently (this is the call to start tasks but once running is cleared) so this is an area where backoff and retry is definitely needed

MODERATOR
answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions