Triggering A Scheduled Task On Fargate

Question

Hello All! 
So I'm on my first foray in dockerized applications and would like to deploy a cron task on fargate to run the docker image. 
**Some background:**
The docker image is published to ECR via CI/CD pipeline from my gitlab account. The image is a slim buster python, and I utilize docker in docker to build the image. The push to ECR is successful.

The basics of the image are: 
1. Create a python env with the libraries specified in the requirements.txt
2. Import libraries into __main__.py along with all the user-defined funcs on a func.py file

__main__.py runs a series of DDL sqls and DML sqls passed to our snowflake via the snowflake python SDK which stages data. Then the staged data is looped through in batches with each batch sent to an external api (smarty streets) and the GET results sent back to snowflake (basically standardizing free text address fields is the point of the application).

**My Goal**
Have this run on a chron via a scheduled Fargate task once per day.

**My Problem**
I have a fargate cluster defined, I have a task definition, and a schedule rule (Event Bridge) created, but nothing happens. When I try to manually run the task definition nothing happens. The task shows that its in a pending status then disappears, and there are no logs given either (telling me something is failing) and nothing happens on the scheduled time from the event bridge rule I created. On the SQL DMLs I have a start|finish time for this process being logged to a snowflake table, which is showing that there is nothing being sent to snowflake. I'm sure there is something very stupid that I'm missing on getting a task to run the docker container but am not finding it. Looked through a lot of tutorials on youtube, but they all seem to be web app driven for a Fargate *service *(typically a flask), and not a simple burst ETL job on a schedule like I"m trying to do.

Any examples on how to tie this all together???

**Added Notes**
I'm pretty sure the subnets I chosen have their IPs whitelisted on our snowflake, but I am expecting the API to fail as its external. But at this point I just want to trigger the docker container/

Mudit Jain · Accepted Answer

Hello,

Please note that, you can check the "Triggered Rules", "Invocations" and "Failed Invocations" for the event rule in the EventBridge metrics[1] and you can observe the rule whether it is triggered once in a day as expected from the EventBridge side or not.

Moreover, you can check the api call “RunTask” in the cloudtrail logs, to check whether the task was logged into the logs section at the scheduled time. If, the RunTask api call logs are present, then you would be able to see more details on why the tasks got stopped[2] or whether it started or not that would help you in narrowing down the issue.

Thank you and have a nice day.

REFERENCE:

[1] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html
[2]https://aws.amazon.com/premiumsupport/knowledge-center/ecs-task-stopped/

Triggering A Scheduled Task On Fargate

Relevant content