So I'm on my first foray in dockerized applications and would like to deploy a cron task on fargate to run the docker image.
The docker image is published to ECR via CI/CD pipeline from my gitlab account. The image is a slim buster python, and I utilize docker in docker to build the image. The push to ECR is successful.
The basics of the image are:
- Create a python env with the libraries specified in the requirements.txt
- Import libraries into main.py along with all the user-defined funcs on a func.py file
main.py runs a series of DDL sqls and DML sqls passed to our snowflake via the snowflake python SDK which stages data. Then the staged data is looped through in batches with each batch sent to an external api (smarty streets) and the GET results sent back to snowflake (basically standardizing free text address fields is the point of the application).
Have this run on a chron via a scheduled Fargate task once per day.
I have a fargate cluster defined, I have a task definition, and a schedule rule (Event Bridge) created, but nothing happens. When I try to manually run the task definition nothing happens. The task shows that its in a pending status then disappears, and there are no logs given either (telling me something is failing) and nothing happens on the scheduled time from the event bridge rule I created. On the SQL DMLs I have a start|finish time for this process being logged to a snowflake table, which is showing that there is nothing being sent to snowflake. I'm sure there is something very stupid that I'm missing on getting a task to run the docker container but am not finding it. Looked through a lot of tutorials on youtube, but they all seem to be web app driven for a Fargate *service *(typically a flask), and not a simple burst ETL job on a schedule like I"m trying to do.
Any examples on how to tie this all together???
I'm pretty sure the subnets I chosen have their IPs whitelisted on our snowflake, but I am expecting the API to fail as its external. But at this point I just want to trigger the docker container/
Thank you for sharing all this information. Could you please check the task logs? Also these two below links can help you to troubleshoot what is going on:
It is likely that the task is failing to start due to a missing configuration. Checking it would help to go deep on this and figure out what is happening.