MWAA environment variables are not accessible in requirements.txt?

0

Hi,

when trying to install private python package as a dependency via requirements.txt, it looks like usual mwaa environment variables are not accessible in the requirements.txt.

My requirements.txt file (does not work):
--extra-index-url=https://${AIRFLOW__GEMFURY__TOKEN}:@pypi.fury.io/myusername/
myprivatepackage==0.2

However, if I don't use environment variables inside requirements.txt file, everything works fine, dependencies are installed:
--extra-index-url=https://privatetoken:@pypi.fury.io/myusername/
myprivatepackage==0.2

One more thing to mention, if I try to print out this environment variable inside the DAG – it is accessible and has the defined value, without any problems.

Example how to use mwaa environment variables inside requirements.txt file is here –https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-different-ways
(Option three: Python dependencies hosted on a private PyPi/PEP-503 Compliant Repo)

Does anyone have/had similar issue?

gefragt vor 3 Jahren938 Aufrufe
1 Antwort
0

From version Airflow 2.0 this problem no longer exist!

beantwortet vor 3 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen