Can we use a conda environment on AWS Glue-ETL?

I have a python file that runs on a conda environment (i.e, qiime2 which is a conda env installed from "https://data.qiime2.org/distro/core/qiime2-2021.4-py38-linux-conda.yml")

Is there a way to setup a conda environment on AWS Glue-ETL before running a job using my python file? Is seems it is possible to provide pip-installable python modules or whl files in the "Python library path" field, but I cannot find how to set up the running-environment using yml file.

주제

분석 데이터베이스

태그

AWS Glue 데이터 추출, 변환 및 로드

언어

English

rePost-User-9256543

질문됨 2년 전754회 조회

1개 답변

최신
최다 투표
가장 많은 댓글

이 답변이 도움이 되었나요?커뮤니티가 여러분의 지식을 활용할 수 있도록 정답을 찬성하세요.

Hi,

AWS Glue lets you install additional Python modules and libraries for use with AWS Glue ETL. If you want to install Python modules that aren’t included out of the box, please follow the steps lined out in the documentation here:

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html

Thanks.

Nick Furr

답변함 2년 전

rePost-User-9256543
2년 전
Hi, thanks for your answer. I am aware of that python modules can be installed by assigning "Python library path" as described in the document you mentioned. However, conda environment also includes "non-python" library dependencies. I wonder if a conda environment including "non-python" library (e.g., https://data.qiime2.org/distro/core/qiime2-2021.4-py38-linux-conda.yml) can be set up in AWS Glue.

Can we use a conda environment on AWS Glue-ETL?

관련 콘텐츠