AWS Glue writing to S3 but not creating table

0

Hey! I have a setup currently with a crawler that connects to a PostgreSQL database with JDBC, this works and the crawler generates around 20 tables for this database. I now want to create an ETL job that extracts the data from one table to a S3 bucket, and make S3 data queryable in Athena. I now have a ETL Glue flow that looks like this:

glue

This seems to work except no Table is being created in the database. The S3 target location contains the parquet files and the job succeeds. But no table.

The auto generated spark code looks like this:

S3bucket_node3 = glueContext.getSink(
    path="s3://data-lake",
    connection_type="s3",
    updateBehavior="UPDATE_IN_DATABASE",
    partitionKeys=[],
    enableUpdateCatalog=True,
    transformation_ctx="S3bucket_node3",
)
S3bucket_node3.setCatalogInfo(
    catalogDatabase="postgres_glue_database", catalogTableName="tableName"
)
S3bucket_node3.setFormat("glueparquet")
S3bucket_node3.writeFrame(ApplyMapping_node2)
job.commit()

Does anybody have any idea on where to look? There seems to be nothing wrong with the connection/crawler/bucket permissions. Its just not creating a table for the data its written to the bucket. I tried:

  • recreating the bucket / roles
  • giving the table other names
  • adding and removing additional Input arguments

Thanks in advance!

D Joe
질문됨 일 년 전931회 조회
1개 답변
0

So I managed to find the solution:

AWS Glue provides you with multiple log streams and apparently I missed one, there I found this:

com.amazonaws.services.glue.model.AccessDeniedException: Insufficient Lake Formation permission(s)

So I added AWSLakeFormationDataAdmin to the role and for now it seems to continue

D Joe
답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠