Unable to run CloudFormation template for 'crawl delta lake tables using AWS crawler' ?

0

Enter image description here

Not able to create crawler using cloud formation template facing this error "Invalid Delta Target with empty delta tables". Though I have data in the tables available and also, I am able to create the crawler using aws console having same data path successfully and is viewed in athena.

AWSTemplateFormatVersion: "2010-09-09"
Description: Delta lake tables using aws glue crawler

Resources:
  Crawl:
    Type: AWS::Glue::Crawler
    Properties:
      Name: crawler_name
      DatabaseName: database_name
      Role: role_arn
      Targets:
        DeltaTargets:
          - DeltaTables: 
            - S3_path_for_delta_table_folder (having deltalogs as child inside)
          - WriteManifest: True  
  • Are you sure the path is correct and available at the time of creation?, maybe check the crawler logs for more info

asked 5 months ago139 views
1 Answer
1
Accepted Answer

Hello,

I tried deploying your template and faced the same error. Then after checking the syntax for the DeltaTargets Property, I noticed there's an extra '-' (hyphen) in the template, which isn't needed. Below is the corrected template.

Resources:
  Crawl:
    Type: AWS::Glue::Crawler
    Properties:
      Name: crawler_name
      DatabaseName: database_name
      Role: role_arn
      Targets:
        DeltaTargets:
          - DeltaTables: 
              - S3_path_for_delta_table_folder (having deltalogs as child inside)
            WriteManifest: True  

DeltaTables and WriteManifest both are part of one DeltaTarget[1]. Hence we need to add '-' only on DeltaTables and not on WriteManifest. I tried the above template and it worked. Hope this helps.

References: [1]https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-crawler-deltatarget.html

AWS
SUPPORT ENGINEER
answered 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions