Are there any ways to create a custom DeltaTargetProperties with cdk since it doesn't exist?
I see the big is that the DeltaTargetProperties is not support in CFn yet.
Attempt 1 with docs as code examples
cfn_crawler = glue.CfnCrawler(self, "MyCfnCrawler",
role=glue_job_crawler_arn,
targets=glue.CfnCrawler.TargetsProperty(
delta_targets=glue.CfnCrawler.TargetsProperty(
delta_table=[
"s3://bucket/dataset_name/table_name/",
],
write_manifest = True
)
),
...
)
I can run aws glue get-crawler --name cralwer_name
and get:
"Crawler": {
"Name": "cralwer_name",
"Role": "cralwer_role",
"Targets": {
"S3Targets": [],
"JdbcTargets": [],
"MongoDBTargets": [],
"DynamoDBTargets": [],
"CatalogTargets": [],
"DeltaTargets": [
{
"DeltaTables": [
"s3://bucket/dataset_name/table_name/",
],
"WriteManifest": true
}
]
},
"DatabaseName": "db_name",
"Classifiers": [],
"RecrawlPolicy": {
"RecrawlBehavior": "CRAWL_EVERYTHING"
},
"SchemaChangePolicy": {
"UpdateBehavior": "UPDATE_IN_DATABASE",
"DeleteBehavior": "LOG"
},
"LineageConfiguration": {
"CrawlerLineageSettings": "DISABLE"
},
"State": "READY",
...
"Version": 2,
"LakeFormationConfiguration": {
"UseLakeFormationCredentials": false,
"AccountId": ""
}
}
Attempt 2 with .... code examples
This loops through a dict with crawler_name as key and crawler_details as value.
crawlers[crawler_name]['crawler'] = glue.CfnCrawler(self, "CfnCrawler_" + crawler_name,
name=crawler_name,
database_name=crawler_details['database_name'],
description=crawler_details['description'] + ' Delta Lake Crawler',
role=self.glue_crawler_role,
targets={},
# 'DeltaTargets': [
# {
# 'DeltaTables': crawler_details['delta_tables'],
# 'WriteManifest': crawler_details['write_manifest']
# }
# ]
# },
schema_change_policy={
'UpdateBehavior': 'UPDATE_IN_DATABASE',
'DeleteBehavior': 'LOG'
}
)
crawlers[crawler_name]['crawler'].add_property_override("Targets.DeltaTargets", [{
"DeltaTables": crawler_details['delta_tables'],
"WriteManifest": crawler_details['write_manifest']
}])
❌ CrawlersStack failed: Error: The stack named CrawlersStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Property validation failure: [Encountered unsupported properties in {/Targets}: [DeltaTargets]]
at FullCloudFormationDeployment.monitorDeployment (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:505:13)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at deployStack2 (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:265:24)
at /usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/deploy.ts:39:11
at run (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/p-queue/dist/index.js:163:29)
❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named CrawlersStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Property validation failure: [Encountered unsupported properties in {/Targets}: [DeltaTargets]]
at deployStacks (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/deploy.ts:61:11)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at CdkToolkit.deploy (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:339:7)
at exec4 (/usr/local/Cellar/aws-cdk/2.61.1/libexec/lib/node_modules/aws-cdk/lib/cli.ts:384:12)
Hey @Divyam, Thanks for jump in and helping out. I added my attempt with your code above in question. Running with CDK version 2.61.1 (build d319d9c)