By using AWS re:Post, you agree to the Terms of Use

create glue meta data tables via CDK without need crawler

0

Hi team,

I have created my glue infra with CDK, jobs, connections, crawlers, and databases,...

I need to run manually the crawler each time and then go over all generated tables by the crawler and add some catalogue table properties and change data types for some columns that have been crawled as bigint but they should be a string.

this process is tedious and tike time, is there a way to get rid of crawlers and create directly the tables via CDK with the right columns, data type, and tables properties ... or do I still need the crawlers?

can create schema catalogue tables, columns, data type, and tables properties via CDK

1 Answer
0

Hello there Jess, It is possible to do this using the AWS Glue construct:

https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-glue-alpha.Table.html

A quick example would be:

declare const myDatabase: glue.Database;
new glue.Table(this, 'MyTable', {
    database: myDatabase,
    tableName: 'my_table',
    columns: [{
        name: 'col1',
        type: glue.Schema.STRING,
    }],
    partitionKeys: [{
        name: 'year',
        type: glue.Schema.SMALL_INT,
    }, {
        name: 'month',
        type: glue.Schema.SMALL_INT,
    }],
    dataFormat: glue.DataFormat.JSON,
    enablePartitionFiltering: true,
});

https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-glue-alpha.Table.html

SUPPORT ENGINEER
answered 2 months ago
EXPERT
reviewed 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions