create glue meta data tables via CDK without need crawler

Question

Hi team,

I have created my glue infra with CDK, jobs, connections, crawlers, and databases,...

I need to run manually the crawler each time and then go over all generated tables by the crawler and add some catalogue table properties and change data types for some columns that have been crawled as bigint but they should be a string.

this process is tedious and tike time, is there a way to get rid of crawlers and create directly the tables via CDK with the right columns, data type, and tables properties ... or do I still need the crawlers?

can create schema catalogue tables, columns, data type, and tables properties via CDK

Answer

Hello there Jess,
It is possible to do this using the AWS Glue construct:

https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-glue-alpha.Table.html

A quick example would be:

```typescript
declare const myDatabase: glue.Database;
new glue.Table(this, 'MyTable', {
    database: myDatabase,
    tableName: 'my_table',
    columns: [{
        name: 'col1',
        type: glue.Schema.STRING,
    }],
    partitionKeys: [{
        name: 'year',
        type: glue.Schema.SMALL_INT,
    }, {
        name: 'month',
        type: glue.Schema.SMALL_INT,
    }],
    dataFormat: glue.DataFormat.JSON,
    enablePartitionFiltering: true,
});
```

https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-glue-alpha.Table.html

create glue meta data tables via CDK without need crawler

Conteúdo relevante