By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

How to filter by file type in a Glue table?

0

I have an S3 bucket with mixed files (CSV and JSON), written there by an external process that is outside of my control.

I need to create two Glue tables:

  1. Contains all of the data from CSVs
  2. Contains all of the data from JSONs

Is this possible?

Topics
Analytics
Tags
AWS Glue
Language
English
profile picture
m0ltar
asked a year ago372 views
1 Answer
0

The only standard way I know to do that is creating a symlink table (it has a manifest that points to the specific files, which you would need to update). But normally you would want to convert to columnar formats, standardize, transform, partitions or do other things before you create the table, take that chance to separate the files and generate good tables. I would save my users the pain of using CSV/JSON tables.

profile pictureAWS
EXPERT
Gonzalo Herreros
answered a year ago
  • m0ltar
    a year ago

    The only standard way I know to do that is creating a symlink table (it has a manifest that points to the specific files, which you would need to update).

    Could you please elaborate on this. I am not too familiar with the concept of a "symlink table".

    But normally you would want to convert to columnar formats, standardize, transform, partitions or do other things before you create the table

    And to do that, we want to use Athena, potentially via dbt. The idea was to define base raw tables, with partitions, then compact the data using Athena.

  • Gonzalo Herreros EXPERT
    a year ago

    https://athena.guide/articles/stitching-tables-with-symlinktextinputformat

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content