Skip to content

Glue crawler: exclude field or predefine it's type

0

I am getting json files to my s3. For example:

{
    "name" : "John",
    "lastname": "Doe",
    "meta" : {
        "x": "a",
        "y": "b",
        "unwanted_field": {
            "some": "stuff",
            "to": "ignore"
        }
    }
}

Is there a way exclude meta.unwanted_field key or predefine it's type as STRING during crawling process? Maybe somebody has a solution how I can achieve this without file reprocessing?

Thank you for any help.

asked a year ago356 views
1 Answer
0

Yes, you can prevent a document or several from being read using glob pattern, in the field where I indicate with an arrow

Enter image description here

EXPERT
answered a year ago
EXPERT
reviewed a year ago
  • this functionality let's you to exclude whole file based on name pattern. i want to exclude key within the file.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.