Crawler generates table with more columns than expected

Question

Hi,

I've set up a crawler for a bucket with two datasets. I see that the generated table from the crawler has many columns than expected although I created a classifier which specify the column headers

Is there a way to allow the crawler to generate the table with only the headers in the csv

Thanks in advance for your help.

Kind regards, Sarah

Answer

Hi, 
could you please share some additional details?
Do the 2 datasets have the same schema?
does any of the data sets have more columns than the other?
are you expecting one table or 2 tables?

If you expect 2 tables to be cataloged, and the data sets are not too different, you should separate each dataset in its own prefix (folder).

some of the files might be having more columns that you were aware of.

any other details on the classifier and the crawler you created , and on the schema of the 2 datasets may help to provide better guidance.

thank you

Crawler generates table with more columns than expected

Relevant content