Crawler pulled Snowflake table, but Athena failed to query it.
Using Glue we can crawl snowflake table properly in catalog but Athena failed to query table data: HIVE_UNSUPPORTED_FORMAT: Unable to create input format
Googling results suggested it's because of table created by crawler has empty "input format", "Output format"... yes they are empty for this table crawled from Snowflake. So the question is 1 why didn't crawler set them? (crawler can classify the table is snowflake correctly) 2 what the values should be if manual edit is needed?
Is Snowflake table able to be queried by Athena?
Any idea? Thanks,
- 최신
- 최다 투표
- 가장 많은 댓글
Hi,
You can query Snowflake using Athena via the Federated query feature of Athena. Crawling the Snowflake tables will add them to your data catalog, but you can't point Athena directly to these tables and query them. I will drop the blog on how to set up the Athena connector for Snowflake, and also a blog on crawling your Snowflake tables and the use cases for that.
Thanks for the links! It seems "crawling Snowflake table schema to catalog" is NOT a way to go. Would you please confirm that?
In general crawled tables are meant to be used by Glue jobs, there is no guarantee that Athena or other tools outside of Glue will be able to read it. But any day Athena can add the support for that kind of table
관련 콘텐츠
AWS 공식업데이트됨 2년 전- AWS 공식업데이트됨 3년 전
That error means Athena doesn't recognize it's a special table and it's trying to use it like a standard Hive table. I don't think it's possible for Athena to query a JDBC table at this time