By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

using athena to read glue database

0

Data layer is not my thing and I need some guidance.

I create a glue crawler to extract compressed JSON files and store them in an aws S3 bucket. I recently learned that I can use Athena to directly connect to the glue database. When I do select * from table-name It starts to load but then errors with long string of stuff

HIVE_METASTORE_ERROR: Error: : expected at the position 407 of 'struct<http:struct<status_code:int,url_details:struct<path:string,queryString:struct<prefix:string,versioning:string,logging:string,encoding-type:string,nodes:string,policy:string,acl:string,policyStatus:string,replication:string,notification:string,tagging:string,website:string,encryption:string,size:string,limit:string,hash:string,accelerate:string,publicAccessBlock:string,code:string,protocol:string,G%EF%BF%BD%EF%BF%BD%EF%BF%BD,%EF%BF%BD%EF%BF%BD%EF%BF%BD`~%EF%BF%BD%00%EF%BF%BD%EF%BF%BD{%EF%BF%BD%D5%96%EF%BF%BDw%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BD%3C:string,cors:string,object- etc etc etc.

I can load one small table but the others fail.

  • Gonzalo Herreros EXPERT
    a year ago

    json tables are often problematic and not always compatible between different tools. That part where it starts G%EF% doesn't look right, look at the table definition, are you sure the json is valid and not using strange characters for the property names?

Topics
Analytics
Tags
Amazon AthenaAnalyticsAWS Glue
Language
English
rePost-User-6916425
asked a year ago216 views
1 Answer
0

It seems like you are encountering an error when querying your Glue database using Athena. The error message you provided suggests an issue with the Hive metastore, which is used by Athena to manage the metadata for your tables.

Here are a few steps you can take to troubleshoot the issue:

Check the Data Catalog Configuration: Ensure that the Glue Data Catalog is properly configured and integrated with Athena. Verify that the database and table metadata in the Data Catalog are accurate and up to date. You can navigate to the Glue Data Catalog in the AWS Management Console and review the tables and their schemas to confirm their correctness.

Validate the Table Schema: Double-check the table schema in the Glue Data Catalog. Make sure that the schema matches the structure of your compressed JSON files stored in the S3 bucket. Any mismatch between the expected schema and the actual data can result in errors.

Data Format and Compression: Confirm that the compressed JSON files in your S3 bucket are in a format that Athena can handle. Athena supports various file formats like Parquet, ORC, Avro, and JSON. Ensure that the compression used in your JSON files (if any) is compatible with Athena. For example, if your files are compressed using Gzip, Athena can handle it, but if they are compressed using other formats like LZ4 or Bzip2, additional steps might be needed.

Query Optimization: If the error occurs when querying large or complex tables, it's possible that the query is hitting resource limits or encountering performance issues. Consider optimizing your queries by applying filters, reducing the amount of data scanned, or partitioning the tables to improve query performance.

Review Query Syntax: Inspect the syntax of your query to make sure there are no typographical errors or issues with the query structure. Small mistakes in the query can lead to unexpected errors.

Review Athena and Glue Logs: Check the query execution logs in Athena and Glue for more detailed error messages. The logs might provide additional information about the specific error and its cause. You can find the logs in the AWS Management Console by navigating to the Athena and Glue service pages and looking for the relevant log sections.

Contact AWS Support: If you've exhausted all troubleshooting options and the issue persists, consider reaching out to AWS Support for further assistance. They can analyze the logs and provide more targeted guidance based on your specific scenario.

By following these steps, you should be able to identify and resolve the issue causing the errors in Athena when querying your Glue database.

Saran Raj J
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content