Questions tagged with Amazon Athena

Can the timeout or resources used by Athena to run a query be configured? I have query which spans over a year of data but it times out after 30 mins after having ready about 54GB.

_temp lake formation blueprint pipeline tables appears to IAM user in Athena editor, although I didn't give this user permission on them below the policy granted to this IAM user,also in lake formation permsissions ,I didnt give this user any permissions on _temp tables: { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1652364721496", "Action": [ "athena:BatchGetNamedQuery", "athena:BatchGetQueryExecution", "athena:GetDataCatalog", "athena:GetDatabase", "athena:GetNamedQuery", "athena:GetPreparedStatement", "athena:GetQueryExecution", "athena:GetQueryResults", "athena:GetQueryResultsStream", "athena:GetTableMetadata", "athena:GetWorkGroup", "athena:ListDataCatalogs", "athena:ListDatabases", "athena:ListEngineVersions", "athena:ListNamedQueries", "athena:ListPreparedStatements", "athena:ListQueryExecutions", "athena:ListTableMetadata", "athena:ListTagsForResource", "athena:ListWorkGroups", "athena:StartQueryExecution", "athena:StopQueryExecution" ], "Effect": "Allow", "Resource": "*" }, { "Effect": "Allow", "Action": [ "glue:GetDatabase", "glue:GetDatabases", "glue:BatchDeleteTable", "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions", "glue:BatchGetPartition" ], "Resource": [ "*" ] }, { "Sid": "Stmt1652365282568", "Action": "s3:*", "Effect": "Allow", "Resource": [ "arn:aws:s3:::queryresults-all", "arn:aws:s3:::queryresults-all/*" ] }, { "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": [ "*" ] } ] }

I followed the instructions from https://docs.aws.amazon.com/athena/latest/ug/connect-to-a-data-source-lambda.html to set up the connection between Athena and snowflake. However I'm getting an error when swtiching to the new data sources from Athena interface, the error msg is: Failed to invoke lambda function due to com.amazonaws.services.lambda.invoke.LambdaFunctionException: 0: java.sql.SQLException: No suitable driver found for jdbc:snowflake://<my connection string> Am I missing something when deploying the connector? Thanks.

I'm having trouble creating iceberg tables in Athena. I keep getting "Invalid S3 request" as an error message. ``` CREATE TABLE iceberg_test ( event_date timestamp, log string ) PARTITIONED BY ( hour(event_date) ) LOCATION 's3://bucket/iceberg/test' TBLPROPERTIES ( 'table_type'='ICEBERG' ); ```

I receive this error when I try to run a query by athena-express library. [ErrorCode: INTERNAL_ERROR_SERVICE] Amazon Athena experienced an internal error while executing this query. Please contact AWS support for further assistance. You will not be charged for this query. We apologize for the inconvenience. `Query: DROP SCHEMA IF EXISTS name_schema CASCADE` Run time: 0 sec Data scanned: 0 MB Till yesterday I didn't see any errors!

I just want to re-raise an issue we've been seeing for a long time, preventing us from querying Hudi MoR realtime tables using AWS Athena. AWS Athena snapshot queries fails if there are two or more record array fields in a Hudi MoR table: `HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://prod-signal-data-platform-spark-experiments/temporary/articles_hudi_merge_on_read_dev/story_published_partition_date=2022-04-29/75f5af94-8369-4b04-b850-421826ef48cc-0_24-38-3759_20220506091903372.parquet (offset=33554432, length=33554432) using org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat: Can't redefine: array This query ran against the "articles" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 7a64950b-624d-4ea3-91d0-33be7a5e74b5` This corresponds to this Apache Hudi issue: https://github.com/apache/hudi/issues/3834 (fixed in Hudi 0.11.0) and requires an AWS Athena change. I'd be great to see an AWS fix for this issue to allow us to fully leverage Hudi on AWS.

Hi After creating quite complex view based on "UNION ALL" of 4 complex selects I've ended up with a curious situation: 1. The view itself is queried OK with simple queries like `select * from <view_name> where <conditions>` 2. Once I wrap it into bigger select like `select count(*) from (select * from <view_name> where <conditions>)` I get the following strange error: `GENERIC_INTERNAL_ERROR: Missing variable: <column_name>` where `<column_name>` is absolutely live and valid column that could be queried and returned as usual The issue is repeatable - once I recreate the same set of views on another Database and same data - issue repeats However it's barely searched on the web - looks like nobody faced such errors

What is the best way to sync my DynamoDB tables to S3, so that I can perform serverless 'big data' queries using Athena? The data must be kept in sync without any intervention. The frequency of sync would depend on the cost, ideally daily but perhaps weekly. I have had this question a long time. I will cover what I have considered, and why I don't like the options. 1) AWS Glue Elastic Views. Sounds like this will do the job with no code, but it was announced 18 months ago and there have been no updates since. Its not generally available, and there is not information on when it might be. 2) Use dynamodb native backup following this blog https://aws.amazon.com/blogs/aws/new-export-amazon-dynamodb-table-data-to-data-lake-amazon-s3/. I actually already use this method for 'one-off' data transfers that I kick-off manually and then configure in Athena. I have two issues with this option. The first is that, to my knowledge, the export cannot be scheduled natively. The blog suggests using the CLI to kick off exports, and I assume the writer intends that the CLI would need scheduling on a cron job somewhere. I don't run any servers for this. I imagine I could do it via a scheduled Lambda with an SDK. The second issue is that the export path in S3 always includes a unique export ID. This means I can't configure the Athena table to point to a static location for the data and just switch over the new data after a scheduled export. Perhaps I could write another lambda to move the data around to a static location after the export has finished, but it seems a shame to have to do so much work and I've not seen that covered anywhere before. 3) I can use data pipeline as described in https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBPipeline.html. This post is more about backing data up than making it accessible to Athena. I feel like this use case must be so common, and yet none of the ideas I've seen online are really complete. I was wondering if anyone had any ideas or experiences that would be useful here?

Imagine if we have a time series values like this, lets say the timestamps are in epoch and device is emitting a payload every minute. |timestamp| device| sensorName| sensorValue| |---|---|---|---| |1612828800001| device1| mode | Running| |1612828800001| device1| fuelUsed |23| |1612828800011| device1| mode | Running| |1612828800011| device1| fuelUsed |33| |1612828800021| device1| mode | Running| |1612828800021| device1| fuelUsed |50| |1612828800002| device1| mode | Not Running |1612823800002| device1| fuelUsed | 45| |1612828800002| device2| mode | Running| |1612823800002| device2| fuelUsed | 77| And we would like to know what was the TotalfuelUsed in the last 30 days when the mode was Running (mode Not Running means engine is on but not moving = idle). We tried using calculated field and periodToDateAvg function but somehow it is just not possible to do this complex query in one or multiple calculations * Calculated Field 1: Productive Productive time = sum(periodToDateAvg({sensorName} = 'mode' and {sensorValue} = 'Running', Month )) <<<---- Need help here * Calculated Field 1: TotalFuelUsedDuringProductive TotalFuelUsedDuringProductive = TotalFuelUsed/Productive * Expected Output device1 TotalFuelUsedDuringProductive = (50-23)/(3-2) Is this even possible to do in Quicksight? or do we have to use Athena SQL Query?

I am trying to create a view using a table have more than 5 billion rows. using the following query:- with date_month as (select distinct as_on_month from prod_edp_db_trusted.keyword_metrics_serp_dataforseo_v2_details where as_on_date >= date_add('Month',-2,current_date) ), sim_data as (select *,date as sim_date,substring(date,1,4)||substring(date,6,2) as sim_yr_month from ( select tenant,locale,search_engine,device_type,url,keyword,traffic,cast(date as varchar(255)) as date,cast(organic_rank as decimal(38,0)) as organic_rank,create_date ,row_number() over(partition by tenant,locale,search_engine,device_type,url,keyword order by date desc,create_date) as rn from prod_edp_db_trusted.simulated_score_v3_4 WHERE save_type='simulation' ) where rn=1 ), serp_data as ( select * from( select sim.tenant,sim.locale,sim.search_engine,sim.device_type,sim.url,sim.keyword,sim.traffic,sim.organic_rank as sim_rank,sim.sim_date as simulation_date,sim.sim_yr_month,srp.position as rank, srp_mn.as_on_month as serp_year_month, srp.as_on_date as serp_as_on_date, row_number() over(partition by sim.tenant,sim.locale,sim.search_engine,sim.device_type,sim.url,sim.keyword,srp_mn.as_on_month order by srp.as_on_date desc,srp.created_date) as rn from sim_data sim join date_month as srp_mn on srp_mn.as_on_month<=sim.sim_yr_month or srp_mn.as_on_month>=sim.sim_yr_month left join prod_edp_db_trusted.keyword_metrics_serp_dataforseo_v2_details srp on srp.locale=sim.locale and srp.search_engine=sim.search_engine and srp.device_type=sim.device_type and srp.url=sim.url and srp.keyword=sim.keyword --and srp.as_on_date>=sim.sim_date and srp.as_on_month=srp_mn.as_on_month and srp.as_on_date>=date_add('Month',-2,date_trunc('Month',current_date)) ) where rn=1 ), sv_data AS ( select tenant,locale,url,search_engine,keyword,search_volume,sv_yr_month from ( select LKP.tenant ,MAIN.locale ,lkp.url ,MAIN.search_engine ,MAIN.keyword ,MAIN.count as search_volume ,MAIN.as_on_month as sv_yr_month ,row_number() over(partition by lkp.tenant,main.locale,lkp.url,main.search_engine,main.keyword ,as_on_month order by as_on_date desc,created_date) rn from prod_edp_db_trusted.keyword_metrics_search_volume_dataforseo_v2 MAIN JOIN (SELECT tenant,locale,url,search_engine,keyword,Min(SIM_YR_MONTH) AS SIM_YR_MONTH FROM sim_data GROUP BY 1,2,3,4,5) LKP ON MAIN.locale=LKP.locale AND MAIN.search_engine=LKP.search_engine and main.keyword=LKP.keyword where MAIN.as_on_month IN(SELECT * FROM date_month ) ) where rn=1 ), base_dataset AS ( select srp.tenant,srp.locale,srp.search_engine,srp.device_type,srp.url,srp.keyword,srp.sim_rank,srp.rank,srp.serp_year_ month,srp.traffic ,srp.serp_as_on_date,srp.simulation_date,srp.SIM_YR_MONTH ,sv.search_volume from serp_data srp left join sv_data sv on srp.tenant=sv.tenant and srp.locale=sv.locale and srp.search_engine=sv.search_engine and srp.keyword=sv.keyword and srp.url=sv.url and srp.serp_year_month=sv.sv_yr_month ), sim_wavg as ( (select * from base_dataset where sim_yr_month=serp_year_month) union (select tenant,locale,search_engine,device_type,url,keyword,sim_rank,rank,serp_year_month,traffic ,serp_as_on_date,simulation_date,SIM_YR_MONTH ,search_volume from (select *,row_number() over(partition by tenant,locale,search_engine,device_type,url,keyword,simulation_date, SIM_YR_MONTH order by serp_year_month) base_rn from base_dataset where sim_yr_month<(select min(as_on_month) from date_month))t3 where base_rn=1 ) ), sim_latest_mnth as ( select * FROM ( select *,row_number() over(partition by tenant,locale,search_engine,device_type,url,keyword ,simulation_date, SIM_YR_MONTH order by serp_year_month desc) base_rn from base_dataset ) t3 where base_rn=1 ), final_base AS ( select sim.tenant,sim.locale,sim.search_engine,sim.device_type,sim.url,sim.keyword,sim.sim_rank,sim.search_volume ,lkp.serp_year_month,lkp.serp_as_on_date,lkp.RANK as SERP_Rank,lkp.traffic ,sim.simulation_date,sim.SIM_YR_MONTH,lkp.search_volume as month_sv ,lst.search_volume as latest_month_sv,sim.sim_rank-lst.rank as rank_diff from sim_wavg sim left join base_dataset lkp on sim.tenant=lkp.tenant and sim.locale=lkp.locale and sim.search_engine=lkp.search_engine and sim.device_type=lkp.device_type and sim.url=lkp.url and sim.keyword=lkp.keyword left join sim_latest_mnth lst on sim.tenant=lst.tenant and sim.locale=lst.locale and sim.search_engine=lst.search_engine and sim.device_type=lst.device_type and sim.url=lst.url and sim.keyword=lst.keyword ) select tenant,locale,search_engine,device_type,url,keyword ,search_volume as "Simulation Month Search Volume",simulation_date,latest_month_sv,traffic as "SIMULATION MONTH TRAFFIC" ,sim_rank as "Rank as on Last Simulation",rank_diff as "Rank Difference With Latest Month" ,kw_imp['202204'] as Current_Month ,kw_imp['202203'] as Last_Month ,kw_imp['202202'] as "2nd_Last_Month" ,kw_imp['202201'] as "3rd_Last_Month" FROM ( select tenant,locale,search_engine,device_type,url,keyword,search_volume,latest_month_sv,simulation_date,traffic,si m_rank ,rank_diff ,map_agg(serp_year_month,SERP_Rank) as kw_imp from final_base group by tenant,locale,search_engine,device_type,url,keyword,search_volume,latest_month_sv,simulation_date,traffic,si m_rank ,rank_diff ) order by tenant,locale,search_engine,device_type,url,keyword;

I have created a database using AWS Lake Formation, and populated it with two tables created using Glue crawlers. The tables seem to be created correctly (all of the columns have been properly mapped out by the crawlers). However, then I try to query them using AWS Athena, I am getting the following error: `HIVE_UNKNOWN_ERROR: com.amazonaws.services.lakeformation.model.InvalidInputException: Unsupported vendor for Glue supported principal` I assume this has to do with the permissions associated with Lake Formation, but I have given the IAM User all possible permissions I could think of in the Lake Formation Console Permissions section. Does anyone know what the problem could be here?

[After resolving my first issue](https://repost.aws/questions/QUXOInFRr1QrKfR0Bh9wVglA/aws-glue-not-properly-crawling-s-3-bucket-populated-by-resource-data-sync-specifically-aws-instance-information-is-not-made-into-a-table) with getting a resource data sync set up, I've now run into another issue with the same folder. When a resource data sync is created, it creates a folder structure with 13 folders following a folder structure like: `s3://resource-data-sync-bucket/AWS:*/accountid=*/regions=*/resourcetype=*/instance.json}` When running the glue crawler over this, a schema is created where partitions are made for each subpath with an `=` in it. This works fine for most of the data, except for the path starting with `AWS:InstanceInformation`. The instance information json files ALSO contain a "resourcetype" field as can be seen here. ``` {"PlatformName":"Microsoft Windows Server 2019 Datacenter","PlatformVersion":"10.0.17763","AgentType":"amazon-ssm-agent","AgentVersion":"3.1.1260.0","InstanceId":"i","InstanceStatus":"Active","ComputerName":"computer.name","IpAddress":"10.0.0.0","ResourceType":"EC2Instance","PlatformType":"Windows","resourceId":"i-0a6dfb4f042d465b2","captureTime":"2022-04-22T19:27:27Z","schemaVersion":"1.0"} ``` As a result, there are now two "resourcetype" columns in the "aws_instanceinformation" table schema. Attempts to query that table result in the error `HIVE_INVALID_METADATA: Hive metadata for table is invalid: Table descriptor contains duplicate columns` I've worked around this issue by removing the offending field and setting the crawler to ignore schema updates, but this doesn't seem like a great long term solution since any changes made by AWS to the schema will be ignored. Is this a known issue with using this solution? Are there any plans to change how the AWS:InstanceInformation documents are so duplicate columns aren't created.

I set up an s3 bucket that collects inventory data from multiple AWS accounts using the Systems Manager "Resource Data Sync". I was able to set up the Data Syncs to feed into the single bucket without issue and the Glue crawler was created automatically. Now that I'm trying to query the data in Athena, I noticed there is an issue with how the Crawler is parsing the data in the bucket. The folder "AWS:InstanceInformation" is not being turned into a table. Instead, it is turning all of the "region=us-east-1/" and "test.json" sub-items into tables which are, obviously, not queryable. To illustrate further, each of the following paths is being turned into it's own table. * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=12345679012/region=us-east-1 * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=12345679012/test.json * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=23456790123/region=us-east-1 * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=23456790123/test.json * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=34567901234/region=us-east-1 * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=34567901234/test.json This is ONLY happening with the "AWS:InstanceInformation" folder. All of the other folders (e.g. "AWS:DetailedInstanceInformation") are being properly turned into tables. Since all of this data was populated automatically, I'm assuming that we are dealing with a bug? Is there anything I can do to fix this?

After running a simple select query in Athena, I get an error - **HIVE_UNKNOWN_ERROR: Error creating an instance of com.facebook.presto.hive.lakeformation.CachingLakeFormationCredentialsProvider**. Athena query is supposed to run agains a LF database within the same S3 bucket (data source and query destination are split into two folders inside the bucket). Any idea what could be the issue?

I created a Athena data source connector, which can query dynamodb data successfully. However, when I tried to create a new dataset in QuickSight with Athena, it can NOT see my 'data source connector'? Do I miss anything? Thanks!

I created a Athena connector for my DynamoDB. However, when running a simple select query, I run into the following errors. Any help will be appreciated! GENERIC_USER_ERROR: Encountered an exception[Runtime.ExitError] from your LambdaFunction[arn:aws:lambda:us-west-1:541750221373:function:my_test_catalog] executed in context[retrieving meta-data] with message[RequestId: 8af8027b-2385-4279-ac6c-d45773912907 Error: Runtime exited with error: exit status 1] This query ran against the "default" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 542232b7-3070-4793-a6f5-78a653164712

Getting below error while querying S3 parquet file from Athena. The schema has changed from timestamp to string for one of the columns, is there any workaround to have the latest schema changed to string (Alter table and msck repair table didn't work). HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'birth_year' in table 'curated_ndb_datapipeline.registered_nurse' is declared as type 'timestamp', but partition 'data_year=2021/prov_terr_reg=959' declared column 'birth_year' as type 'string'

Following the well architected labs 200: Cost and usage analysis I get the following error when adding partitions in Athena Query Editor: ``` MSCK REPAIR TABLE `cost_optimization_10XXXXXXXX321`; ``` and it returned the following error: > FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. null This query ran against the "costfubar" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 856e146a-8b13-4175-8cd8-692eef6d3fa5 The table was created correctly in Glue with ``` Name cost_optimization_10XXXXXXXXX21 Description Database costfubar Classification parquet Location s3://cost-optimization-10XXXXXXX321// Connection Deprecated No Last updated Wed Apr 20 16:46:28 GMT-500 2022 Input format org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat Output format org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat Serde serialization lib org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe Serde parameters serialization.format 1 Table properties sizeKey 4223322objectCount 4UPDATED_BY_CRAWLER costfubarCrawlerSchemaSerializerVersion 1.0recordCount 335239averageRecordSize 27exclusions ["s3://cost-optimization-107457606321/**.json","s3://cost-optimization-1XXXXXXXX21/**.csv","s3://cost-optimization-107457606321/**.sql","s3://cost-optimization-1XXXXXXXX321/**.gz","s3://cost-optimization-107457606321/**.zip","s3://cost-optimization-107457606321/**/cost_and_usage_data_status/*","s3://cost-optimization-107457606321/**.yml"]CrawlerSchemaDeserializerVersion 1.0compressionType nonetypeOfData file ``` and has the following partitions shown in Glue: ``` partition_0 partition_1 year month detailed-cur-1XXXXXXXX57 detailed-cur-1XXXXXXXX57 2018 12 View files View properties detailed-cur-1XXXXXXXXX57 detailed-cur-1XXXXXXXXX57 2022 4 View files View properties detailed-cur-1XXXXXXXXX57 detailed-cur-1XXXXXXXXX57 2018 11 View files View properties detailed-cur-1XXXXXXXX57 detailed-cur-1XXXXXXXX57 2018 10 View files View properties ```

[ErrorCode: INTERNAL_ERROR_QUERY_ENGINE] Amazon Athena experienced an internal error while executing this query.

I've this CSV content file "one","two","three","four","five","six","seven","eight","nine","ten" "one","two","three","four","five \" quote \" five2","six","seven","eight","nine","ten" "one","two","three","four","five \ five2","six","seven","eight","nine","ten" * Row 2 is normal * Row 3 has a field with \" escaped quotes * Row 4 has escaped newlines This table is not imported correctly due to newlines. Is there any other way? I found this way but it's not correct ``` CREATE EXTERNAL TABLE IF NOT EXISTS `test-general-log`.`general-log` ( `A` string, `B` string, `C` string, `D` string, `E` string ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = '"*([^"]*)"*,"*([^"]*)"*' ) LOCATION 's3://test-general-log/exportedlogs/0baa57384-2c6e-4e016-85f0-bc58b31e6300/' TBLPROPERTIES ('has_encrypted_data'='false'); ```

I ran the following Amazon Athena query on a table created by AWS Glue, which had crawled an Amazon S3 export of Synthea data from Amazon HealthLake: ``` SELECT * FROM "glue01234567890_fhir_export_abcdefghijklmnopqrst"; ``` That resulted in this error: > HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table and partition schemas. The types are incompatible and cannot be coerced. The column 'code' in table 'demo.glue01234567890_fhir_export_abcdefghijklmnopqrst' is declared as type 'struct<coding:array<struct<system:string,code:string,display:string>>,text:string>', but partition 'partition_0=ImagingStudy' declared column 'id' as type 'string'. This query ran against the "demo" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id I saw [one answer](https://repost.aws/questions/QU1vPk1069Q5qg4iyuQWTk6Q/while-importing-s-3-data-into-quicksight-from-glue-database-getting-the-following-error) to go to edit the Crawler and then to output -->configuration and "Update all new and existing partitions with metadata from the table." However, that didn't resolve the error for me. When I edit the schema manually an change column 'code' from `struct` to `string`, the error goes away, and Athena brings my attention to the next mismatch: > partition 'partition_0=Immunization' declared column 'patient' as type 'struct<reference:string>'. I don't think, however, manually replacing all structs with strings will get me the results I want, because I need Glue and Athena to be aware of the fields inside the structs. What is the best approach to overcoming this error while preserving the structure of the data, in this situation where I want to make the HealthLake export queryable by Athena? I understand I'll want to add a step to the ETL in Glue to first convert to Parquet or ORC for Athena performance reasons, but right now I'm dealing with a small sample dataset and just want to focus on getting a minimum viable end-to-end flow going.

Anywhere i can get further context to this error? I can download this file and decode it with out issue using the same avro schema locally, i'm assuming i have not done something correctly in the athena SerDe setup or something like that. HIVE_CANNOT_OPEN_SPLIT: Error opening Hive split s3://si-kafkaconnect/topics/lidar_analysis/year=2022/month=03/day=22/hour=08/lidar_analysis+0+0371435909.bin (offset=0, length=3287906) using org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat: Not a data file.

I have a S3 bucket with some AVRO formatted files, I have successfully been through the steps to create a table based on the AVRO schema in Athena and can run SQL queries on the data that return a successful "Completed" response but no data is returned, In the query result bucket i have specified a csv file is created for each query that contains a Error xml block and a "Access Denied" message. I believe i have set up the bucket policy correctly, i am attempting to run this query on an user that is under the same account ID that the bucket was created in, so that seems like it should be straight forward. but it is not working for me and the lack of errors in Athena are confusing. I am stuck right now on what i have not set up correctly.

1. What is the best way to Create a subset of factory location data Current process: Query location data for specific factories, save in a new Athena table with a direct insert statement 2. Get factory data and generate statistics about the factories Objective: save statistics in a database that automatically updates to a visualization tool 3. Is it possible to do a batch insert to put the contents of a pandas dataframe into Athena? 3. What is the best tool to connect Athena data to excel?

**Situation**: If only specify the partition clause, it will be divided into multiple files. The size of one file is less than 1MB (~ 40 files). **What I am thinking o**f: I want to explicitly specify the size of the files to be split or the number of files when registering data with CTAS or INSERT INTO. I have read this article: https://aws.amazon.com/premiumsupport/knowledge-center/set-file-number-size-ctas-athena/ **Problem**: Using bucketing method (like said in above article ) can help me specify the number of file or file size. However, it also said that "Note: The INSERT INTO statement isn't supported on bucketed tables". I would like to register data daily with Athena's INSERT INTO. **Question**: what is the best way to build a partitioned data mart without compromising search efficiency? Is it best to register the data with Glue and save it as one file?

**ERROR MESSAGE:** An error occurred while calling o518.pyWriteDynamicFrame. Unsupported case of DataType: com.amazonaws.services.glue.schema.types.StringType@235d3a6f and DynamicNode: integernode. I utilize glue catalog and use s3 as my DB. I'm uploading CSV files and processing them with glue jobs, however I continue to get the error message **"An error occurred while calling o518.pyWriteDynamicFrame. Unsupported case of DataType: com.amazonaws.services.glue.schema.types.StringType@235d3a6f and DynamicNode: integernode.Datatype". ** I've updated and recreated the glue tables and crawlers multiple times as well as the schema to match data types but I still get the same error message. The glue job only succeeds when I update the schema to have string as the only data type but the data type needs to be int for integers when querying in Athena so this will cause an issue down stream. Can anyone help?

I am using Athena to query my Cloudtrail logs to find security groups that have rules that allow access to/from the internet. To achieve this, I am looking to see if any cidrIp in requestParameters.ipPermissions.items[].ipRanges.items[] equals '0.0.0.0/0' or '::/0'. I have constructed a query that does this (pasted below), but it is very lengthy. I would like to know if there is a better way to write this specific query or query nested arrays in general. ``` WITH parent as (SELECT items FROM "default"."my_cloudtrail_logs" CROSS JOIN UNNEST (cast(json_extract(requestParameters,'$.ipPermissions.items') AS array(json))) AS i (items) WHERE eventname = 'AuthorizeSecurityGroupIngress' AND eventtime >= '2021-04-06T02:00:00.000' order by eventtime desc) Select * FROM parent CROSS JOIN UNNEST (cast(json_extract(items,'$.ipRanges.items') AS array(json))) AS t (cidr_items) WHERE json_extract_scalar(cidr_items, '$.cidrIp') in ('0.0.0.0/0', '::/0') ```

* At a high level, I have a Glue Crawler that generates a table based on CloudTrail logs in an S3 bucket. The problem is that Athena always generates an error when trying to run the preview query over the generated table. The preview query is just selecting the first 10 rows in the table. I can't get any Athena queries to work on this table. * The error I'm receiving is very similar to the first error under the short description heading on this page: https://aws.amazon.com/premiumsupport/knowledge-center/athena-hive-metastore-error/ * I did notice that some of the column names in my partition did have hyphens because the folders in S3 for the regions have hyphens. After converting the folders with hyphens to underscores in S3 and rerunning the Glue Crawler, I noticed the partition column names that were previously hyphens were now underscores; however, I still received the same hive metatstore error. At this point, I am stumped.

[Simba][AthenaJDBC](100071) An error has been thrown from the AWS Athena client. Row is not a valid JSON Object - JSONException: Expected a ',' or '}' at 149 [character 150 line 1] [Execution ID: fd5f862a-7558-4329-93df-08ba753e7ee1]

Please help me on how to execute multiple query and capture the output in a file on e2 **Tried with :** `aws athena start-query-execution --query-string "<select query>" > query_output.sql` But above statement is giving only key in the file query_output.sql but I want the output to be capture in a file, please help

This is the query I use in the view to put the string to date: "date_parse"("ctr"."currentagentsnapshot"."agentstatus"."starttimestamp", '%Y-%m-%dT%H:%i:%sZ') "StartTimestamp" starttimestamp=2022-04-05T01:03:08.234Z I am getting the below error. Can anyone help, I am guessing there is an issue with '%sZ': INVALID_FUNCTION_ARGUMENT: Invalid format: "2022-04-04T23:04:59.001Z" is malformed at ".001Z" This query ran against the "connect_metrics" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: b4920385-1417-40aa-b788-ae7e9432cbd7 Thanks John

A crawler reported it created a table but the table is not visible in the Glue console under tables. * I can see the table in Athena and when I query it data is returned as expected * When I use the AWS CLI command `aws glue get-tables` I see the table in the results * When I use this table in an ETL job with `glueContext.create_dynamic_frame.from_catalog()` no data is returned. Can anyone explain what is happening here? Why does the table show and work in Athena, but not show in the glue console, and not return data in my glue job?

I have access denied issue when running AWS sagemaker and running a script that will access AWS Athena I can execute the Athena query directly via the AWS console But when using sagemaker executing the query, I got following error ``` Permission denied on S3 path: s3://... This query ran against the "mydatabse" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 3a825c36-7389-409c-be1e-34d4bcd4256a ``` I'm pretty sure I provide the sagemaker execution role all the s3:* permission on my bucket but still couldn't access it. Please help me solving the issue.

I am seeing Athena queries over a bucket containing ORC files fail with the error message 'HIVE_CURSOR_ERROR: Failed to read ORC file'. Any query over entirety of the data in the bucket fails. A specific example query has been ```SELECT * FROM reachcounts_outbound WHERE calculation='a8d9458d-83e2-4e94-b272-3dbcd91296a0'``` where calculation is set up as a partition in the reachcounts_outbound table (which is backed by an S3 bucket unscoreit-reachcounts-outbound). I've validated that the file referenced by the error message is a valid ORC file by downloading it and running ```orc-tools data ``` on it, and the contents are what I'd expect. I've downloaded other ORC files in the bucket and compared them. They have the same schema and that schema is what I'd expect it to be; it matches the schema I've defined for the table. I've tried deleting the individual file referenced when the error message first appeared. However, it continues to fail with the same message with a different file in the bucket. However if I specify a limit clause of any number under 1597894 on the query above, it will succeed. I've tried running MSCK REPAIR TABLE on the reachcounts_outbound table. This did not change anything. The query id of a request that caused a failure is ```54480f27-1992-40f7-8240-17cc622f91db```. Thanks! Update: The ORC files that are rejected all appear to have exactly 10,000 rows, which is the stride size for the file

getting this error from athena "HIVE_CURSOR_ERROR: java.lang.UnsupportedOperationException: The separator, quote, and escape characters must be different!". This error is showing up only for some particular partitions.

I'm trying to schedule a data transformation with Athena using python and boto 3 (via glue). Once the query is launched, the data should be stored at an S3 sub-bucket. I need the subbucket to have just the data, but the query creates a metadata file. I didn't find a way to avoid the query to create the metadata file I'm using the start_query_execution from boto 3 to run the query: ``` queryStart = client.start_query_execution( QueryString = query, QueryExecutionContext = { 'Database': database }, ResultConfiguration = { 'OutputLocation': 's3://' + bucket + '/' + subbucketpath} ) ``` I tried with the below function to remove the metadata file ``` s3 = session.resource('s3') my_bucket = s3.Bucket(bucket) for item in my_bucket.objects.filter(Prefix=subbucketpath): if item.endswith('.csv.metadata'): item.delete() ``` but it gives an error: ` AttributeError: 's3.ObjectSummary' object has no attribute 'endswith'`. Is there any other way to launch the Athena query from Glue or to remove the '.csv.metadata' files?

Hi, I´m doing some reasearch to find the best place to centralize lots of data logs generated by my application considering pricing ,performance and scalabilty. Today all my application data including logs are stored on an Oracle database, but I´m thinking to move all the LOG related data outside it to reduce it´s size and not to worry about storage performance etc... Just put everything on a "infinite" storage apart from my actual database using CDC or a regular batch process **Below are some needs:** - Only inserts are necessary (no updates or deletes) - Customers will need access to this historic data - Well defined pattern of access (one or two indexes at maximum) - Latencies of few seconds is ok - Avoid infrastrucure, DBA, perfomance bottleneck log term... - Infinite Retentiton period (means I don´t want to worry about performance issues, storage size in long term. But something that can handle a few terabytes of data ) **Use case example: ** Historical Sales order by items ( id_item | id_customer | qty_sold | date_inserted ... ), aprox 50 millions records per day Where I would need to see the historical data by item, and by customer for example (two dimensions) I´ve done some research with the options below **S3 + Athena **-> Put everthing on s3, no worries about infrastructure perfomance issues, however as I need query by item and customer, probably it´would be necessary to break files by item or customer , generate millions of partitions to avoid high costs searching on every file etc.. **Postgre** -> Not sure if could be performance bottleneck once tables gets too big even with partition strategies **DynamoDB **-> Not sure if it´s a good alternative to historical data regarding pricing once seconds latency is ok **MongoDB/ DocumentDB **-> Not very familiar with it (I´d prefer SQL language type) but I know it´s has a good scalability **Cassandra**-> dont´know very much **Timeseries db as influxDB, timestream etc..**-> dont´know very much, but it seems appropriate for timeseries What option would you choose ? Sorry in advance if I saying something wrong or impossible :) Thank you!

I am reading files from S3 and using a Glue ETL job to populate a Data Catalog Table. My S3 files look like this: ``` VID,ALTID,VTYPE,TIME,TIMEGMT,VALUE ABC, ABC, DATAFEED,31-10-2021 22:00:00,01-11-2021 02:00:00, 11775.685 ``` The scheme read in looks like this ``` root |-- VID: string |-- ALTID: string |-- VTYPE: string |-- TIME: string |-- TIMEGMT: string |-- VALUE: string ``` I am changing fields 3 and 4 from "strings" to timestamps, which matches the schema of my Data Catalog Table. I am also doing a few other transformations. I am transforming them like so: ``` df = df.withColumn("time", to_timestamp("time", 'dd-MM-yyyy HH:mm:ss')) df = df.withColumn("timegmt", to_timestamp("timegmt", 'dd-MM-yyyy HH:mm:ss')) ``` When I try to read the data with Athena, I get an error: `HIVE_BAD_DATA: Error parsing field value '2021-10-31 22:00:00.0' for field 3: For input string: "2021-10-31 22:00:00.0"` The Data Catalog Table Schema looks like this: | Column Name | Data Type | | --- | --- | | vid | string | | altid | string | | vtype | string | | time | timestamp | | timegmt | timestamp | | value | int | | filename | string | And the line in the `run-1647806179090-part-r-00000` file it is choking on looks like this: ``` vid,altid,vtype,time,timegmt,value,filename ABC,ABC, DATAFEED,"2021-10-31 22:00:00.0","2021-11-01 02:00:00.0",11775,"lf_buf_20211101_005904.csv" ``` Does anyone know why it would throw this error? I believe according to the documentation, this is the correct timestamp format.

I received the following issue when connecting to Amazon Athena from the IDE (client desktop). However, I see a query running in the Athena console under the specified workgroup. [Simba][AthenaJDBC](100123) An error has occurred. Exception during column initialization: Details: Unable to execute HTTP request:Connect to [Athena us-east-1-amazonaws.com……..] failed: Connection timed out: connect[Execution ID:xxxxxxxx-xxxx...........]

My source db is DynamoDB. It has 26 tables. I am trying to build AWS QuickSight reports using AWS Athena that builds the specific views for said reports. however, I seem to only be able to select a single table in creating the Glue job despite being able to select all tables i need for the crawler of the entire DB from Dynamo. What is the simplest route to get a complete extract of all tables that is queryable in Athena. I dont want to connect the reports direct to dynamoDB as it s a production database and want to create some separation to avoid any performance degradation by a poor query etc. How to extract these tables?

Hello, I have multiple different CSV files that I receive via email, on a day to day basis. I would like to build a pipeline to ensure that any datasets that I receive through emails, can be picked up by Quicksight automatically and all the updates are done without much intervention. As a result I thought of the following: 1) Update all the CSVs (from email) I get each day, into S3 Buckets. 2) Point the S3 buckets to AWS Athena. 3) Quicksight will use Athena to do automatic visual updates. For this to work, I will have to ensure that the data manipulation is done outside AWS, and then manually update CSVs inside S3 buckets on my own, each day. Another alternative idea is the following: 1) Place all data into S3 Buckets. 2) Use Glue data brew to manipulate data. 3) Store it back to S3 buckets. 4) Use Quicksight to visualise data (pointing it back to S3 buckets). Can someone please advise of a better method than this ? I am looking for more automation and easier methods of manipulating data for Quicksight to update each day. My data columns don't change, but new date fields and values are added each day.

Can the SAM Lambda AthenaDynamoDBConnector (Athena DynamoDB connector) be available for CDK? Currently it can only be deployed manually, is there a way to use CDK to deploy it?

Hello, I've been able to create tables and various views in Athena console after running a Glue crawler against an S3 bucket as a datasource. When running a Preview View from Athena on a view generated from the Glue crawler, I am getting an error with some reference to a Hive split on a CSV file in our S3 bucket similar to the following: `Error opening Hive split s3://bucket_name/Folder1/folder/File.csv (offset=33554432, length=33554432) using org.apache.hadoop.mapred.TextInputFormat: Permission denied on S3 path: s3://bucket_name/Folder1/folder/File.csv This query ran against the "athena_database_name" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: e9d6e300-c82d-40d0-b9a0-879e97f72db6` It is similar to the issue outlined in : https://stackoverflow.com/questions/68859517/amazon-athena-error-opening-hive-split-s3-path-and-access-denied I do not think that this is a permission related to my IAM user but I am unsure what the Hive split is related to. Is this service related? If anyone has a solution for this, please let me know. Thank you in advance. - M

Hi, I followed this article - https://aws.amazon.com/blogs/mt/how-to-query-your-aws-resource-configuration-states-using-aws-config-and-amazon-athena/ When I check the Lambda logs - It is trying to query "ConfigHistory" too. Not sure why. Seems like it is getting triggered not only when configsnapshot is updated, but the confighistory path is updated as well. Error 1: Ignoring event for non-configuration snapshot object key AWSLogs/<Acct>/Config/us-east-1/2022/3/13/ConfigHistory/*.json.gz Error 2: However, it is adding and dropping tables, adding dt=latest as well. But, still I do see these while executing the 1st sample query in the document for - '$.instancetype') = 't3a.large' Row is not a valid JSON Object - JSONException: Duplicate key "name" This query ran against the "sampledb" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: cd87ba3c-018c-49e5-9724-fbd0cf201ab4 At a high level, AWS config is sending configsnapshots to the S3 bucket and the Lambda is triggered, but something is missing which I couldn't find it. Any help is greatly appreciated. Thank you.

AWSTemplateFormatVersion: '2010-09-09' Description: 'Creating QuickSight data source' Resources: QuickSightDataSource: Type: AWS::QuickSight::DataSource Properties: AwsAccountId: !Ref AWS::AccountId Name: Testing Data Source Type: ATHENA DataSourceId: testing-data-source DataSourceParameters: AthenaParameters: Workgroup: primary Permissions: - Actions: - quicksight:DescribeDataSource - quicksight:DescribeDataSourcePermissions - quicksight:PassDataSource Principal: !Sub - 'arn:aws:quicksight:us-east-1:${Account}:user/default/my-user-name' - Account: !Ref AWS::AccountId QSDataSet: DependsOn: QuickSightDataSource Type: AWS::QuickSight::DataSet Properties: AwsAccountId: !Ref 'AWS::AccountId' DataSetId: 'QSDataSet-test' ImportMode: SPICE Name: Dataset-test Permissions: - Actions: - 'quicksight:CreateDataSet' - 'quicksight:DeleteDataSet' - 'quicksight:DescribeDataSet' - 'quicksight:DescribeDataSetPermissions' - 'quicksight:PassDataSet' - 'quicksight:UpdateDataSet' - 'quicksight:UpdateDataSetPermissions' Principal: !Sub - 'arn:aws:quicksight:us-east-1:${AWS::AccountId}:user/default/my-user-name' - Account: !Ref 'AWS::AccountId' PhysicalTableMap: downtime_data_json: DataSourceArn: !GetAtt QuickSightDataSource.Arn InputColumns: - Name: downtime Type: INTEGER Schema: XXX??? I have data source & data tables in Athena and want to refer that in above template. I am not finding DatasourceArn, the above template throwing error as Model validation failed (#: extraneous key [DataSourceArn] is not permitted)". can anyone please help me out to hash out this.

How would column stats work with Athena partition projection on a partitioned table? I'm assuming that it doesn't. Partition projection avoids any lookup in Glue and hence stats for the partition columns are also not fetched. Or would I be wrong about this and Athena would still try and use table level (as opposed to partition level) column statistics and hence it's still worth periodically analyzing the partitioned table in order to get some table stats?

I am storing 400,000 parquet files in S3 that are partitioned based on a unique id (e.g. 412812). The files range in size from 25kb to 250kb of data. I then want to query the data using Athena. Like so, ``` Select * From Table where id in (412812, 412813, 412814) ``` This query is much slower than anticipated. I want to be able to search for any set of ids and get a fast response. I believe it is slow is because Athena must search through the entire glue catalog looking for the right file (i.e., a full scan of files). The following query is extremely fast. Less than a second. ``` Select * From Table where id = 412812 ``` partition.filtering is enabled on the table. I tried adding an index to the table that was the same as the partition, but it did not speed anything up. Is there something wrong with my approach or a table configuration that would make this process faster. Thanks!

SageMaker Feature Store is full of questions for me, ever since I started working with it. The thing which is absolutely not clear for me is about integration with Athena/Glue. Since there is no built-in option in FeatureStore to, for instance, get all records, or get records by criteria, I have to use AthenaQuery branched off the FeatureGroup for that purpose. In case if I do something like ```python feature_group = FeatureGroup(name='<...>', sagemaker_session=sagemaker_session) query = feature_group.athena_query() query.run('SELECT * FROM <whatever> LIMIT 1000', output_location='s3://...') query.wait() query.as_dataframe() ``` Will I also be charged for querying Athena? What's about Hive? If yes, then what's the purpose of using FeatureStore? It's API is too poor for implementing continuous training, - I'm instead forced to use at least Athena, and, therefore, pay for the extra service besides I already pay for FeatureStore.

This question has come up before on the [AWS developer forums](https://forums.aws.amazon.com/thread.jspa?messageID=983086&tstart=0), but without any resolution. In our case, we have a step function that does the following: * Copy a Parquet file (generated by a separate process) from a temporary location in S3 to a permanent location via a lambda. This lambda blocks until the copy is complete. * Runs an Athena query on the copied file with the `arn:aws:states:::athena:startQueryExecution` state * Invokes a lambda that waits for the query to finish. This lambda checks the query status with the `get_query_execution` API call. We're receiving an error from the `get_query_execution` in ~4% of our step function executions: ``` HIVE_FILESYSTEM_ERROR: Incorrect fileSize 150486 for file s3://<redacted>.parquet, end of stream not reached ``` Like one of the commenters on the original issue, it seems to use like the copy is still in progress when the Athena query starts even though our lambda blocks until the copy had finished. We plan to workaround this issue for now by retrying on this error. But, I would still like to know: * Does anyone have any specific insight on why this happens? * Does AWS offer any guidance on how to prevent this issue? Thanks in advance!

We are exploring usecases where we want to achieve in-place transformation and querying of S3 data lake data. We don't want to provision database and create tables (so we are not keen to consider Redshift or Athena) and we want the querying to be most cost-efficient. While we can use S3 Select to directly query S3 data, it has its own limitations such as we can query single object using S3 Select, etc. Are there any alternatives to achieve this? Please guide

Hi, could someone please give me an example in using a very simple UDF . From that simple UDF I could make it a more complex one, but I just need a very simple start in returning a substring from a database field. I have 2 problems: 1. How to define a correct UDF in my query. My query now is: USING EXTERNAL FUNCTION test(ID VARCHAR) RETURNS VARCHAR LAMBDA 'test' select test(ID) FROM "BibleLinkDynDB"."default"."bible_link_statistics" limit 10; ID is the database field. 2. How to use the input from the UDF in the lambda function. Based on the lambda template my function looks like this: index.js exports.handler = async (event) => { let response = substring(event.arguments,1,10); return response; }; I know this is not correct. I don't now how to get and process the input string because I think event is an object. If someone could please give me a hint or a simple example, then this beginner would be very grateful. kind regards Reyer

When I try to run an Athena Query with a UDF I get the following error message: "Amazon Athena experienced a permission error. Please provide proper permission and submitting the query again. If the issue reoccurs, contact AWS support for further assistance. You will not be charged for this query. We apologize for the inconvenience. This query ran against the "default" database, unless qualified by the query. " How can I find out which permission is missing?

Dears, I have a file in S3, and while querying got the following error: Need your suggestion to get this fixed. **Qry **: SELECT * FROM "de-dup-db"."input_data" limit 10; **Error **: SYNTAX_ERROR: line 1:8: Column 'name' cannot be resolved This query ran against the "de-dup-db" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: c7dde2fc-69be-4814-831f-71448486d4cb Regards, Keshava

I am attempting to use Lambda to run an Athena query in order to schedule it to run weekly for a report. When i run the query directly in Athena, I get results. When i run it as a python boto3 query, it does not work if I attempt to use 'current_date + interval **Working in Athena Query Editor:** `SELECT * FROM "datebasename"."table" where date < current_date + interval '1' day);` --- **Error when ran from Lambda ** ``` Syntax error in module 'lambda_function': invalid syntax (lambda_function.py, line 6) "stackTrace": [ " File \"/var/task/lambda_function.py\" Line 6\n query ='SELECT * FROM \""datebasename"\".\"table\" where date < (current_date + interval '1' day))'\n" ] ``` If I remove the '+ interval '1' day' from the lambda athena query , it works. I have tried with parenthesis, as well as tried using date_add, with similar results when i use an interval. Is there something I need to change to allow interval to work in boto3 athena query?

Hi, When enabling s3 access logs on a bucket, there is this great feature that the generated logs can be queried through Athena. One thing I can however not figure out, is how to partition the external table so I can limit the data scanned e.g. by date. The generated file structure is like * s3://my-access-logs/2022-02-15-11-11-18-6E829E27FAAA289A * s3://my-access-logs/2022-02-15-11-11-19-2870C85C7E0D2B20 * .. * s3://my-access-logs/2022-02-16-11-01-01-6E829E27FAAA289A * ... I added a "PARTITIONED BY (dt string)" to my table creation DDL, and then added partition like > ALTER TABLE my_logs_table ADD PARTITION (dt = '2022-02-15') location 's3://my-access-logs/2022-02-15' However, when querying the table no data is scanned, and no rows are returned (even if I don't specify a filter on dt in my query). Most examples I could find always deal with s3 structures where there is a hierarchical folder structure, like 2022/02/15/xxx Anybody has any solution for this? Thanks, Bert

I configured an Athena-DynamoDB-connector and tried a first query. In the query editor I selected a datasource and a database and then the list of tables is shown. When I run the query the result is INVALID_INPUT without any further specification of the invalid input. Can somebody help me?

According [to documentation](https://docs.aws.amazon.com/athena/latest/ug/data-types.html) Athena support binary fields. > binary – Used for data in Parquet. Is there any way to prepare insert statement to insert data in binary field using aws-sdk for NodeJS? For me is is not clear how to put binary data into [queryString](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-athena/interfaces/startqueryexecutioncommandinput.html#querystring) of `INSERT INTO` statement? Thank you in advance!

I am trying to execute in Athena a simple query on a hive table stored in s3. --- ***SELECT * FROM "database"."table" limit 10;*** --- but i get the following error. **HIVE_UNKNOWN_ERROR: Duplicate key string** These are the characteristics of the table: --- **Classification**: parquet --- **Input format:** org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat --- **Serde serialization lib:** org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe --- **EXTERNAL: **true --- **has_encrypted_data:** false --- **compressionType:** none --- **typeOfData:** file Any ideas ???

I have two catalogs in Athena - AwsDataCatalog and another named 'rds_catalog'. The 'rds_catalog' has been created to connect to RDS database instance using the Amazon Athena Federated Query connectors. I have the Simba Athena ODBC driver installed on my machine and a 'User' DSN created. I have ensured the second catalog has been setup correctly. When i connect to Athena using Power BI and the ODBC DSN, I see only the 'AwsDataCatalog' listed. The Other Catalog does not show up. This is so even when i set the schema to 'rds_catalog'. Makes me wonder if the driver is designed to show only the 'AwsDataCatalog' or if this is a power BI issue or something wrong that i am doing. Appreciate any help on this issue.

Hello, I am trying to analysis a few datasets (which are all bigger than 1GB), and they are mostly either CSV or Excel datasets. The goal is to be able to build dashboards with different datasets. I found that upon trying to ingest them into Quicksight, I got this error: > quicksight File size cannot be greater than 1 GB I would appreciate some help from the experts on methods to resolve this. I have read that some methods are to use Athena or S3 as a solution, before being able to ingest the data into Quicksight to build dashboards - but I am not sure which is the best way. Could someone please help with a tutorial to get this resolved ? Thank you,

Hi, I created tables with special characters in it. I am unable to delete those tables from my database. Please suggest me a fix. I tried number of queries but I could not delete these tables. I have used back ticks to enclose the table name but it did not work. DROP TABLE `customer_id_partitioned_{$$.time.now}` DROP TABLE `customer_id_partitioned_"{$$.time.now}"` DROP TABLE `customer_id_partitioned_"{""$""$".time.now"}"` Error: FAILED: SemanticException [Error 10255]: Invalid table name customer_id_partitioned_{$$.time.now} This query ran against the "dev_systems" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 3ae412b1-044a-4b5c-abe3-f9e39568daa6 Is there a way to forcefully delete it? Thanks, Arora

I have plenty of other Athena tables using parquet files, so the setup is okay, but this particular table has a column (metadata) that is of type `map<string, string>`. The parquet file - when loaded with pyarrow from the CLI correctly shows the table schema for that column: ``` study_id: string subject_id: string sample_id: string collection_date: date32[day] genotyping: string metadata: map<string, string> child 0, entries: struct<key: string not null, value: string> not null child 0, key: string not null child 1, value: string ``` And I load load the parquet file just in back into Pandas as well. However, when I attempt to query the table with Athena I get this: ``` HIVE_METASTORE_ERROR: Error: type expected at the position 11 of 'map<string, string>' but ' ' is found. ``` I've been fighting this for several days now trying different things to no avail. How can I properly save this parquet file with the metadata column so that Athena can read it?

I am creating a dataset in AWS Quicksight using custom SQL which I prepare/test in Athena. However, unless I define each join/table "databasename".table, the QS custom SQL fails. I have tried the below but it has failed. Is it possible to instruct the query to fun against a specific DB at the beginning of the query? ``` USING AwsDataCatalog."databasename" ```

Hi there, I am attempting to follow this tutorial: https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/ However when I run the Athena query against the S3 content, I receive the following SlowDown error: ``` Error opening Hive split s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2018-05/subset=warc/part-00223-248eba37-08f7-4a53-a4b4-d990640e4be4.c000.gz.parquet (offset=0, length=67108864): com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: QQ7HDYXF8HS44HRR; S3 Extended Request ID: yp+/JdDECD+pTbEIv3CrIcgVMgO8RZZgGN+u7Zf6v0gtACf/C3x7ANUDUq2I03gFcnhdBzlNnOs=; Proxy: null), S3 Extended Request ID: yp+/JdDECD+pTbEIv3CrIcgVMgO8RZZgGN+u7Zf6v0gtACf/C3x7ANUDUq2I03gFcnhdBzlNnOs= This query ran against the "ccindex" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 3bd4d647-091b-42c0-b60e-f689e7aa0f49 ``` Should the Athena Query Editor automatically handle such errors, and implement exponential backoffs so that I can successfully run the query?

I am querying Athena using Boto3 from python script. Most of the queries return more than 1000 records. As a result, I am using 'NextToken' in 'get_query_results' for fetching subsequent records. I observed each query takes significant long time (in minutes) to complete ( fetching 1171 records took more than a minute). Surprisingly, the data size is small - in KB. I am not sure if this behavior is a result of wrong usage on my part OR are there any underlying technical reasons. Any specific areas that I should look at? Appreciate any help with this.

I have Delta Lake tables (using Symlink text input format) catalogued in Glue, stored in a S3 bucket, with all its resources tagged with LakeFormation Tags (for tag-based governance). The problem is that, although the users can see the database, tables, and metadata within Athena's catalogue, they cannot perform queries against the specific tables because of "Permission denied on S3 path" errors. LakeFormation has the data location registered for the datalake bucket, with AWSServiceRoleForLakeFormationDataAccess role. And this role has IAM permissions automatically added to the resources: ``` LakeFormationDataAccessServiceRolePolicy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets" ], "Resource": [ "arn:aws:s3:::*" ] } ] } ``` and ``` LakeFormationDataAccessPolicyForS3 { "Version": "2012-10-17", "Statement": [ { "Sid": "LakeFormationDataAccessPermissionsForS3", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::MYBUCKET/*" ] }, { "Sid": "LakeFormationDataAccessPermissionsForS3ListBucket", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::MYBUCKET" ] } ] } ``` I have also tried registering the data location with a role with Admin permissions (Action "*" and Resources "*"), but even so the same error is thrown. Looking through the CloudTrail logs, I found that LakeFormation passes custom policies to the role when running AssumeRole: ``` "policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Action\": [\"s3:GetObject\"],\n \"Effect\": \"Allow\",\n \"Resource\": [\"arn:aws:s3:::MYBUCKET\",\n \"Condition\": {\"ForAnyValue:StringLike\":{\"s3:prefix\":[\"MYTABLE/_symlink_format_manifest\",\"MYTABLE/_symlink_format_manifest/*\"]}}\n },\n {\n \"Action\": [\"kms:Decrypt\"],\n \"Effect\": \"Allow\",\n \"Resource\": [\"*\"],\n \"Condition\": {\"StringEquals\":{\"kms:ViaService\":[\"s3.us-east-2.amazonaws.com\"]}}\n } ]\n}" } ``` This seems like a malformatted json string that is being passed to the assumed role. Can this be causing the errors I'm having? And does anyone have had this issue before? PS: I have manually removed ACL control over the S3 bucket and objects. Still same behavior. The error is not shown if I remove the data location, and Athena ignores Lake Formation.

Any help to fix this error will be appreciated.. Thanks your your time.. This was the error seen: `GENERIC_INTERNAL_ERROR : java.net.URISyntaxException : Relative path in absolute URI: arn:aws:dynamodb:us-east-1:xyz:table/xyz-ServiceConf This query ran against the "" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 555c323d-39c8-44ba-821a-8f507b446233` This was the query: `select * from "AwsDataCatalog"."ddb_glue_db"."xyz_serviceconf"` This is the definition of the table in AWS Glue data catalog: ``` Name xxx_serviceconf Description Database ddb_glue_db Classification dynamodb Location arn:aws:dynamodb:us-east-1:xyz:table/xyz-ServiceConf Connection Deprecated No Last updated Wed Feb 02 13:27:24 GMT+530 2022 Serde parameters - Table properties sizeKey 606 hashKey configVersion UPDATED_BY_CRAWLER ddb_crawl_xxxx-xxx-server CrawlerSchemaSerializerVersion 1.0 recordCount 1 averageRecordSize 606 CrawlerSchemaDeserializerVersion 1.0 compressionType none typeOfData table ```

After I create a query in Athena using the AWS console, how can I find the Query ID associated with that query? If I run the query and it fails, I can see the Query ID in the AWS console by going to "Recent Queries" and clicking the "Failed" link next to the query run (https://docs.aws.amazon.com/athena/latest/ug/querying.html), but this doesn't work for queries where the status is "completed". I can also use the API to first call ListNamedQueries, then iterate through that resulting list of query ids and call GetNamedQuery on each one to see if the name matches the query I'm looking for. But that's a really roundabout process just to get the query id for a query I just created. Is there another way to simply find a named query's id via the console that I'm not seeing?

I have configured a data lake using Lake Formation and Glue to populate and upsert the database and tables. This upsert includes assigning LF Tags to each table and column (for LF Tag-based governance). I'm trying to query the table using Athena, with an administrator level user account, plus Lake formation permissions on LF Tags, datalake administrator, database creator, and permissions to the s3 data location resource. However, when I query a ``` select count(*) from table ``` on Athena, I'm thrown this error: ``` Insufficient Lake Formation permission(s): Illegal permission combination (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 6280c520-8891-4461-8a7e-2e6bbb5a2fec; Proxy: null) This query ran against the "raw" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 2f3f0e09-0cda-4e07-a390-80689a66c022 ``` I've tried adding all possible permissions to access the data, but still can't run a query using Athena. Can someone spot what I'm missing out? Thanks! Denis

I'm stroing cloudtrail logs into the S3 and through glue crwaler trying to crawl the S3(cloudtrail logs) and creating a glue table. While I'm running query against that table in Athena , getting below error HIVE_UNKNOWN_ERROR: serDe should not be accessed from a null StorageFormat Can someone please help in fixing the issue or tell the workaround. I'm creating all the resources through terraform.

I am troubleshooting a long query with many joins, and getting the error in the title. I've found a part of the query that appears to trigger the error; it's a query that is joined to the rest of tables and subqueries with a CROSS JOIN, as the result is a single row of average values to be attached to every row in the full result set. The first of the averages makes use of to_unixtime(from_iso8601_date(date)) to calculate an average date. This average uses a subquery that explicitly excludes null and empty string values for the calculation of the average. This part of the query runs fine on its own. It's when that result is combined with other averages calculated from columns in the same source table that the error occurs. Originally, this first average and the others were all part of a single SELECT statement inside the CROSS JOIN; I've also tried separating that first average into its own CROSS JOIN, and keeping the rest of the averages (which also run fine on their own) in another CROSS JOIN. Same error though. This full query formerly ran without error; it's only in the last week or two that this has happened.

Getting this error: `Error opening Hive split s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2017-17/subset=crawldiagnostics/part-00189-ac1cf8ef-3644-4b49-ac73-0fe6bef46adf.c000.gz.parquet (offset=0, length=36266581): com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: J88VWPD7BXAFP01E; S3 Extended Request ID: 89ZPRgZ/qx3n0Gs4zFonvfA50JUPjf9ep5vHxhCKHIFXwVr70vgbnLSL9Ctx22GNikrR+p/3gQU=; Proxy: null), S3 Extended Request ID: 89ZPRgZ/qx3n0Gs4zFonvfA50JUPjf9ep5vHxhCKHIFXwVr70vgbnLSL9Ctx22GNikrR+p/3gQU= This query ran against the "ccindex" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 066d4ff2-89ce-4683-a4c0-71e2f45348cb`

amazon athena takes long time to run my query I want all s3 object lists in csv format

My query is showing time out error in athena I want list of s3 objects in csv format from my bucket

I am using Athena to connect to my DocumentDB database with the intention of visualising the data in QuickSight. I can see the connection is correct as the database name and table name load properly. When I go to create an analysis or even preview the data it timesout. A refresh just causes the same result. The table only contains 40 or so documents so its not an issue of the amount of data. I have tried with direct query and SPICE. This is the error returned when attempting to reconnect from QuickSight: ``` Error Details Error type: SQL_EXCEPTION Learn more Error details: [Simba][AthenaJDBC](100071) An error has been thrown from the AWS Athena client. GENERIC_USER_ERROR: Encountered an exception[com.amazonaws.SdkClientException] from your LambdaFunction[arn:aws:lambda:eu-west-1:717816899369:function:documentdb-connector] executed in context[retrieving meta-data] with message[Unable to execute HTTP request: Connect to s3.eu-west-1.amazonaws.com:443 [s3.eu-west-1.amazonaws.com/52.218.57.91] failed: connect timed out] [Execution ID: 56486351-0cba-4e81-9d9f-3ff4722c198f] Ingestion Id: 061d9b7e-c7f3-49a7-88ba-531551f6cc9a ``` The SQL_EXCEPTION errors states that it can be caused by a timeout so I figure thats the issue. `SQL_EXCEPTION – A general SQL error occurred. This error can be caused by query timeouts, resource constraints, unexpected data definition language (DDL) changes before or during a query, and other database errors. Check your database settings and your query, and try again.` Could there be an issue with the Lambda function being used by Athena to connect to the DocumentDB? Looking at the logs the connection seems to be working. I thought QuickSight would just be able to pull in the data now that everything is linked up. Any advice is appreciated.

 I want to use this CREATE VIEW option in the Athena console to run a query which will create a View in the Glue Database. But I want to use CDK. My question is : What’s the easiest way to create an Athena View using monocdk, which will create a view in the Glue DataBase? Seeing that creating CfnNamedQuery construct with the 'CREATE OR REPLACE VIEW' query will just create a Saved Query. So I’ll have to run it once via the console to actually create the view, which defeats the purpose. I’m fairly new to Athena and CDK, so would be great help if anyone can point me towards the right direction. Thanks! (PS : Markdown for the embedding the picture doesn't seem to be working. The picture is a screenshot of the Athena console : https://tiny.amazon.com/1dzv084cv/drivcorpamazviewananpublScre)

I'm attempting to run an Athena query using AWS Data Wrangler which is failing with an error indicating a column cannot be resolved. However when I view the failing query in the Athena console and rerun it there, it succeeds. I have a feeling it has something to do with the formatting with table or column names but I've been unsuccessful finding a work around. When I attempt the same query using vanilla boto3, it also fails with the same error. Here is an example of the code which fails followed by the same successful query ran in the console. (with some names anonymized for security) Adding to the strangeness further, if I remove the joins, it will error indicating the first column in the select cannot be resolved but the same query runs successfully in the console. ### Python Code ``` import awswrangler as wr query = """ SELECT t1.description, t1.model_number as model, t1.p_id as p_id, t1.created_at as createdAt, t1.updated_at as updatedAt, t3.description as platform, t2.name as module FROM database_with_underscores.first_table t1 LEFT JOIN database_with_underscores.other_table t2 ON t2.id = t1.module_id LEFT JOIN database_with_underscores.lasttable t3 ON t1.platform_id = t3.id """ df = wr.athena.read_sql_query(query, workgroup="primary", database="database_with_underscores") ``` ### Python Result `awswrangler.exceptions.QueryFailed: SYNTAX_ERROR: line 19:5: Column 't2.id' cannot be resolved` ### AWS Athena Console ``` SELECT t1.description, t1.model_number as model, t1.p_id as p_id, t1.created_at as createdAt, t1.updated_at as updatedAt, t3.description as platform, t2.name as module FROM database_with_underscores.first_table t1 LEFT JOIN database_with_underscores.other_table t2 ON t2.id = t1.module_id LEFT JOIN database_with_underscores.lasttable t3 ON t1.platform_id = t3.id ``` ### Console Result Successful Link to a diff of the execution where you can see the Query is identical but one succeeds and the other fails https://imgur.com/a/V9QSbIn

I'm trying to load a JSON file containing a geometry in the "EnclosedJsonInputFormat" following a pattern demonstrated in the Athena geospatial query [sample](https://docs.aws.amazon.com/athena/latest/ug/geospatial-example-queries.html). The table is created but contains no data and I'm assuming there is some sort of parsing error. Is there any way to get more information on the problem?

I got this error after loading a parquet file from S3 into Athena. HIVE_BAD_DATA: Field metadata_file's type BINARY in parquet file s3://dil-google-books/data/dataset.parquet/part-00108-33efb610-5e72-4e37-b646-3b4ab4578bad-c000.snappy.parquet is incompatible with type boolean defined in table schema Could you please suggest what I need to do? ``` CREATE EXTERNAL TABLE IF NOT EXISTS all_dataset ( AccessDate string, Authors string, Chapter string, Chron string, City string, Degree string, Edition string, Encyclopedia string, Format string, ID_list string, Issue string, Pages string, Periodical string, PublicationPlace string, PublisherName string, Series string, SeriesNumber string, Title string, TitleType string, URL string, Volume string, citations string, id int, page_title string, r_id int, r_parentid int, sections string, type_of_citation string, updated_identifier array <string> , conf_score array <double>) STORED AS PARQUET LOCATION 's3:/' tblproperties ("parquet.compress"="SNAPPY"); ```

In the Athena [documentation](https://aws.amazon.com/athena/pricing/), it suggests that there are no charges for CREATE TABLE commands. However, if a user runs a CTAS query to create a new table from an original table, Athena runs a SELECT statement and scans the data in the original table in order to generate the new table. So I have two question based on this observation: 1) Does as CREATE TABLE AS SELECT command generate costs the same way as any other query would would? I can't see this mentioned in the docs, but looking at the Console a large amount of data is scanned when creating a table with CTAS. 2) Let's imagine a new CTAS table is created from an original Athena table that gets its data from text files in an s3 bucket. Once the new table is created using CTAS, in order for it to remain in sync with new files (in s3), does it intermittently run a query against the original table, thus generating a regular new cost? This seems like things could get expensive just to keep the CTAS table up to date?

I've glue catalog and use s3 as DB. Datatype of columns are numeric in Redshift which i'm casting as decimal in Glue. But getting issue while querying the data in Athena. Column in redshift looks like (-12.887686) with data type Number. we cannot change the data type in source table

I've created an Athena table to analyse ALB logs stored in s3 using the recommended create table command in the [AWS docs](https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html). I'm running some queries on this table to retrieve data from specific columns. However, when I look at the data analysed, it appears the data used in the query amount the the total data in the logs files. These log files contain many columns that I am not referencing in my query. I thought if I am only retrieving data from specific columns, that it would only search the data from those columns, and therefore the data used in the query would be less? Is there something I'm missing here? Query: `SELECT FROM_UNIXTIME(FLOOR(TO_UNIXTIME(parse_datetime(time,'yyyy-MM-dd''T''HH:mm:ss.SSSSSS''Z'))/60)*60) as time, ssl_protocol, count(ssl_protocol) as total FROM alb_logs WHERE ssl_protocol LIKE '%' AND request_verb = 'POST' AND REGEXP_LIKE(request_url, 'https:\/\/myapp.*\/v3\/me\/order.*') AND elb_status_code = 200 AND parse_datetime(time,'yyyy-MM-dd''T''HH:mm:ss.SSSSSS''Z') BETWEEN TIMESTAMP '2021-12-06 12:12:44' AND TIMESTAMP '2022-01-05 12:12:44' GROUP BY 1, ssl_protocol ORDER BY 1 limit 20`

AWS Region: ap-southeast-1 Currently I am using the AWS Kinesis Firehose Delivery Stream to Stream the Cloudwatch Logs to S3 with the Prefix appending Timestamp, like this /year=!{timestamp:YYYY}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/ However, found that after 25 Dec 2021, the Log File to S3 was wrongly put the Object to e.g. /year=2022/month=12/day=26/hour=00 now. Anyone has the same issue now, looks critical issue for us as we use this timestamp for Partitioning on AWS Athena

Trying to query a json file but getting an error HIVE_METASTORE_ERROR: Error: : expected at the position 63 of 'array<struct<picture_link_text:string,title_text:string,Created Date:string,Created By:string,Modified Date:string,_id:string>>' but ' ' is found. Just getting started with Athena so have no idea what I am doing wrong. Any ideas? This table was imported by Glue.

I have log files in my onpremise application, Individual file size is 100 MB (file rollover/appender defined in Log4J in onpremise application) and 50 such files get generated on daily basis which is 5 GB total. We then zip these files at day end and push to S3 bucket manually today. 1. We dont want to use cloudwatch agent in onpremise application to push logs to CloudWatch (as onpremise applications are not having CPU/Memory and running at peak) 2. We zip the files at day end else we have to do 50-55 S3 Copy manually each day (yes we can create some script for this - but again not elegant) 3. Yes Elasticsearch is an option but we are building solution which will take 2-3 months to ingest data to Elasticsearch and use ELK stack Now till 1 year, whenever we get customer complaint we have to extract specific file. We know the date, but unfortunately we have to download all 5 GB for that day and extract the required file and content. As part of this usecase wanted to check: 1. If Athena works for Log4J log files (from JBoss, Websphere) - do we have Serde's (serialization / deserialization libraries) for the same, glue catalog for the same

I created a table with partition projection This is the setup: ``` PARTITIONED BY ( `userid` string, `taskid` string, `accountid` string, `month` int, `year` int) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 's3://bucketName/' TBLPROPERTIES ( 'has_encrypted_data'='false', 'projection.accountid.type'='inject', 'projection.enabled'='true', 'projection.month.digits'='2', 'projection.month.range'='1,12', 'projection.month.type'='integer', 'projection.taskid.type'='inject', 'projection.userid.type'='inject', 'projection.year.range'='2018,2051', 'projection.year.type'='integer', 'storage.location.template'='s3://bucketName/userid=${userid}/taskid=${taskid}/accountid=${accountid}/month=${month}-${year}/', ``` activae query ``` where userid='626b4b4e-9518-46cd-a7af-0ebda24425bd' and taskid='6d45c163-d5f5-4a44-9aa5-f80e8f8152e4' and accountid = 'blabla-blabla@abc.com' and "month" = 12 and "year" = 2020 ``` got **GENERIC_INTERNAL_ERROR: No value present** query id - Query Id: de3b869c-6d1a-4b10-989f-be79356569a1 Any idea why? Thanks

I am trying to create a Glue/Athena table using CloudFormation or Terraform for analyzing ALB logs. I can easily do it using AWS Console following this guide https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html However I can't find any recipe how to do it using CloudFormation or Terraform. I was trying to follow this example ( https://stackoverflow.com/a/56347331 ) but it just creates a Presto view, not using any regex to parse anything in S3 bucket. How can I achieve it? Could anyone please point some example or documentation?

Athena and redshift are incorrectly grouped under analytics. quicksight is the correct tool in that category. But redshift should be under database. https://i.stack.imgur.com/8O65F.png

Hello - I am accessing tables in an RDS Aurora Postgres by using an Athena Federated Query. The Lambda can access the postgres because it is reading the tables into the catalog in Athena, but when I try to query the data I get a timeout error. ``` GENERIC_USER_ERROR: Encountered an exception[com.amazonaws.SdkClientException] from your LambdaFunction[arn:aws:lambda:MyRegion:MyAccount:function:MyFunction] executed in context[retrieving meta-data] with message[Unable to execute HTTP request: Connect to s3.amazonaws.com:443 [s3.amazonaws.com/54.231.192.16] failed: connect timed out] ``` This seems to be related to the s3 permissions, but I can't suss out where the mismatch is. Do I need a bucket policy? As a principal, I have access to all the s3 resources needed, permissions on Athena, and Lambda also. The Lambda seems to have the correct s3 permissions (read/write for the bucket and all objects in the bucket). Any insight would be helpful! Thanks!

How much will it cost if I complete all the steps mentioned in this article? will I be charged for Athena as well as glue? https://aws.amazon.com/blogs/big-data/improve-amazon-athena-query-performance-using-aws-glue-data-catalog-partition-indexes/