Questions tagged with Data Lakes
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
I've got a fairly simple ETL job that reads several catalog tables or views and does some joins. the job errors out with the following error:
```
Error Category: UNCLASSIFIED_ERROR; An error...
0
answers
0
votes
166
views
asked 4 days agolg...
Is it possible to integrate opentelemetry(ADOT) and promethus managed by AWS with a data lake. The data lake could be both on edge or on AWS? If yes then how does it work? How can oepntemetry and...
1
answers
0
votes
91
views
asked 8 days agolg...
At every 30 minutes it saying "ICEBERG_VACUUM_MORE_RUNS_NEEDED: Removed 20000 files in this round of vacuum" but when I calculate the my table metadata size it didn't changed before and after the...
2
answers
0
votes
451
views
asked 3 months agolg...
Hi! I have been searching and playing around with services and cannot seem to find what I need.
I am using the following architecture to guide me in building out my end-to-end solution:...
2
answers
0
votes
323
views
asked 7 months agolg...
When executing a task the last step is validating the data migrated with the source against target apparently using Athena, I have the following error:
2023-11-07T22:09:04 [VALIDATOR_TARGE ]E: Not...
1
answers
1
votes
769
views
asked 7 months agolg...
Hello
I have created a resource link to a shared database from a different account. I am able to query the tables within the database but "Show tables from <database>" and "View Tables" on the AWS...
1
answers
0
votes
477
views
asked 8 months agolg...
My Glue 4.0 jobs have suddenly stopped working with error message below. As it is related to boto3, I am unable to make any changes to library config. Pls advise.
NB: I noticed that urllib3 released...
0
answers
0
votes
259
views
asked 9 months agolg...
Hello everyone,
### 1. Context
We have a Delta Lake where we write our tables in **S3** in **Delta format**, and we use the **Glue Catalog** for queries in **Athena**. Tables are created both with...
1
answers
1
votes
693
views
asked 9 months agolg...
I am trying to create a Delta Table from spark sql using the Glue meta catalog.
I can correctly query a Delta table using the Glue metastore:
```
%%sql
select * from `my_table` VERSION AS OF 1 limit...
2
answers
0
votes
1676
views
asked a year agolg...
Looks like attempting to write to a Delta Lake table from a DynamicFrame is not working. The Visual Glue interface generates a script like:
```
s3 =...
2
answers
0
votes
600
views
asked a year agolg...
Hello team,
We are planning to build a data lake in AWS that will contain regularly extracted data from an on-prem data warehouse. The purpose of this data lake is to serve the following purposes in...
2
answers
0
votes
530
views
asked a year agolg...
This CDK code produces a "Resource did not stabilize" error:
```
data_location = lakeformation.CfnPrincipalPermissions.DataLocationResourceProperty(
catalog_id=Aws.ACCOUNT_ID,
...
1
answers
0
votes
440
views
asked a year agolg...