Questions tagged with Amazon EMR
Content language: English
Select up to 5 tags to filter
Sort by most recent
Browse through the questions and answers listed below or filter and sort to narrow down your results.
I am using emr-6.12.0 and trying to set environment varibles which are stored in secret manager in bootstrap.sh file.
```
SECRET_NAME="/myapp/dev/secrets"
SECRETS_JSON=$(aws secretsmanager...
1
answers
0
votes
331
views
asked 6 months agolg...
I want my EMR cluster to be terminated automatically post an idle time.
I have configured 'Automatically terminate cluster after idle time' and set the idle time as '5 minutes' .
In my cluster i have...
1
answers
0
votes
246
views
asked 6 months agolg...
If my environment is full of Apache Hudi integrating with EMR and Lake Formation, I found out that Hudi environment is not very friendly to be used by Redshift nor Athena. There are many advanced...
1
answers
0
votes
466
views
asked 6 months agolg...
My customer is using AWS EMR and is storing all the Hive meta data on an external RDS instance, using MySQL 5.7.* And since MySQL 5.7 is running out of its lifecycle, we are pushing them to upgrade...
1
answers
1
votes
296
views
asked 6 months agolg...
Everyday a new emr cluster span up and terminated after completing the step job. Checking the cloudtrail, seems a Data Pipeline created it. I am not sure how to get more details like who created, what...
2
answers
1
votes
269
views
asked 6 months agolg...
I want to save my pyspark dataframe in RecordIO protobuf format. I am using Amazon EMR to run my pyspark scripts, and I want to use AWS SageMaker to train a machine learning model. SageMaker pipe mode...
1
answers
0
votes
209
views
asked 6 months agolg...
Hello,
I have a driver and executor pod template files exist in s3. But when I run the job, it failed wirh FileNotFoundException: s3:/<s3bucket>/podtemplate/driver.yaml (no such file or directory)...
1
answers
2
votes
257
views
asked 6 months agolg...
Hello,
Please share the difference between AWS Glue and AWS EMR and which one we should use and when?
Thanks,
1
answers
0
votes
1490
views
asked 6 months agolg...
In continuation to the ticket - https://repost.aws/questions/QUfxjbaGrXRTSKGy4-rnQ8Uw/how-to-upgrade-python-version-in-emr-since-python-3-7-support-discontinued , the latest version of EMR 6.14.0...
1
answers
0
votes
303
views
asked 7 months agolg...
I am facing issues using inline clustering and compaction in EMR, with the following error..
EMR : 6.13.0
Hudi: 0.13.1
com.esotericsoftware.kryo.KryoException: Unable to find class:...
0
answers
0
votes
104
views
asked 7 months agolg...
We have some parquet format files on AWS s3 and we want to create iceberg table with these files. Can Glue Crawler do this?
2
answers
0
votes
680
views
asked 7 months agolg...
I am using EMR 6.13.0(EMR on EC2) and it is present in eu-central-1 region. i have my s3 buckets in eu-central-1 and eu-west-1 region, i can be able to access the s3 buckets present in eu-central-1...
1
answers
1
votes
634
views
asked 7 months agolg...