An error occurred while calling o187.pyWriteDynamicFrame. Unable to execute HTTP request:


Dear community

This is happening to me when I´m running an ETL job trying to transform an Oracle database table to parquet format and store it in a S3 bucket. I´ve been checking permissions and everything that I could imagine but I´m not able to figure out where does the error comes from.

I highly apreciate if anyone can give me some lights about it.


asked 2 years ago346 views
1 Answer

Hello, The error message you're encountering "Unable to execute HTTP request" typically indicates a network connection issue. This could be caused by many factors, like network connectivity problems between the Glue job and the S3 service or the Oracle database, permissions on the network level. Without more details, please find below a wire array of troubleshooting steps that go in depth.

Check Network Connectivity: Make sure that your AWS Glue job and the Oracle database are able to communicate properly. If they are hosted in different networks (VPCs) or even different regions, you might need to set up appropriate network routes, security groups, and NAT gateway or VPC endpoints to allow traffic between them.

Check S3 Connectivity: Similarly, make sure that the AWS Glue job can reach the S3 service. If you're running the Glue job in a VPC, you might need to set up a VPC endpoint for S3.

Verify Permissions: Validate the IAM permissions for the Glue job and make sure it has necessary permissions to read from Oracle database and write to the S3 bucket. Also, make sure the Oracle database user used in the Glue connection has the necessary privileges.

Try a Smaller Dataset: Sometimes, the issue could be related to the size of the data you're trying to process. Try to run the job on a smaller dataset and see if you still face the same issue.

AWS Glue Version: Make sure that you're using a compatible version of AWS Glue for your use case.

Check Service Health: Lastly, it's worth checking the AWS Service Health Dashboard to see if there were any outages or disruptions in the AWS Glue or S3 service in your region at the time the job was executed.

If the following connectivity steps are in order, if you are using AWS spark for table conversions based on the .py shown above. Please attempt the following troubleshooting steps.

AWS Glue Connectivity: Ensure that your Glue job has the necessary network access to your Oracle database as well as S3.

Oracle JDBC Driver: If you are using JDBC to connect to your Oracle database, ensure that you're using the correct JDBC driver for your Oracle database version.

S3 Bucket Permissions: Check the permissions for the S3 bucket where you're trying to store the data. Make sure that the IAM role associated with your Glue job has the necessary permissions to write to the S3 bucket.

Check AWS Glue Job Log: Inspect the AWS Glue job logs in CloudWatch Logs for more detailed error messages. These logs could provide more context about the underlying problem.

thank you


answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions