In a AWS Glue job, I am getting issue when I am trying to execute sql query in below code:
source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
{"useConnectionProperties": "true",
"connectionName": "d-glue-connection",
"query": "select * from myschema.table1 where 1=1"
},
transformation_ctx = "source_dataset"
)
Error Message:
source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
File "/opt/amazon/lib/python3.7/site-packages/awsglue/dynamicframe.py", line 609, in from_options
return self._glue_context.create_dynamic_frame_from_options(connection_type,
File "/opt/amazon/lib/python3.7/site-packages/awsglue/context.py", line 237, in create_dynamic_frame_from_options
return source.getFrame(**kwargs)
File "/opt/amazon/lib/python3.7/site-packages/awsglue/data_source.py", line 37, in getFrame
jframe = self._jsource.getDynamicFrame()
File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
return_value = get_return_value(
File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
return f(*a, **kw)
File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o95.getDynamicFrame.
: java.util.NoSuchElementException: key not found: location
at scala.collection.MapLike.default(MapLike.scala:236)
at scala.collection.MapLike.default$(MapLike.scala:235)
at scala.collection.AbstractMap.default(Map.scala:65)
at scala.collection.MapLike.apply(MapLike.scala:144)
at scala.collection.MapLike.apply$(MapLike.scala:143)
at scala.collection.AbstractMap.apply(Map.scala:65)
at com.amazonaws.services.glue.util.JDBCWrapper.$anonfun$qualifiedTableName$2(JDBCUtils.scala:923)
at scala.Option.getOrElse(Option.scala:189)
at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName$lzycompute(JDBCUtils.scala:923)
at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName(JDBCUtils.scala:921)
at com.amazonaws.services.glue.util.JDBCWrapper.dbtable(JDBCUtils.scala:934)
at com.amazonaws.services.glue.util.JDBCWrapper.sqlTableName(JDBCUtils.scala:927)
at com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:944)
at com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)
at com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)
at com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1027)
at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:102)
at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:102)
at com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:726)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:750)
Issue is with query option in the code, In AWS documentation I have read that we can use query
option as well in place of dbTable
.
What is the issue and its correct solution?
Yes, myschema.table1 exist in my database and select * from myschema.table1 returns 1840 rows when it is executed outside the Glue code.