An error occurred while calling o95.getDynamicFrame. key not found: location

0

In a AWS Glue job, I am getting issue when I am trying to execute sql query in below code:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
{"useConnectionProperties": "true", 
"connectionName": "d-glue-connection", 
"query": "select * from myschema.table1 where 1=1"
},
transformation_ctx = "source_dataset"
 )

Error Message:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/dynamicframe.py", line 609, in from_options
    return self._glue_context.create_dynamic_frame_from_options(connection_type,
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/context.py", line 237, in create_dynamic_frame_from_options
    return source.getFrame(**kwargs)
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/data_source.py", line 37, in getFrame
    jframe = self._jsource.getDynamicFrame()
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o95.getDynamicFrame.
: java.util.NoSuchElementException: key not found: location
	at scala.collection.MapLike.default(MapLike.scala:236)
	at scala.collection.MapLike.default$(MapLike.scala:235)
	at scala.collection.AbstractMap.default(Map.scala:65)
	at scala.collection.MapLike.apply(MapLike.scala:144)
	at scala.collection.MapLike.apply$(MapLike.scala:143)
	at scala.collection.AbstractMap.apply(Map.scala:65)
	at com.amazonaws.services.glue.util.JDBCWrapper.$anonfun$qualifiedTableName$2(JDBCUtils.scala:923)
	at scala.Option.getOrElse(Option.scala:189)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName$lzycompute(JDBCUtils.scala:923)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName(JDBCUtils.scala:921)
	at com.amazonaws.services.glue.util.JDBCWrapper.dbtable(JDBCUtils.scala:934)
	at com.amazonaws.services.glue.util.JDBCWrapper.sqlTableName(JDBCUtils.scala:927)
	at com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:944)
	at com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)
	at com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)
	at com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1027)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:102)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:102)
	at com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:726)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:750)

Issue is with query option in the code, In AWS documentation I have read that we can use query option as well in place of dbTable.

What is the issue and its correct solution?

已提問 10 個月前檢視次數 2119 次
2 個答案
0

Hi, your query 'select * from myschema.table1 where 1=1' is very peculiar! What is the purpose of this 1=1 ? You should retry your code with only ''select * from myschema.table1'.

BTW, your NoSuchElementException usually means that Java JDBC driver tries to iterate over the result set of the query but doesn't find any corresponding item.

So,

  1. does table myschema.table1 exist in your database ?
  2. does it contain any row at all?

Best, Didier

profile pictureAWS
專家
已回答 10 個月前
  • Yes, myschema.table1 exist in my database and select * from myschema.table1 returns 1840 rows when it is executed outside the Glue code.

0

"query" is not part of the API, "sampleQuery" is, see: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-jdbc Using DataFrame you can use "query" instead of dbTable.

profile pictureAWS
專家
已回答 10 個月前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南