An error occurred while calling o95.getDynamicFrame. key not found: location

0

In a AWS Glue job, I am getting issue when I am trying to execute sql query in below code:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
{"useConnectionProperties": "true", 
"connectionName": "d-glue-connection", 
"query": "select * from myschema.table1 where 1=1"
},
transformation_ctx = "source_dataset"
 )

Error Message:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/dynamicframe.py", line 609, in from_options
    return self._glue_context.create_dynamic_frame_from_options(connection_type,
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/context.py", line 237, in create_dynamic_frame_from_options
    return source.getFrame(**kwargs)
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/data_source.py", line 37, in getFrame
    jframe = self._jsource.getDynamicFrame()
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o95.getDynamicFrame.
: java.util.NoSuchElementException: key not found: location
	at scala.collection.MapLike.default(MapLike.scala:236)
	at scala.collection.MapLike.default$(MapLike.scala:235)
	at scala.collection.AbstractMap.default(Map.scala:65)
	at scala.collection.MapLike.apply(MapLike.scala:144)
	at scala.collection.MapLike.apply$(MapLike.scala:143)
	at scala.collection.AbstractMap.apply(Map.scala:65)
	at com.amazonaws.services.glue.util.JDBCWrapper.$anonfun$qualifiedTableName$2(JDBCUtils.scala:923)
	at scala.Option.getOrElse(Option.scala:189)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName$lzycompute(JDBCUtils.scala:923)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName(JDBCUtils.scala:921)
	at com.amazonaws.services.glue.util.JDBCWrapper.dbtable(JDBCUtils.scala:934)
	at com.amazonaws.services.glue.util.JDBCWrapper.sqlTableName(JDBCUtils.scala:927)
	at com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:944)
	at com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)
	at com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)
	at com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1027)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:102)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:102)
	at com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:726)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:750)

Issue is with query option in the code, In AWS documentation I have read that we can use query option as well in place of dbTable.

What is the issue and its correct solution?

asked 9 months ago1983 views
2 Answers
0

Hi, your query 'select * from myschema.table1 where 1=1' is very peculiar! What is the purpose of this 1=1 ? You should retry your code with only ''select * from myschema.table1'.

BTW, your NoSuchElementException usually means that Java JDBC driver tries to iterate over the result set of the query but doesn't find any corresponding item.

So,

  1. does table myschema.table1 exist in your database ?
  2. does it contain any row at all?

Best, Didier

profile pictureAWS
EXPERT
answered 9 months ago
  • Yes, myschema.table1 exist in my database and select * from myschema.table1 returns 1840 rows when it is executed outside the Glue code.

0

"query" is not part of the API, "sampleQuery" is, see: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-jdbc Using DataFrame you can use "query" instead of dbTable.

profile pictureAWS
EXPERT
answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions