An error occurred while calling o95.getDynamicFrame. key not found: location

0

In a AWS Glue job, I am getting issue when I am trying to execute sql query in below code:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
{"useConnectionProperties": "true", 
"connectionName": "d-glue-connection", 
"query": "select * from myschema.table1 where 1=1"
},
transformation_ctx = "source_dataset"
 )

Error Message:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/dynamicframe.py", line 609, in from_options
    return self._glue_context.create_dynamic_frame_from_options(connection_type,
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/context.py", line 237, in create_dynamic_frame_from_options
    return source.getFrame(**kwargs)
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/data_source.py", line 37, in getFrame
    jframe = self._jsource.getDynamicFrame()
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o95.getDynamicFrame.
: java.util.NoSuchElementException: key not found: location
	at scala.collection.MapLike.default(MapLike.scala:236)
	at scala.collection.MapLike.default$(MapLike.scala:235)
	at scala.collection.AbstractMap.default(Map.scala:65)
	at scala.collection.MapLike.apply(MapLike.scala:144)
	at scala.collection.MapLike.apply$(MapLike.scala:143)
	at scala.collection.AbstractMap.apply(Map.scala:65)
	at com.amazonaws.services.glue.util.JDBCWrapper.$anonfun$qualifiedTableName$2(JDBCUtils.scala:923)
	at scala.Option.getOrElse(Option.scala:189)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName$lzycompute(JDBCUtils.scala:923)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName(JDBCUtils.scala:921)
	at com.amazonaws.services.glue.util.JDBCWrapper.dbtable(JDBCUtils.scala:934)
	at com.amazonaws.services.glue.util.JDBCWrapper.sqlTableName(JDBCUtils.scala:927)
	at com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:944)
	at com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)
	at com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)
	at com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1027)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:102)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:102)
	at com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:726)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:750)

Issue is with query option in the code, In AWS documentation I have read that we can use query option as well in place of dbTable.

What is the issue and its correct solution?

gefragt vor 10 Monaten2113 Aufrufe
2 Antworten
0

Hi, your query 'select * from myschema.table1 where 1=1' is very peculiar! What is the purpose of this 1=1 ? You should retry your code with only ''select * from myschema.table1'.

BTW, your NoSuchElementException usually means that Java JDBC driver tries to iterate over the result set of the query but doesn't find any corresponding item.

So,

  1. does table myschema.table1 exist in your database ?
  2. does it contain any row at all?

Best, Didier

profile pictureAWS
EXPERTE
beantwortet vor 10 Monaten
  • Yes, myschema.table1 exist in my database and select * from myschema.table1 returns 1840 rows when it is executed outside the Glue code.

0

"query" is not part of the API, "sampleQuery" is, see: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-jdbc Using DataFrame you can use "query" instead of dbTable.

profile pictureAWS
EXPERTE
beantwortet vor 10 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen