An error occurred while calling o95.getDynamicFrame. key not found: location

0

In a AWS Glue job, I am getting issue when I am trying to execute sql query in below code:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
{"useConnectionProperties": "true", 
"connectionName": "d-glue-connection", 
"query": "select * from myschema.table1 where 1=1"
},
transformation_ctx = "source_dataset"
 )

Error Message:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/dynamicframe.py", line 609, in from_options
    return self._glue_context.create_dynamic_frame_from_options(connection_type,
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/context.py", line 237, in create_dynamic_frame_from_options
    return source.getFrame(**kwargs)
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/data_source.py", line 37, in getFrame
    jframe = self._jsource.getDynamicFrame()
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o95.getDynamicFrame.
: java.util.NoSuchElementException: key not found: location
	at scala.collection.MapLike.default(MapLike.scala:236)
	at scala.collection.MapLike.default$(MapLike.scala:235)
	at scala.collection.AbstractMap.default(Map.scala:65)
	at scala.collection.MapLike.apply(MapLike.scala:144)
	at scala.collection.MapLike.apply$(MapLike.scala:143)
	at scala.collection.AbstractMap.apply(Map.scala:65)
	at com.amazonaws.services.glue.util.JDBCWrapper.$anonfun$qualifiedTableName$2(JDBCUtils.scala:923)
	at scala.Option.getOrElse(Option.scala:189)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName$lzycompute(JDBCUtils.scala:923)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName(JDBCUtils.scala:921)
	at com.amazonaws.services.glue.util.JDBCWrapper.dbtable(JDBCUtils.scala:934)
	at com.amazonaws.services.glue.util.JDBCWrapper.sqlTableName(JDBCUtils.scala:927)
	at com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:944)
	at com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)
	at com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)
	at com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1027)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:102)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:102)
	at com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:726)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:750)

Issue is with query option in the code, In AWS documentation I have read that we can use query option as well in place of dbTable.

What is the issue and its correct solution?

preguntada hace 10 meses2120 visualizaciones
2 Respuestas
0

Hi, your query 'select * from myschema.table1 where 1=1' is very peculiar! What is the purpose of this 1=1 ? You should retry your code with only ''select * from myschema.table1'.

BTW, your NoSuchElementException usually means that Java JDBC driver tries to iterate over the result set of the query but doesn't find any corresponding item.

So,

  1. does table myschema.table1 exist in your database ?
  2. does it contain any row at all?

Best, Didier

profile pictureAWS
EXPERTO
respondido hace 10 meses
  • Yes, myschema.table1 exist in my database and select * from myschema.table1 returns 1840 rows when it is executed outside the Glue code.

0

"query" is not part of the API, "sampleQuery" is, see: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-jdbc Using DataFrame you can use "query" instead of dbTable.

profile pictureAWS
EXPERTO
respondido hace 10 meses

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas