An error occurred while calling o95.getDynamicFrame. key not found: location

0

In a AWS Glue job, I am getting issue when I am trying to execute sql query in below code:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
{"useConnectionProperties": "true", 
"connectionName": "d-glue-connection", 
"query": "select * from myschema.table1 where 1=1"
},
transformation_ctx = "source_dataset"
 )

Error Message:

source_dataset = glueContext.create_dynamic_frame.from_options(connection_type="oracle", connection_options =
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/dynamicframe.py", line 609, in from_options
    return self._glue_context.create_dynamic_frame_from_options(connection_type,
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/context.py", line 237, in create_dynamic_frame_from_options
    return source.getFrame(**kwargs)
  File "/opt/amazon/lib/python3.7/site-packages/awsglue/data_source.py", line 37, in getFrame
    jframe = self._jsource.getDynamicFrame()
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o95.getDynamicFrame.
: java.util.NoSuchElementException: key not found: location
	at scala.collection.MapLike.default(MapLike.scala:236)
	at scala.collection.MapLike.default$(MapLike.scala:235)
	at scala.collection.AbstractMap.default(Map.scala:65)
	at scala.collection.MapLike.apply(MapLike.scala:144)
	at scala.collection.MapLike.apply$(MapLike.scala:143)
	at scala.collection.AbstractMap.apply(Map.scala:65)
	at com.amazonaws.services.glue.util.JDBCWrapper.$anonfun$qualifiedTableName$2(JDBCUtils.scala:923)
	at scala.Option.getOrElse(Option.scala:189)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName$lzycompute(JDBCUtils.scala:923)
	at com.amazonaws.services.glue.util.JDBCWrapper.qualifiedTableName(JDBCUtils.scala:921)
	at com.amazonaws.services.glue.util.JDBCWrapper.dbtable(JDBCUtils.scala:934)
	at com.amazonaws.services.glue.util.JDBCWrapper.sqlTableName(JDBCUtils.scala:927)
	at com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:944)
	at com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)
	at com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)
	at com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1027)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:102)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:102)
	at com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:726)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:750)

Issue is with query option in the code, In AWS documentation I have read that we can use query option as well in place of dbTable.

What is the issue and its correct solution?

질문됨 10달 전2120회 조회
2개 답변
0

Hi, your query 'select * from myschema.table1 where 1=1' is very peculiar! What is the purpose of this 1=1 ? You should retry your code with only ''select * from myschema.table1'.

BTW, your NoSuchElementException usually means that Java JDBC driver tries to iterate over the result set of the query but doesn't find any corresponding item.

So,

  1. does table myschema.table1 exist in your database ?
  2. does it contain any row at all?

Best, Didier

profile pictureAWS
전문가
답변함 10달 전
  • Yes, myschema.table1 exist in my database and select * from myschema.table1 returns 1840 rows when it is executed outside the Glue code.

0

"query" is not part of the API, "sampleQuery" is, see: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-jdbc Using DataFrame you can use "query" instead of dbTable.

profile pictureAWS
전문가
답변함 10달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠