Glue secret manager integration: secretId is not provided

0

I have an existing AWS Glue script that has been successfully running in Glue 2.4 for some time. I went in today to upgrade it to Glue 3.0 and am unable to connect to my database. I am simply reading in data from an Oracle database using glueContext.create_dynamic_frame.from_catalog and then writing it to a postgresql database.

This is the line that it is failing on

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = databaseName, table_name = tableName, transformation_ctx = f"dropfields{tableName}")

I have a connection defined and am able to successfully connect to the database. But I am getting the following error: Glue secret manager integration: secretId is not provided when I run the job.

Here is the full stack

24/01/31 20:57:39 ERROR GlueExceptionAnalysisListener: [Glue Exception Analysis] Last Executed Line number from script longview.py: 38
24/01/31 20:57:38 ERROR GlueExceptionAnalysisListener: [Glue Exception Analysis] {"Event":"GlueETLJobExceptionEvent","Timestamp":1706734658975,"Failure Reason":"Traceback (most recent call last):\n  File \"/tmp/longview.py\", line 38, in <module>\n    datasource0 = glueContext.create_dynamic_frame.from_catalog(database = databaseName, table_name = tableName, transformation_ctx = f\"dropfields{tableName}\")\n  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py\", line 787, in from_catalog\n    return self._glue_context.create_dynamic_frame_from_catalog(db, table_name, redshift_tmp_dir, transformation_ctx, push_down_predicate, additional_options, catalog_id, **kwargs)\n  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/context.py\", line 188, in create_dynamic_frame_from_catalog\n    return source.getFrame(**kwargs)\n  File \"/opt/amazon/lib/python3.6/site-packages/awsglue/data_source.py\", line 36, in getFrame\n    jframe = self._jsource.getDynamicFrame()\n  File \"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py\", line 1305, in __call__\n    answer, self.gateway_client, self.target_id, self.name)\n  File \"/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 111, in deco\n    return f(*a, **kw)\n  File \"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py\", line 328, in get_return_value\n    format(target_id, \".\", name), value)\npy4j.protocol.Py4JJavaError: An error occurred while calling o93.getDynamicFrame.\n: java.sql.SQLRecoverableException: Listener refused the connection with the following error:\nORA-12514, TNS:listener does not currently know of service requested in connect descriptor\n  (CONNECTION_ID=bB1y1v6uSpCWCObiZ7c4Yg==)\n\tat oracle.jdbc.driver.T4CConnection.handleLogonNetException(T4CConnection.java:870)\n\tat oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:675)\n\tat oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:1032)\n\tat oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:90)\n\tat oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:681)\n\tat oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:602)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectionProperties$5(JDBCUtils.scala:992)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectWithSSLAttempt$2(JDBCUtils.scala:944)\n\tat scala.Option.getOrElse(Option.scala:121)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectWithSSLAttempt$1(JDBCUtils.scala:944)\n\tat scala.Option.getOrElse(Option.scala:121)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.connectWithSSLAttempt(JDBCUtils.scala:944)\n\tat com.amazonaws.services.glue.util.JDBCWrapper$.connectionProperties(JDBCUtils.scala:988)\n\tat com.amazonaws.services.glue.util.JDBCWrapper.connectionProperties$lzycompute(JDBCUtils.scala:752)\n\tat com.amazonaws.services.glue.util.JDBCWrapper.connectionProperties(JDBCUtils.scala:752)\n\tat com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:878)\n\tat com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)\n\tat com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)\n\tat com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1014)\n\tat com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:101)\n\tat com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:101)\n\tat com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:725)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:750)\nCaused by: oracle.net.ns.NetException: Listener refused the connection with the following error:\nORA-12514, TNS:listener does not currently know of service requested in connect descriptor\n  (CONNECTION_ID=bB1y1v6uSpCWCObiZ7c4Yg==)\n\tat oracle.net.ns.NSProtocolNIO.createRefusePacketException(NSProtocolNIO.java:816)\n\tat oracle.net.ns.NSProtocolNIO.handleConnectPacketResponse(NSProtocolNIO.java:396)\n\tat oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:207)\n\tat oracle.net.ns.NSProtocol.connect(NSProtocol.java:350)\n\tat oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1967)\n\tat oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:640)\n\t... 31 more\n","Stack Trace":[{"Declaring Class":"get_return_value","Method Name":"format(target_id, \".\", name), value)","File Name":"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py","Line Number":328},{"Declaring Class":"deco","Method Name":"return f(*a, **kw)","File Name":"/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py","Line Number":111},{"Declaring Class":"__call__","Method Name":"answer, self.gateway_client, self.target_id, self.name)","File Name":"/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py","Line Number":1305},{"Declaring Class":"getFrame","Method Name":"jframe = self._jsource.getDynamicFrame()","File Name":"/opt/amazon/lib/python3.6/site-packages/awsglue/data_source.py","Line Number":36},{"Declaring Class":"create_dynamic_frame_from_catalog","Method Name":"return source.getFrame(**kwargs)","File Name":"/opt/amazon/lib/python3.6/site-packages/awsglue/context.py","Line Number":188},{"Declaring Class":"from_catalog","Method Name":"return self._glue_context.create_dynamic_frame_from_catalog(db, table_name, redshift_tmp_dir, transformation_ctx, push_down_predicate, additional_options, catalog_id, **kwargs)","File Name":"/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py","Line Number":787},{"Declaring Class":"<module>","Method Name":"datasource0 = glueContext.create_dynamic_frame.from_catalog(database = databaseName, table_name = tableName, transformation_ctx = f\"dropfields{tableName}\")","File Name":"/tmp/longview.py","Line Number":38}],"Last Executed Line number":38,"script":"longview.py"}
24/01/31 20:57:38 ERROR ProcessLauncher: Error from Python:Traceback (most recent call last):
  File "/tmp/longview.py", line 38, in <module>
    datasource0 = glueContext.create_dynamic_frame.from_catalog(database = databaseName, table_name = tableName, transformation_ctx = f"dropfields{tableName}")
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py", line 787, in from_catalog
    return self._glue_context.create_dynamic_frame_from_catalog(db, table_name, redshift_tmp_dir, transformation_ctx, push_down_predicate, additional_options, catalog_id, **kwargs)
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/context.py", line 188, in create_dynamic_frame_from_catalog
    return source.getFrame(**kwargs)
  File "/opt/amazon/lib/python3.6/site-packages/awsglue/data_source.py", line 36, in getFrame
    jframe = self._jsource.getDynamicFrame()
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o93.getDynamicFrame.
: java.sql.SQLRecoverableException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
  (CONNECTION_ID=bB1y1v6uSpCWCObiZ7c4Yg==)
	at oracle.jdbc.driver.T4CConnection.handleLogonNetException(T4CConnection.java:870)
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:675)
	at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:1032)
	at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:90)
	at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:681)
	at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:602)
	at com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectionProperties$5(JDBCUtils.scala:992)
	at com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectWithSSLAttempt$2(JDBCUtils.scala:944)
	at scala.Option.getOrElse(Option.scala:121)
	at com.amazonaws.services.glue.util.JDBCWrapper$.$anonfun$connectWithSSLAttempt$1(JDBCUtils.scala:944)
	at scala.Option.getOrElse(Option.scala:121)
	at com.amazonaws.services.glue.util.JDBCWrapper$.connectWithSSLAttempt(JDBCUtils.scala:944)
	at com.amazonaws.services.glue.util.JDBCWrapper$.connectionProperties(JDBCUtils.scala:988)
	at com.amazonaws.services.glue.util.JDBCWrapper.connectionProperties$lzycompute(JDBCUtils.scala:752)
	at com.amazonaws.services.glue.util.JDBCWrapper.connectionProperties(JDBCUtils.scala:752)
	at com.amazonaws.services.glue.util.JDBCWrapper.tableDF(JDBCUtils.scala:878)
	at com.amazonaws.services.glue.util.NoCondition$.tableDF(JDBCUtils.scala:88)
	at com.amazonaws.services.glue.util.NoJDBCPartitioner$.tableDF(JDBCUtils.scala:174)
	at com.amazonaws.services.glue.JDBCDataSource.getDynamicFrame(DataSource.scala:1014)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame(DataSource.scala:101)
	at com.amazonaws.services.glue.DataSource.getDynamicFrame$(DataSource.scala:101)
	at com.amazonaws.services.glue.AbstractSparkSQLDataSource.getDynamicFrame(DataSource.scala:725)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:750)
Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
  (CONNECTION_ID=bB1y1v6uSpCWCObiZ7c4Yg==)
	at oracle.net.ns.NSProtocolNIO.createRefusePacketException(NSProtocolNIO.java:816)
	at oracle.net.ns.NSProtocolNIO.handleConnectPacketResponse(NSProtocolNIO.java:396)
	at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:207)
	at oracle.net.ns.NSProtocol.connect(NSProtocol.java:350)
	at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1967)
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:640)
	... 31 more

24/01/31 20:57:37 INFO GlueContext: The DataSource in action : com.amazonaws.services.glue.JDBCDataSource
24/01/31 20:57:37 INFO GlueContext: Glue secret manager integration: secretId is not provided.
24/01/31 20:57:37 INFO GlueContext: nameSpace: longview-prod, tableName: dwpbrdmp_longview_budget_data, connectionName longview-prod, vendor: oracle
24/01/31 20:57:37 INFO GlueContext: getCatalogSource: transactionId: <not-specified> asOfTime: <not-specified> catalogPartitionIndexPredicate: <not-specified> 
24/01/31 20:57:37 INFO GlueContext: getCatalogSource: catalogId: null, nameSpace: longview-prod, tableName: dwpbrdmp_longview_budget_data, isRegisteredWithLF: false, isGoverned: false, isRowFilterEnabled: false, useAdvancedFiltering: false, isTableFromSchemaRegistry: false
24/01/31 20:57:35 INFO GlueContext: GlueMetrics configured and enabled
24/01/31 20:57:32 INFO Utils: Successfully started service 'sparkDriver' on port 34409.

I was hoping not to have to change to using AWS Secrets. I just want to figure out why this is not working all of a sudden. I have tried providing additional_options = {"useConnectionProperties": "true","connectionName": "name-of-connection"}, but that didn't work. I even went back to Glue 2.0, but got the same result.

Any help would be greatly appreciated.

1 Answer
1

You don't have to use secret manager, that's just an info message.
The real issue there is that the service specified in the JDBC url doesn't seem to be recognized, if you didn't change anything else I wonder if the newer thin driver provided on Glue 3 maybe cannot handle it (never heard of such backwards compatibility breach). What about Glue 4?

profile pictureAWS
EXPERT
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions