Analytics

Hi I'm using Athena to query AWS glue tables setup by a crawler from Dynamo DB. A Athena query to : show tables works perfect However when i do a simple select on any of the tables I get: GENERIC_INTERNAL_ERROR: java.net.URISyntaxException: Relative path in absolute URI: arn:aws:dynamodb:eu-west-1:[account]:table/[table-name] My query ID is: Query Id: 22e3a909-f7b4-41a7-9dfb-96b081f4b3b5

I am syncing my AWS Neptune nodes in an AWS OpenSearch cluster as per the documentation https://docs.aws.amazon.com/neptune/latest/userguide/full-text-search.html. The name of the OpenSearch index is amazon_neptune. The OpenSearch index type is _doc. Following is the index configuration ``` { "settings": { "number_of_shards": 1, "number_of_replicas": 1, "analysis": { "normalizer": { "useLowercase": { "type": "custom", "filter": "lowercase" } } } }, "mappings": { "properties": { "document_type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "entity_id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "entity_type" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "predicates": { "properties": { "content": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above" : 1000, "normalizer": "useLowercase" } } }, "visibilityType": { "type": "keyword" }, "status": { "type": "keyword" }, "type": { "type": "keyword" }, "firstName": { "type": "text", "fields": { "keyword": { "type": "keyword", "normalizer": "useLowercase" } } }, "lastName": { "type": "text", "fields": { "keyword": { "type": "keyword", "normalizer": "useLowercase", "ignore_above" : 1000 } } } } } } } } ``` Using the npm gremlin package, I'm trying to query my documents. Following is the code: ``` 'use strict'; const gremlin = require('gremlin'); exports.handler = async (event, context) => { try { const DriverRemoteConnection = gremlin.driver.DriverRemoteConnection; const Graph = gremlin.structure.Graph; const dc = new DriverRemoteConnection(<neptune_endpoint>,{}); const graph = new Graph(); const dbClient = graph.traversal().withRemote(dc); const res = await dbClient .withSideEffect("Neptune#fts.endpoint",<https_opensearch_endpoint>) .withSideEffect('Neptune#fts.queryType', 'term') .V().has("visibilityType","Neptune#fts PUBLIC") .toList(); console.log('res:', res); } catch(err) { console.error('Failed to query', err); } } ``` But I'm getting the following error ``` Failed to query ResponseError: Server error: {"detailedMessage":"method [POST], host [<https_opensearch_endpoint>], URI [/amazon_neptune/_search?typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 403 Forbidden]\n{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost\"}","requestId":"23a9e7d7-7dde-465b-bf29-9c59cff12e86","code":"BadRequestException"} (500) ``` I have given the following permission to my lambda ``` Type: AWS::IAM::Policy Properties: PolicyName: <Policy_Name> Roles: - 'Ref': <lambda_role> PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - es:ESHttpGet - es:ESHttpPost - es:ESHttpPut - es:ESHttpDelete Resource: <opensearch_cluster_arn> ``` My OpenSearch cluster as well as Neptune cluster are located inside the same VPC. My lambda is hosted inside the same VPC as well. Please help me in understanding why I'm getting the 403 error when I've given the proper reading permissions to my lambda. Any help would be highly appreciated.

I I have performed some AWS Glue version 3.0 jobs testing using Docker containers as detailed [here](https://aws.amazon.com/blogs/big-data/develop-and-test-aws-glue-version-3-0-jobs-locally-using-a-docker-container/). The following code outputs two lists, one per connection, with the names of the tables in a database: ```python import boto3 db_name_s3 = "s3_connection_db" db_name_mysql = "glue_catalog_mysql_connection_db" def retrieve_tables(database_name): session = boto3.session.Session() glue_client = session.client("glue") response_get_tables = glue_client.get_tables(DatabaseName=database_name) return response_get_tables s3_tables_list = [table_dict["Name"] for table_dict in retrieve_tables(db_name_s3)["TableList"]] mysql_tables_list = [table_dict["Name"] for table_dict in retrieve_tables(db_name_mysql)["TableList"]] print(f"These are the tables from {db_name_s3} db: {s3_tables_list}\n") print(f"These are the tables from {db_name_mysql} db {mysql_tables_list}") ``` Now, I try to create a dynamic dataframe with the *from_catalog* method in this way: ```python import sys from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job from awsglue.dynamicframe import DynamicFrame source_activities = glueContext.create_dynamic_frame.from_catalog( database = db_name, table_name =table_name ) ``` When `database="s3_connection_db"`, everything works fine, however, when I set `database="glue_catalog_mysql_connection_db"`, I get the following error: ```python Py4JJavaError: An error occurred while calling o45.getDynamicFrame. : java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver ``` I understand the issue is related to the fact that I am trying to fetch data from a mysql table but I am not sure how to solve this. By the way, the job runs fine on the Glue console. I would really appreciate some help, thanks!

Hi, I'm using OpenSearch, and for paginating the search results, should I use the simple from/size, or should I go with search after. I've read that if the number of hits is less than 10k, then from/size is acceptable. Even though my total search space is much more than 10k, but the number of hits/matches I'm expecting is around 5k at max. For this, would there be a lot of performance difference between from/size and search after? I'm leaning towards from/size since search after is relatively trickier to implement, specially when there's a requirement for going back to a previous page. Thanks!

Hi, we are using X-Ray for application logs tracing but X-Ray is not able to show case all the logs and after some times it drops off the tracing. Is it some limitations with X-Ray or how we can enables X-Ray for the complete logs tracing.

I was looking at ways in which we can ingest data using Kinesis data streams without creating an IAM user & generating access & security tokens. Possible alternatives I have found include: 1. Let api gateway assume a role with correct permissions & use it to send data. I think this might be prohibitively expensive. Any insights? 2. Generate temporary credentials using STS & send it to end user. Drawback would be needing to replace credentials every now and then. 3. Maybe use anonymous users functionality in Cognito identity pool & allow access to Kinesis that way. Not sure if this is even viable. Any insight would be very valuable. Thanks in advance!

I have been using QuickSight for a while. I am importing data from Athena and RDS into Spice and building reports on them. I am now being told in the QuickSight console that my Spice Capacity is full - apparently I am using 320GB of data. However, when I look at all my data sets, the sizes only add up to about 20GB, so I've no idea how I've hit 320GB. Is there some way I can see the Spice usage breakdown? Or could there be some hidden old versions of data that it is holding onto?

I'm trying to join a simple "spine" table with a "feature" table. The spine table may or may not have additional columns besides the join key. If there are additional columns (in this case, random_column), the following query works fine (please note that these queries are self-sufficient. they don't depend on any external tables - all tables are inline created by the query itself): ``` with spine_table as (select 'abc' as user_id, 'random_value' as random_column), feature_table as (select 'abc' as user_id, '123' as feature_column) select user_id, spine_table.*, feature_column from spine_table join feature_table using (user_id) ``` If, however, there's no additional column, the query throws an exception: ``` with spine_table as (select 'abc' as user_id), feature_table as (select 'abc' as user_id, '123' as feature_column) select user_id, spine_table.*, feature_column from spine_table join feature_table using (user_id) ``` `Error: SYNTAX_ERROR: line 5:17: Table 'spine_table' not found` The second query works fine if I omit spine_table. : with spine_table as ``` (select 'abc' as user_id), feature_table as (select 'abc' as user_id, '123' as feature_column) select user_id, feature_column from spine_table join feature_table using (user_id) ``` The problem is that my application dynamically generates the query, and it doesn't know ahead of time whether there are additional columns in the spine_table besides the join keys.

Hi, the current pipeline I am implementing involves a Kinesis Data Stream -> Kinesis Delivery Stream -> S3 route, however when setting the buffer intervals (Both s3 and transformation) to near 5 minutes, data still rapidly appears within the S3 bucket. For my purposes I need the data for every 5 minutes to be combined into a single large file to be pushed into S3, which I was under the assumption Kinesis delivery streams should handle under the hood when taking in from a Kinesis data stream. I'd appreciate any help in pointing out where my implementation could potentially be going astray :).

Hello, The use case I have is this - There are two types of AWS IoT Greengrass V2 core devices that are implemented, which are connected (in the same private LAN network) in hub and spoke architecture. None of them are connected to client devices (Greengrass is being used because of its IPC and orchestration benefits): 1. [Spoke] AWS IoT Greengrass V2 core device is directly attached to a camera. The Video stream is sent to an Hub AWS IoT Greengrass V2 core device for ML processing (inference) that must be near-real time. 2. [Hub] AWS IoT Greengrass V2 core device that is processing and Fan-Out video streams: **A)** to ML inference interface **B)** to Kinesis Firehose (S3; to re-train the model) **C)** AWS Kinesis Video Stream (for human to view the video online) I have a couple of questions: 1. Is the architecture feasible? Make sense? 2. What is the best (performance and security wise) technology (open source, AWS component, protocol) to use in Spoke and Hub devices to send the video stream from the spokes to the hub (the video has to be high quality with minimal/no compression to keep the inference accuracy high)? 2. Can the Stream Manager component of AWS IoT Greengrass V2 core send (Hub) streams in fun-out mode (three different destinations concurrently)? Thank you, Yossi

How can DMS do more complex mapping that is not simply 1-1 from source to target? For example, if you need to check for data across multiple tables for some business logic and based on those results, map a certain string over to the target.

Hello. We got reserved nodes that will expire in a few days. I don't want to prolong them because we changed the instance type from `ds` to `ra`. I'm asking this question because I want to be 100% sure that there will be **no** payment, right now I've got the following: ``` Reserved nodes: 4 Reserved nodes used: 0 Reserved nodes available: 4 ```

Is there a way to create a Lakeformation Resource Link to a database/table through Boto3 client. I am creating a database using glue_client.create_database() function. Now , I need to create a Lakeformation Resource Link Pointing to that database.

I'm using a AWS Glue crawler for a RDS Aurora PostgreSQL, the table contains some columns with the money datatype, the crawler create the table with those columns as double. `AWSGlueDataCatalog_node1652381535100 = glueContext.create_dynamic_frame.from_catalog( database="tmp_db", table_name="postgres_public_list", transformation_ctx="AWSGlueDataCatalog_node1652381535100",) ` But then when I start to working on the AWS Glue, any action, even a simple AWSGlueDataCatalog_node1652381535100.count() gives me this kind of error: > ... Caused by: org.postgresql.util.PSQLException: Bad value for type double : 2,400.00 ... Doesn't matter if I try to change the columns datatype to string, keeps saying the same error. Any ideas how to solve this issue?

After integrating with KeyCloak, SAML post is giving 500 response. Nothing in logs. What is the best way to debug/log the error?

I have an RDS PostgreSQL database that has table names and column names with upper cased characters. I have created a glue crawler that connects to the database via jdbc and populates the glue data catalog with the database schemas but in that translation the upper case characters are converted to lower case characters when stored in the data catalog tables. When I run a glue job to query I get this error: An error occurred while calling o94.getDynamicFrame. ERROR: column "id" does not exist I made a copy of this table and changed the table names and column names to have all lower case characters and the same glue jobs and queries run successfully. Changing the table names and column names to lower cased characters in our production environment is just not an option due to the extensive work it would require. I found the 'Edit Schema' option in the Glue UI where you can change column names and data types and thought for a moment the solution had been found. However, when you change a character to upper case and then select "Save" it is reverted to lower case as it is saved. I have edited the pyspark script directly and worked with the glueContext.create_dynamic_frame.from_catalog method using the additional_options parameter to build my select statement using upper and lower case characters but that still fails with the error message noted above. # Script generated for node PostgreSQL table PostgreSQLtable_node1 = glueContext.create_dynamic_frame.from_catalog( database="mydatabase", table_name="mytable", additional_options={"query":"SELECT id from mytable;"}, transformation_ctx="PostgreSQLtable_node1" I believe the failure is because the schema as it is stored in the data catalog contains lower characters while the actual schema in the database is upper characters so when Glue tries to work with the table it is looking for "id" while the actual is "ID" and so "not found" is returned. I have read about the CaseSensitive option and looking in that direction next for a solution. I have not seen any recent (less than couple years old) posts about this issue so not sure if I'm missing something. Any assistance would be greatly appreciated.

I have an event rule rule that triggers a glue job. I would like to pass information from the event details as a parameter to the Glue job. For example, say my event contains: {"details": {"database_name": "my_database"}} and and my job has a parameter --DATABASE_NAME. Is it possible to pass database_name from the event as the --DATABASE_NAME parameter in the job? I haven't been able to find anything in the documentation or online on how to do it. Thanks!

I am trying to perform a fairly simple delete via Athena on a table defined with the tabletype of Iceberg. *delete from emaildelivery where uniqueid in (select b.uniqueid from deletestaging b)* The following error message is returned. *GENERIC_INTERNAL_ERROR: Invalid descendant for DeleteNode or UpdateNode: com.facebook.presto.sql.planner.plan.JoinNode* Performing the query with values directly in the query is successful. *delete from emaildelivery where uniqueid in ('6e41279b-a22d-4bad-ad8d-448625ae9d88','2831e5a1-8a5c-424f-8d07-198a57f6a04a')* The same query as a select works successfully. *select uniqueid from emaildelivery where uniqueid in (select b.uniqueid from deletestaging b);* Any assistance with resolving the issue with the delete query is greatly appreciated.