Analytics

Using AWS Glue jobs, I want to insert, update and delete the rows in a Table in RDS Database based on the primary key. How do I achieve it.

Hi, I'm trying to iterate through the results returned from SelectFields action using the following command ``` SelectFields_node1656679636429.toDF().index ``` but i get the following error > AttributeError: 'DataFrame' object has no attribute 'index' it appears that even I convert the DynamicFrame to DataFrame, many attributes are missing. I found a workaround using RDD, and re-inialising a new DataFrame, like below ``` rdd = SelectFields_node1656679636429.toDF().rdd df = pd.DataFrame(rdd.collect()) df.index ``` but this approach seems an overhead. Are there any better, native ways of getting the properly initialized DataFrame from DynamicFrame?

Hello. I wonder why this doesn't work in my aws redshift. ``` select json_typeof(p.commentids), p.commentids from dbsparta_posts as p ``` gives an output "array" , [........] and when I query ``` select c._id from dbsparta_posts as p left join dbsparta_comments as c on c._id = any(array['62bfb16cb14835992402de91','62bfb37c5388dc79bdf23055']) ``` the join works fine. But when I query ``` select c._id from dbsparta_posts as p left join dbsparta_comments as c on c._id = any(p.commentids) ``` it prints out error > ERROR: op ANY/ALL (array) requires array on right side [ErrorId: 1-62bfe046-583cc20f60f8ed00400c46bc] Please help me out.

**Background** We've configured an export to s3 from dynamodb using the native dynamodb s3 export, and ION as the format output. After this, we've created a table in Athena ``` CREATE EXTERNAL TABLE export_2022_07_01_v4 ( `_PK` STRING, URL STRING, Item struct< `_PK`:string, URL:string > ) ROW FORMAT SERDE 'com.amazon.ionhiveserde.IonHiveSerDe' WITH SERDEPROPERTIES ( "ignore_malformed_ion" = "true" ) STORED AS ION LOCATION '...'; ``` Querying this works all right for small simple queries, but attempting to produce a full output with ``` UNLOAD ( SELECT Item.URL FROM "helioptileexports"."export_2022_07_01_v4" WHERE Item.URL IS NOT NULL ) to '...' WITH (format = 'TEXTFILE') ``` Results in this error ``` HIVE_CURSOR_ERROR: Syntax error at line 1 offset 2: invalid syntax [state:STATE_BEFORE_ANNOTATION_DATAGRAM on token:TOKEN_SYMBOL_OPERATOR] This query ran against the "helioptileexports" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: f4ca5812-1194-41f1-bfda-00a1a5b2471b ``` **Questions** 1. Is there a way to make Athena more tolerant of formatting errors on specific files? As shown in the example, we are attempting without success to use `ignore_malformed_ion`. Is there anything beyond that that can be done? 2. Is this a bug on DynamoDB ION export process? 3. Is there any mechanism or logging to identify the files which have the malformed data and remove them?

We are looking to expand services via AWS Quicksight, our use case would include Amazon subsidiary users who can be both in Amazon network and not. My question is it possible to setup new AWS accounts to allow BOTH Active Directory (for in network users) and unique IAM roles (for subsidiary/off-network users)? If not what are the options to allow this type of access using SSO where possible.

This query ran against the "sf_sav_raw" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 70884e2f-ed4d-4268-b5b5-1c630aa47570

My use case requires using neptune db for hitting cypher queries which gets converted to gremlin queries(automatically) and fetching the data. My question is does neptune db support APOC library given by cypher

Here is the error I am getting while loading partition to table and data is not getting push to table automatically but it is getting push to S3 Agent Event bucket. This query ran against the "akctrathenaquicksitereportdata" database, unless qualified by the query. Please help in fixing this issue.

Are there any ways to read delta lake files from s3 and create Data catalog on top of this to run Glue ETL job? When I crawl in delta folders it creates separate schema for log, manifest & parquets rather then each tables with all the delta log, manifest files and parquet files , https://docs.aws.amazon.com/glue/latest/dg/crawler-data-stores.html says Crawler now have native client then why it is not recognizing the path , Am I missing something.

Hi, I'm using AWS Glue Studio and once I click "data preview" it fails with the following error. The flow consists of 2 actions - PosgtreSQL JDBC Data source and Select Field action. The error is thrown instantly once the "Data Preview" button is clicked. The overall flow run successfully if i click "RUN" button - there are no errors and I get the outcome if I add the Target Source to dump the results back to PosgtreSQL table. It's just a Data Preview functionality that fails. Any idea what could be wrong and how to troubleshoot it? > Py4JJavaError: An error occurred while calling o538.getSampleDynamicFrame. : java.lang.UnsupportedOperationException: empty.reduceLeft at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:180) at scala.collection.AbstractTraversable.reduceLeft(Traversable.scala:104) at scala.collection.TraversableOnce$class.reduce(TraversableOnce.scala:208) at scala.collection.AbstractTraversable.reduce(Traversable.scala:104) at com.amazonaws.services.glue.SparkSQLDataSource.com$amazonaws$services$glue$SparkSQLDataSource$$getPaths(DataSource.scala:724) at com.amazonaws.services.glue.SparkSQLDataSource$$anonfun$getSampleDynamicFrame$7.apply(DataSource.scala:799) at com.amazonaws.services.glue.SparkSQLDataSource$$anonfun$getSampleDynamicFrame$7.apply(DataSource.scala:793) at com.amazonaws.services.glue.util.FileSchemeWrapper$$anonfun$executeWithQualifiedScheme$1.apply(FileSchemeWrapper.scala:89) at com.amazonaws.services.glue.util.FileSchemeWrapper$$anonfun$executeWithQualifiedScheme$1.apply(FileSchemeWrapper.scala:89) at com.amazonaws.services.glue.util.FileSchemeWrapper.executeWith(FileSchemeWrapper.scala:82) at com.amazonaws.services.glue.util.FileSchemeWrapper.executeWithQualifiedScheme(FileSchemeWrapper.scala:89) at com.amazonaws.services.glue.SparkSQLDataSource.getSampleDynamicFrame(DataSource.scala:792) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748)

Hi, I am Eko. I need help with how the connection from Cloud Machine AWS to Remote PC [my PC]? I've researched the setup of Cloud Machine AWS. but only work on Output Audio still not connected to the input Audio from Remote PC. I am very grateful for all the answers and responses

My team works with many 3rd parties and uses Workdocs to share artifacts with them. We are currently in the process of auditing our system to ensure that we are unsharing artifacts once each contract is up/if an employee leaves the 3rd party. The issue we're facing is the number of artifacts we own (2.69 TB) making it extremely tedious and unsustainable to see who has access to each folder/file. Is there a way to pull a list of users who have access to each folder and file under a parent folder? We have up to 12 layers of sub-folders under our parent folder, so clicking into each folder and file to see who has access to what would take weeks.

Hi folks, I'm trying to set the spark executor instances & memory, driver memory and switch of dynamic allocation. What is the correct way to do it?

An architectural mismatch has often led to issue like - When trying to connect through a dotnet application it fails with the following error message: "ERROR [01000] [unixODBC][Driver Manager]Can't open lib '/opt/amazon/redshift/lib/libamazonredshiftodbc.dylib' : file not found"

Hi, I found out this weird bug in Athena when showing table partitions using Athena engine version 2. When retrieving the most recent partition for a table, I can easily create this simple query which picks the latest year according to the partition. In my example I have a table which is partitioned by year e.g. year=2021/file.csv , year=2022/file.csv. If I run this query in Athena ``` select * from "my_year_table$partitions" order by year desc limit 1 ``` I get 2022 back However, If I try to create a view that does this automatically for me ``` create or REPLACE view HURR_DURR as select * from "my_year_table$partitions" order by year desc limit 1 ``` I'm greeted with the following error > line 5:2: Table [aws-account-number].my_database.my_year_table$partitions does not exist. This query ran against the "my_database" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 266fb6e9-4a14-47de-8760-a9a21b88387e The reason why I want to create a view is because I have tables with much more partitioning and I want some end users that query athena to simply run the view to retrieve the most recent partition key. Is there perhaps another way to go on about it? Best regards

I'm trying to use newest package for Amazon integration in MWAA. In this particular case I want to use `GlueJobOperator` which is a part of the latest `apache-airflow-providers-amazon` package. ([Link to the documentation](https://airflow.apache.org/docs/apache-airflow-providers-amazon/4.0.0/operators/glue.html)) MWAA Airflow version: ` 2.2.2 ` I added this to the *requirements.txt*: ``` apache-airflow-providers-amazon==4.0.0 ``` and tried to import it and use it like in the examples: ``` from airflow.providers.amazon.aws.operators.glue import GlueJobOperator glue_job = GlueJobOperator( task_id='airflow_id', job_name='job_name' wait_for_completion=True, script_location='bucket_script_prefix', s3_bucket='bucket_name', iam_role_name='iam_role' create_job_kwargs=job_arguments, script_args=script_arguments ) ``` Unfortunately, whenever DAG is parsed I get this error: ``` ... from airflow.providers.amazon.aws.operators.glue import GlueJobOperator ImportError: cannot import name 'GlueJobOperator' from 'airflow.providers.amazon.aws.operators.glue' (/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/glue.py) ``` It's not my first rodeo with MWAA and some extra packages, plugins etc. but I am lost. In this case, I tried many things, went through docs from cover to cover and I still couldn't find the reason. I verified in MWAA that packages were successfully installed both in the logs and in the UI with the version prompted in *requirements.txt*. | Package Name | Version | Description | | --- | --- | --- | | apache-airflow-providers-amazon |4.0.0 |Amazon integration (including Amazon Web Services (AWS)). | Fun fact: I'm using `S3Hook` in some other DAGs and it parses just fine. ``` from airflow.providers.amazon.aws.hooks.s3 import S3Hook s3_hook = S3Hook() ... ```

Hi, Currently am using DMS to capture db changes and sending them to Kafka. Am only interested in data (json values without "data" keyword included) and not metadata. I am following the user guide and specifically the below configuration but not working. Could you please assist. So am only interested in flat json payload with only the data values inserted to table. Example: { "prop1":"value1","prop2":"value2"} { "rules": [ { "rule-type": "selection", "rule-id": "1", "rule-name": "1", "rule-action": "include", "object-locator": { "schema-name": "BW3", "table-name": "INT_TRANSACTIONS" } }, { "rule-type": "object-mapping", "rule-id": "2", "rule-name": "DefaultMapToKafka", "rule-action": "map-record-to-record", "object-locator": { "schema-name": "BW3", "table-name": "INT_TRANSACTIONS" } } ] }

Hi I am trying to create an Athena table from S3 bucket that contains access logs from API Gw. The format of the data is as follows: ``` 2022-06-24T17:09:01.037Z { 'requestId’:’341122-4312-4311-9989-b837438748', 'ip': ’127.0.0.1, 'caller':'-', 'user':'-','requestTime':'24/Jun/2022:17:09:01 +0000', 'xrayTraceId':'-', 'wafResponseCode':'-', 'httpMethod':'GET','resourcePath’:’/items/list’, 'status':'200','protocol':'HTTP/1.1', 'responseLength':'238' } ``` I tried with and without row input serde to identify or process the data but nothing seems to be working. Without any input configuration I do the content in the table but its formatted incorrectly. I used RegEx, and tested my results on online resources and though the regex looks good online I cannot seem achieve the same result once the table is created. Either I will get a empty response or following message: HIVE_CURSOR_ERROR: Number of matching groups doesn't match the number of columns Any help is appreciated. I am expecting the result to be in this format: | timestamp | requestId | ip | caller | user | requestTime | xrayTraceId | wafResponseCode | httpMethod | resourcePath | status | protocol | responseLength | --- | --- | --- | --- | --- | --- | --- |--- | --- | --- | --- | --- | --- | |2022-06-24T17:09:01.037Z |341122-4312-4311-9989-b837438748 |127.0.0.1 |- |- |- |- |- |- |- |- |- |- |