Questions tagged with Analytics

Hi, is there a way to setup Quicksights incremental refresh to update “modified” records? I have a large database of sales records and want “near real time” metrics. Right now it is scheduled to refresh each hour, but would like to get that down to 15 minutes. Thought I could setup incremental refresh for this, but my problem is a row may not be modified for a year and I want to capture that change when it does happen. Any way to achieve this with Quicksight (Direct query is too slow for my calculations).

I have an etl pipeline that loads json data from a source bucket, runs an etl job with bookmarking enabled, and writes as parquet to a target bucket. I'd like to ensure that the target bucket never contains duplicate records, what's the best way to achieve that? My json records have a "requestid" field that's unique.

When does AWS EMR be available with AL2 with kernel 5.x or AL2022 with kernel 5.x. If we wanted to use custom AMI's, can you provide documentation of how to create custom AMI's for EMR.

Hi Team, what are the AWS services and products equivalent to google analytics to catch the user's behaviours from the frontend ? to set it up in AWS environment.

I'm syncing data written to S3 using Apache Hudi with Hive & Glue. Hudi options: ``` hudi_options: 'hoodie.table.name': mytable 'hoodie.datasource.write.recordkey.field': Id 'hoodie.datasource.write.partitionpath.field': date 'hoodie.datasource.write.table.name': mytable 'hoodie.datasource.write.operation': upsert 'hoodie.datasource.write.precombine.field': LastModifiedDate 'hoodie.datasource.hive_sync.enable': true 'hoodie.datasource.hive_sync.partition_fields': date 'hoodie.datasource.hive_sync.database': hudi_lake_dev 'hoodie.datasource.hive_sync.table': mytable ``` EMR Configurations: ``` ... { "Classification": "yarn-site", "Properties": { "yarn.nodemanager.vmem-check-enabled": "false", "yarn.log-aggregation-enable": "true", "yarn.log-aggregation.retain-seconds": "-1", "yarn.nodemanager.remote-app-log-dir": config[ "yarn_agg_log_uri_s3_path" ].format(current_date), }, "Configurations": [], }, { "Classification": "spark-hive-site", "Properties": { "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory" # noqa }, }, { "Classification": "presto-connector-hive", "Properties": {"hive.metastore.glue.datacatalog.enabled": "true"}, }, { "Classification": "hive-site", "Properties": { "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory", # noqa "hive.metastore.schema.verification": "false", }, }, ... ``` Getting the following error: ``` Traceback (most recent call last): File "/home/hadoop/my_pipeline.py", line 31, in <module> ... File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 1109, in save File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o84.save. : org.apache.hudi.hive.HoodieHiveSyncException: Got runtime exception when hive syncing at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:74) at org.apache.hudi.HoodieSparkSqlWriter$.syncHive(HoodieSparkSqlWriter.scala:391) at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$4(HoodieSparkSqlWriter.scala:440) at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$4$adapted(HoodieSparkSqlWriter.scala:436) at scala.collection.mutable.HashSet.foreach(HashSet.scala:79) at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:436) at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:497) at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:223) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90) at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:194) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:232) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:229) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:190) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:134) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:133) at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989) at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107) at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232) at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135) at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107) at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989) at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438) at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to create HiveMetaStoreClient at org.apache.hudi.hive.HoodieHiveClient.<init>(HoodieHiveClient.java:93) at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:69) ... 46 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory not found) at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:239) at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:402) at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:335) at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:315) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:291) at org.apache.hudi.hive.HoodieHiveClient.<init>(HoodieHiveClient.java:91) ... 47 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory not found) at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3991) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:251) at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:234) ... 52 more Caused by: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory not found) at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClientFactory(HiveUtils.java:525) at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClient(HiveUtils.java:506) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3746) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3726) at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3988) ... 54 more 22/01/12 16:37:53 INFO SparkContext: Invoking stop() from shutdown hook ```

Hello, I am wondering if it is possible to "link" multiple parameters together so that a change in one is reflected in the other. For example, I am working with a dashboard and I want the user to be able to filter the data by a date range. Currently, I have `startDate` and `endDate` parameters which are used to filter all visuals in the dataset. This works fine, but I also want the user to quickly be able to select all the days in a month without having to go through the hassle of changing the date parameters. For example, if the user wants to view data for **September 2019**, rather than having to navigate all the way back to this date in the date parameter controls, it would be nice if I could add a dropdown, in which the user could simply select `Sept 2019`. The main problem is that I don't know how to keep these parameters in sync. In the above example, when the user selects `Sept 2019` from the month dropdown, it would be nice if the `startDate` and `endDate` parameters would automatically change to `2019-09-01` and `2019-09-30`, respectively. Is there a way to do this within QuickSight? Or if not, can anyone suggest any helpful workarounds for my described problem? I greatly appreciate it!

I am trying to get my google analytics data to S3. I am using AppFlow for this. At First it worked fine but after few days it started showing below error to me: > Error authenticating to connector: Login failed with status code 400: { "error": "invalid_grant", "error_description": "Token has been expired or revoked." } {requestUri=https://www.googleapis.com/analytics/v3/metadata/ga/columns, instanceUrl=https://analyticsreporting.googleapis.com} What could be issue for this and what is possible solution. In one of the comments someone said we need SSL for AppFlow. Can you you tell me where to get one for google analytics.

I've glue catalog and use s3 as DB. Datatype of columns are numeric in Redshift which i'm casting as decimal in Glue. But getting issue while querying the data in Athena. Column in redshift looks like (-12.887686) with data type Number. we cannot change the data type in source table

The dataset present: Time. Odometer reading t1, 120 t2, 210...... I am trying to find out how many miles the car has driven in the last 30 days in QuickSight.On reviewing a help document(https://docs.aws.amazon.com/quicksight/latest/user/periodToDateAvg-function.html) I thought of using PeriodTodateAvg function. periodToDateAvg(Odometer reading,timeStamp,MONTH) The above formula is giving me null values. It would be really helpful if someone could guide me. Thank you

Hi, I am signed in with a userId which has IAM Administrator access and DataLake Administrator access. The userId also has permissions to table columns set up in Lake Formation. However when creating a Data Filter on a table using console, I get following error message. > Error creating data cell filter: data-filter-1. Not authorized to create filter. Any suggestions to resolve the issue will be greatly appreciated.

Hi I'm trying to start an elasticsearch container via - docker compose (aws-cli and switching to ecs context), but it fails to start - AccessDeinedExcception - cant write to /usr/share/elasticsearch/data/nodes/ directory. I have researched the issue on google and its because of the permission on that folder - from my understanding I need to fix the permissions in the host directory mapped to /usr/share/elasticsearch/data/nodes/ (I think) running sudo chown -R 1000:1000 [directory} However my container shuts down and how am I supposed to update the permission on that directory? this is my docker-compose - any help appreciated version: '3.8' services: elasticsearch01: user: $USER image: docker.elastic.co/elasticsearch/elasticsearch:7.14.1 #image: 645694603269.dkr.ecr.eu-west-2.amazonaws.com/smpn_ecr:latest container_name: es02 restart: unless-stopped environment: cluster.name: docker-es-cluster discovery.type: single-node bootstrap.memory_lock: "true" # ES_JAVA_OPTS: "-Xms2g -Xmx2g" xpack.security.enabled: "false" xpack.monitoring.enabled: "false" xpack.watcher.enabled: "false" node.name: es01 network.host: 0.0.0.0 logger.level: DEBUG ulimits: memlock: soft: -1 hard: -1 volumes: - es_data01:/usr/share/elasticsearch/data:rw ports: - "9200:9200" - "9300:9300" healthcheck: test: "curl -f http://localhost:9200 || exit 1" networks: - smpn_network volumes: es_data01: driver: local networks: smpn_network: driver: bridge

We are migrating from GitLab to AWS CodeCommit. With GitLab the admins has access to tons of https://docs.gitlab.com/ee/user/analytics/. After exploring the AWS CodeCommit console can't find anything similar. We don't want to be writing our own analytics tool obviously, so my question is for the AWS CodeCommit community - how do you generate and visualize similar metrics & trends?

Hi, has anyone encountered the need to send QuickSight reports (from dashboard) via email including ALL content of specific Pivot Tables in terms of rows and columns? Not just a snapshot in pdf, a complete report, Excel or pdf format. Thanks for your help.

Hello world, Let's say I have 3 sets of users consuming 2 topics and I want: User 1 to have full control to both topics; User 2 can have full control to topic 1; User 3 can subscribe to topic 2. What is the best way of implementing such? Thank you

Athena and redshift are incorrectly grouped under analytics. quicksight is the correct tool in that category. But redshift should be under database. https://i.stack.imgur.com/8O65F.png

We recently switched our Redshift cluster from 14 x dc2.large to 4 x ra3.xlplus as we really liked the idea to have the storage decoupled from the nodes. I expected some kind of change in performance since we're having now 16vCPU vs 28vCPU and 210GB ram vs 128GB. However, first tests showed an massive decrease in loading times. First of all, the bucket contains still the same amount of files: ~500'000. Resulting in 720'000 records. We are aware that this could be improved, but we want to compare the performance: 14 x dc2.large: 15min 4 x ra3.xlplus: 42min For both loads, no other queries were running. Exactly the same COPY statement (STATUPDATE & COMPUPDATE off). CPU utilization shows that all 4 nodes are at max 50%. WLM is set to automatic with the current user executing to priority high. Same performance drop can be measured in bigger join operations etc. Anyone else facing the same experience? Do those ra3 nodes need some time to make the magic (like distinguish between cold & hot etc) happen?

The dataset used in the analysis is of 850MB. It is imported to spice mode from amazon redshift. When I try to use edit dataset option, I get the error that says "Connecting to your data source took too long. Retrying this request may help. Contact technical support for further assistance" I am not able to reload preview for the same even after trying multiple times. I was able to add calculated fields and use edit dataset option for other datasets in the same account. Can anyone please help me resolve this issue?

Hello, I've searched the AWS Quicksight documentation and I've not been able to find what I am looking for. I am working within Quicksight and I've created a table that provides the following information in 4 columns: Model Serial Number, Minimum Hour Meter Reading, Maximum Hour Meter Reading, and Total Hours This table has two filters. First I select a model that I am interested in. Then I selected the timespan that I am interested in. Therefore, the first column of the table (Model Serial Number) lists all of the serial numbers of the model which show up within the timespan that I've selected. The Min and Max hour meter readings are Calculated Fields within my data. I simply use min(hour_meter) and max(hour_meter). The min and max values which are returned are those that exist in the timespan filter. The last column - Total Hours, is another calculated field which simply subtracts the max reading from the min reading to give me the total usage or run time within the time period that I've selected. The table is fine - but what I really want is a summary table that gives me: number of units, median value of Total Hours, 90th percentile of Total Hours, and 99th percentile of Total Hours. This is very similar to the Describe function within Pandas. I tried creating calculated fields in my dataset using the percentileDisc function but I am either not using correctly or it doesn't do what I am looking for it to do. Does anyone know how to create a simple table of the data that I want? Thank you! Best Regards, David

We currently run into the Problem, that some data analytic workloads exceed the 15 minutes timeout of a lambda. It is a multistep process with steps that are parallelizable and some that are not. The main driver for the runtime is a list of regular expressions that need to run sequentially over each of the provided data items, this step is parallelizable (by splitting the input data), but the next step needs to have all the input data. Input and output data is stored in a postresql RDS. Now to the Question: what is the right scalable approach to tackle this problem? Is it: having one 'main lambda' that spawns synced child-lambdas to speedup the parallelizable steps? But the compilation of the regular expressions takes some time, so the spawning of multiple child lambdas will most likely be not feasable. Or is it building a data pipeline using AWS Glue? Or with Steps? Or even building a data stream using Kinesis? I think using EMR is a little bit too big for that use case. Some metrics of the usecase: * size of the regular expression list ~10Mb in RAM * compilation time of all regular expressions around 5-10 sekunds * runtime of a single item going through all steps is about 100ms, but some requests need to process >30k items, therefore we run into the timeout. Any help in choosing the right architecture will be greatly appreciated Alexander

What are the key points to choose one of the following: * Data pipeline, * Step function * Amazon Managed Workflows for Apache Airflow

We've been using DDB streams as an event source for lambda functions serving our analytics pipeline. Additionally, we configured the lambda to send the failed records metadata to a DLQ in case the processing fails after a given number of retries - by doing so, we get the event metadata rather than the actual failing records. Once we fix the underlying issue we currently have no easy way to reprocess the failing records. Looked at [KCL](https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html) but it requires Java to be installed. As we use Python, we find it somewhat strange to install Java just for the sake of getting the impacted DDB records. To me, the data that's sent to the DLQ should include the impacted records. Would appreciate any easy way to fetch the impacted records given the metadata sent in such a case.

MoM (Month over Month) KPI visual in QuickSight compares this month to last month, but using all the dates available for each month. How do you compare MTD over MTD? This is, this month up to date vs. the last month up to the same date, not the full month. For example, you have all order sales with dates for October up to day13th (the most recent), and sales for September whole month. Sales for October from day 1 to 13 totals $30,000. But sales for whole September adds up to $90,000. When comparing MoM this shows that October is worst than September. Need to compare October sales from 1-13 vs September sales from 1-13, as this will show October (up to 13th) = $30,000 but September (up to 13th) = $25,000. So, in an equivalent time period October has more sales than last month. Same case for comparing this year sales (up to the current day) vs last year (up to the equivalent date), so it is really comparable. How can a KPI comparison make this calculation? Or what calculated field has to be created?

I found the [link ](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postgresql-s3-export.html#aws_s3.export_query_to_s3) for Postgres to s3 query export, and I wanted to check if I can use this to perform change data capture from RDS to S3.

From the MWAA [documentation][1] there is a provided template for VPC creation that specifies a CIDR range of /16 and each subnet (public & private) with size of /24. Are there any specific reasons on the default value? If not, what is the minimum size allowed? Can we modify the CIDR range starts with 10.192.0.0 in this template? [1]: https://docs.aws.amazon.com/mwaa/latest/userguide/vpc-create.html#vpc-create-template-components

Does AWS Glue Elastic Views with source as Amazon S3 today?

A customer swaps out table nightly. For example table_1 built last night. Then tonight: - table_1_tmp is built - table_1 renamed to table_1_stale - table_1_tmp renamed to table_1 - table_1_stale is dropped Customer wants know if the new "Automatic Table Optimization" discussed in the [blog][1] is able to recognize the table swaps when gathering data for recommendations. Does it go by table name or a unique table identifier? [1]: https://aws.amazon.com/blogs/big-data/optimizing-tables-in-amazon-redshift-using-automatic-table-optimization/

A customer is asking how to achieve best uptime with HBase on EMR. He is looking for a managed solution, however they are not happy with the fact that the master nodes are deployed in a single AZ (data can be S3 for example, so it will survive AZ outage). I checked the docs [1] and it seems that even with multiple master nodes, they are deployed into a single subnet, hence a single AZ. They are looking for a solution that will automatically failover to another AZ. Do we have any offering on that front? [1] https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-ha-launch.html

Hello All, A customer is currently using AWS ElasticSearch in order to run their primary search function on their e-commerce website. Currently their queries often run for extended periods of time which ends up putting pressure onto the ES instances themselves, which forces them to crash and reboot. This causes their websites functionality to be down until the AWS ElasticSearch service reboots the nodes. They are currently working on reducing the query times and have already been in contact with Premium support. Ideally, I would just like to suggest any alternatives or failover solutions that they could implement until they are able to reduce the violent query requests they receive. I was wondering if the [Cross-Cluster][1] functionality could also be used as a backup option? Or perhaps implementing Route 53 Health Checks as well as another solution. Either way, any feedback or input would be greatly appreciated! [1]: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/cross-cluster-search.html#cross-cluster-search-set-up-connection

Hello- Is it possible to get baseline analytics with Cognito and Pinpoint for a website using Cognito? https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-pinpoint-integration.html The above reads like it should, but I was just told by an Amazon chat support gentleman that sorry the reason I don&#39;t have data is it only works for Mobile Apps or Amplify SDK. Am I really not able to obtain sign up counts, and login numbers? We are using C#/.NET SDK and a couple of our Lambda Triggers use the JS SDK. Any help is appreciated Dan

Is Apache Flink 1.9.0 supported in Amazon EMR?

Is there is a limit on the number of dashboards customers can have when using QuickSight?

I've created Cognito user pool and Pinpoint project both in eu-central-1. When I go to "Cognito -> General Settings -> Analytics -> Add analytics and campaigns" and try to add analytics and user campaigns to user pool, I can't select the Pinpoint project from the drop down list, it just shows "No results found". If I create a Pinpoint project in us-east-1 it appears in the drop down list. This seems to be a Cognito bug, it only scans Pinpoint projects created in us-east for the dropdown list. Where can I report this bug?

A customer is working on a PoC to validate both Athena & Redshift spectrum. Need help with following questions to provide some clarity. 1. Is it fine to have a small Redshift cluster (e.g. 2-node dc2.large cluster) and have almost all data in S3 to be queried through Redshift Spectrum. Is this as recommended approach? What is the minimum recommended size of the cluster to use Redshift Spectrum? 2. What is the correlation between performance and size of data stored in S3. Do we have any numbers that we shall share on what will be the general query performance for specific size of data using both Athena & Redshift Spectrum. 3. Is there any major difference in query syntax between Athena & Redshift Spectrum. They want to understand how easy it's to swap between Athena & Redshift Spectrum. If there is a difference is there any known transition layer to support both of them?

Hi all, My customer mentioned that they have been able to connect and secure Alteryx directly to S3 . Can they connect Power BI directly to S3 or thru Athena or via some other method in order to do similar reporting? I know this is possible via Redshift, but their various analytics requirements may not necessitate the use of a DW. Thanks, Chris

A customer is asking if the new Timestream time series database supports SQL as a query language.

A customer is building a data lake solution, and will leverage RedShift. He wants a tool for "Data Dictionary" (to show relations between data, columns, and catalogue the data in a "dictionary"), and mentioned SQL Doc does this. Do we offer anything similar?

A customer would like to process streaming data in Spark Streaming using **PySpark and would like to output the results on a Kinesis stream**. While PySpark does support reading data from Kinesis (http://spark.apache.org/docs/latest/streaming-kinesis-integration.html), I do not see any support for writing data to Kinesis. DataBricks has some documentation for creating a Kinesis sink for Spark (scala) (https://docs.databricks.com/spark/latest/structured-streaming/kinesis.html), but if I understand the documentation correctly this is based on the `ForeachSink` which is not supported in PySpark (http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach) > The foreach operation allows arbitrary operations to be computed on the output data. As of Spark 2.1, this is available only for Scala and Java. To use this, you will have to implement the interface ForeachWriter (Scala/Java docs), which has methods that get called whenever there is a sequence of rows generated as output after a trigger. Note the following important points. Has anybody come across a way to achieve this?

Client creates EMR cluster as instructed here: http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-glue.html, along with a data catalog in Glue. Client then attempts to access table with code below but receives error, “ops.eventnote” table doesn’t exist. Confirmed table is in catalog. Is there a different way to specify Glue context? ``` public class TestAWSGlueCatalog { private static SparkSession session; private static SQLContext sqlContext; public static void main(final String[] args) throws Exception { try { session = SparkSession.builder().appName("Operation Metrics Transformation") .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") .getOrCreate(); session.sparkContext().hadoopConfiguration() .set("fs.s3a.access.key", "access-key"); session.sparkContext().hadoopConfiguration() .set("fs.s3a.secret.key", "secret-key"); sqlContext = session.sqlContext(); final Dataset<Row> rows = sqlContext .sql("select * from ops.eventnote"); rows.show(); } catch (final Exception e) { e.printStackTrace(); throw e; } } ```