Questions tagged with AWS Lake Formation

I received below error in one of Blueprint jobs : An error occurred while calling o452.getDynamicFrame. ERROR: permission denied for table tmp_company I found below errors in the error log of the job: py4j.protocol.Py4JJavaError: An error occurred while calling o452.getDynamicFrame. org.postgresql.util.PSQLException: ERROR: permission denied for table tmp_company I already granted super permission to the lake formation role on the lake formation database! also lake formation admin user has all sufficient permissions!! The error in one job only and the other jobs finished successfully!! Any clue??

_temp lake formation blueprint pipeline tables appears to IAM user in Athena editor, although I didn't give this user permission on them below the policy granted to this IAM user,also in lake formation permsissions ,I didnt give this user any permissions on _temp tables: { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1652364721496", "Action": [ "athena:BatchGetNamedQuery", "athena:BatchGetQueryExecution", "athena:GetDataCatalog", "athena:GetDatabase", "athena:GetNamedQuery", "athena:GetPreparedStatement", "athena:GetQueryExecution", "athena:GetQueryResults", "athena:GetQueryResultsStream", "athena:GetTableMetadata", "athena:GetWorkGroup", "athena:ListDataCatalogs", "athena:ListDatabases", "athena:ListEngineVersions", "athena:ListNamedQueries", "athena:ListPreparedStatements", "athena:ListQueryExecutions", "athena:ListTableMetadata", "athena:ListTagsForResource", "athena:ListWorkGroups", "athena:StartQueryExecution", "athena:StopQueryExecution" ], "Effect": "Allow", "Resource": "*" }, { "Effect": "Allow", "Action": [ "glue:GetDatabase", "glue:GetDatabases", "glue:BatchDeleteTable", "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions", "glue:BatchGetPartition" ], "Resource": [ "*" ] }, { "Sid": "Stmt1652365282568", "Action": "s3:*", "Effect": "Allow", "Resource": [ "arn:aws:s3:::queryresults-all", "arn:aws:s3:::queryresults-all/*" ] }, { "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": [ "*" ] } ] }

I'm looking into Cloudtrail Lake and need tips/help on regarding queries. The given Query returns records as expected, however I need to queries where todays date is taken into consideratio,. This without having to rewrite the eventTimes dates every time. Is there a function like **now()**, **current_date()** etc? I see that supported date and time functions are using Presto 0.266 syntax, but are not able to find a solution to this issue https://docs.aws.amazon.com/awscloudtrail/latest/userguide/query-limitations.html ``` SELECT awsRegion, recipientAccountId, count(* ) as numRec FROM $EDS_ID WHERE eventTime >= '2022-05-01 00:00:00' #I want something like now() - 7 days and eventSource = 'states.amazonaws.com' ```

When trying to access a specific database (e.g *default*, the default HIVE database) through the console (e.g, using the URL `https://us-east-1.console.aws.amazon.com/lakeformation/home?region=us-east-1#database-details/default?catalogId=$AWS::AccountID`), an **Unknown Error** is being thrown and shown in the console (screenshot sadly not attached because it's unsupported on this platform). Since there is no additional information with the error, it's impossible (for us) to know how to remediate the issue. As for IAM permissions, the user trying to access the view has this following policy attached: ``` { "Version": "2012-10-17", "Statement": [ { "Action": "lakeformation:*", "Resource": [ "*" ], "Effect": "Allow" } ] } ```

I have created a database using AWS Lake Formation, and populated it with two tables created using Glue crawlers. The tables seem to be created correctly (all of the columns have been properly mapped out by the crawlers). However, then I try to query them using AWS Athena, I am getting the following error: `HIVE_UNKNOWN_ERROR: com.amazonaws.services.lakeformation.model.InvalidInputException: Unsupported vendor for Glue supported principal` I assume this has to do with the permissions associated with Lake Formation, but I have given the IAM User all possible permissions I could think of in the Lake Formation Console Permissions section. Does anyone know what the problem could be here?

After running a simple select query in Athena, I get an error - **HIVE_UNKNOWN_ERROR: Error creating an instance of com.facebook.presto.hive.lakeformation.CachingLakeFormationCredentialsProvider**. Athena query is supposed to run agains a LF database within the same S3 bucket (data source and query destination are split into two folders inside the bucket). Any idea what could be the issue?

Hello. Development Endpoint only supports Glue version <= 1.0. With upgraded Glue Versions, will Glue Version 1.0 eventually be deprecated? I saw the following post related to development under Glue version 2.0 and 3.0, is the intention that we move development to Glue Studio Notebook and Glue Interactive Sessions? Will Glue Development Endpoint eventually go away as well? Thank you!

How should I write the string to the PartitionPredicate attribute of the get_table_objects() function? Thank you

Hi, I have a database with around 40 tables. However, some end users don't need to see all tables in the database. I'm using Lake Formation Tagging and know that if a tag is added to the database that the tag is then inherited by all the tables. I also found by going into the table that the inherited tags can't be removed. I tried adding a tag to just the table level and granting permissions but the database won't appear as the tag is only at the table level. Is there a way to hide certain tables in a database by using Lake Formation Tagging?

I have implemented LakeFormation on my data bucket. I have a step function in which one step consists of running a GlueJob that reads and writes to the data catalog. I have upgraded my DataLake permissions as reported [here][1]. The Service Role that runs my Step Function has a root-type policy (granted just for debugging this issue): ```yaml Statement: - Effect: "Allow" Action: - "*" Resource: - "*" ``` On lake formation the service role has: - Administrator Rights - Database Creation rights (and Grantable) - Data Location access to the entire bucket (and Grantable) - Super rights on read and write Database (and Grantable) - Super rights on ALL tables within above Databases (and Grantable). The bucket is not encrypted. But, somehow, its access to the tables is denied with the error: ``` (AccessDeniedException) when calling the GetTable operation: Insufficient Lake Formation permission(s) on table ``` What's really strange is that the Glue Job succeeds when writing to some tables, and fails on others. And there is no real substantial difference across tables: all of them are under the same S3 prefix, parquet files, partitioned on the same key. Given the abundance of permissions granted, I am really clueless about what is causing the error. Please, send help. [1]: https://docs.aws.amazon.com/lake-formation/latest/dg/upgrade-glue-lake-formation.html

Hi! hope you can help me with a concern about an error with awswrangler. this is the case: i have 2 aws accounts, AccountA and AccountB, both with lakeformation enabled, i have a set of databases in AccA and another set in AccB, so we share AccountB databases to AccountA through lakeformation so we can query their Db/tables with Athena in AccountA. i am trying to automate a sql with python, so i'm using awswrangler to achieve this, but i'm getting a not very specific error when in run the query in python. when i run "select * from DatabaseAccB.Table" get this error "HIVE_METASTORE_ERROR: Table is missing storage descriptor" what could be the cause? i tried with boto3.Athena session and same result. this may should help, when i query select * from DatabaseAccB.Table with my user, this runs fine. but when i try to do it with lambda or glue job, fails with error mentioned before. PD: AccountA has only select/describe permission on tables in AccountB. Can show some code if you need. Thanks!

## Problem I want to know, understand and correct my knowledge, approach on, Setting up an Data Ingestion pipeline, which collects "events" or "data" from any possible external application sources (applications of 3rd party) The rate of ingestion can be about 5000 (5K) events per day (normal) on peak it can go slightly more 20K ### Approach I been thinking about I am planning to setup AWS Lambda endpoint to which external systems can post(HTTP POST) the data, which then can load into OpenSearch to form a Data Lake The ingestion pipeline operates between Lambda and OpenSearch, to perform - Parsing of data - Fetch more data if needed, by making API calls - Process, transform, enrich - Post to OpenSearch Indices as per indices I have been googling and exploring on AWS but so far can't find any thing which can validate above. Hence request you experts to comment, suggest and direct me to a practical solution

Hello, I've been able to create tables and various views in Athena console after running a Glue crawler against an S3 bucket as a datasource. When running a Preview View from Athena on a view generated from the Glue crawler, I am getting an error with some reference to a Hive split on a CSV file in our S3 bucket similar to the following: `Error opening Hive split s3://bucket_name/Folder1/folder/File.csv (offset=33554432, length=33554432) using org.apache.hadoop.mapred.TextInputFormat: Permission denied on S3 path: s3://bucket_name/Folder1/folder/File.csv This query ran against the "athena_database_name" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: e9d6e300-c82d-40d0-b9a0-879e97f72db6` It is similar to the issue outlined in : https://stackoverflow.com/questions/68859517/amazon-athena-error-opening-hive-split-s3-path-and-access-denied I do not think that this is a permission related to my IAM user but I am unsure what the Hive split is related to. Is this service related? If anyone has a solution for this, please let me know. Thank you in advance. - M

Doing a POC with LF Governed table for a small dimension. I'd like to purge and reload the table each time it runs. I have tried to include a purge_table command within a governed table transaction but I receive the below error. I've tried adding a transactionId param to the purge_table command directly or within the options dict and neither has worked. Any ideas? Or anyone found a good way to truncate and reload a governed table? "An error occurred while calling o119.purgeTable. Specify Either Transaction Id or Query AsOf Time"

I have a large dataset (table) with >1e9 records (rows) in Glue. The tables are partitioned by column A, which is a n-letters subtring of column B. For example: | A (partition key) | B | ... | | --- | --- | --- | | abc | abc123... | ... | | abc | abc123... | ... | | abc | abc456... | ... | | abc | abc456... | ... | | abc | abc456... | ... | | abc | abc789... | ... | | abc | abc789... | ... | | ... | ... | ... | | xyz | xyz123... | ... | | xyz | xyz123... | ... | | xyz | xyz123... | ... | | xyz | xyz456... | ... | | xyz | xyz456... | ... | | xyz | xyz456... | ... | | xyz | xyz789... | ... | | xyz | xyz789... | ... | There are >1e6 possible different values of column B and correspondingly significantly less for column A (maybe 1e3). Now I need to group records/rows by column B and the assumption is that it could be advantageous if the table was partitioned by column A, as it would be sufficient to load dataframes from single partitions for grouping instead of running the operation on the entire table. (Partitioning by column B would lead to unreasonably large numbers partitions.) Is my assumption right? How would I tell my Glue job the link between column A and B and profit from the partitioning? Alternatively I could handle the 1e3 dataframes (one for each partition) separately in my Glue job and merge them lateron. But this looks a bit complicated to me. This question is a follow-up question to https://repost.aws/questions/QUwxdl4EwTQcKBuL8MKCU0EQ/are-partitions-advantageous-for-groupby-operations-in-glue-jobs.

Hi, I'm relatively new to AWS glue and was having trouble in the following transformation codes: ``` DataSource4 = glueContext.create_dynamic_frame.from_catalog(database = "beta", table_name = "[table_name]", transformation_ctx = "DataSource4") Transform9 = ApplyMapping.apply(frame = DataSource4, mappings = [("project_unique_id", "int", "(src) project_unique_id", "int")], transformation_ctx = "Transform9") Transform9DF = Transform9.toDF() Transform3DF = Transform3.toDF() Transform12 = DynamicFrame.fromDF(Transform3DF.join(Transform9DF, (Transform3DF['project_unique_id'] == Transform9DF['previous_project_id']), "leftanti"), glueContext, "Transform12") ``` The job is failing with error : raise AnalysisException:(s.split(': ',1)[1], stackTrace) 'Cannot resolve column name "previous_project_id" among ((src) project_unique_id);' on checking the tables, both columns "project_unique_id" and "previous_project_id" are filled with NULL values, could that be the reason for the above error?

I have an errror when I try to login with an IDP (Auth0) and EMR integrated with Lake formation. I'm following the workshop [Lake formation & EMR integration](https://catalog.us-east-1.prod.workshops.aws/workshops/78572df7-d2ee-4f78-b698-7cafdb55135d/en-US/emr-integration ) I have configured an Auth0 account, aws IDP, EMR cluster (aws service) and data lake permissions with (idp users). But I have an error when I do the login with [EMR Zeppelin] (https://EMRMasterNodeDNS:8442/gateway/default/zeppelin/). I do the login with Auth0 and EMR but I can't do it with lakeformation. This is the error that I had on EMR proxy agent: `Caused by: java.lang.NullPointerException at org.apache.knox.gateway.util.SamlUtils.getSamlAwsRoleAttributeValues(SamlUtils.java:149) at org.apache.knox.gateway.pac4j.aws.AwsLakeFormationSamlImpl.getAwsCredentials(AwsLakeFormationSamlImpl.java:106) at org.apache.knox.gateway.pac4j.aws.AwsSamlHandler.processSamlResponse(AwsSamlHandler.java:78) at org.apache.knox.gateway.pac4j.filter.Pac4jDispatcherFilter.doFilter(Pac4jDispatcherFilter.java:234) at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:372) at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:272) at org.apache.knox.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:30) at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)` I think that I need to do the step 6 on the documentation [amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-lf-federation.html). But I don't know were I have to do this configuration. Any help? Thank you

I am trying to execute in Athena a simple query on a hive table stored in s3. --- ***SELECT * FROM "database"."table" limit 10;*** --- but i get the following error. **HIVE_UNKNOWN_ERROR: Duplicate key string** These are the characteristics of the table: --- **Classification**: parquet --- **Input format:** org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat --- **Serde serialization lib:** org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe --- **EXTERNAL: **true --- **has_encrypted_data:** false --- **compressionType:** none --- **typeOfData:** file Any ideas ???

I'm using a Table resource to create a Glue Table like so: ``` this.glueTable = new Table(scope, `${props.glueTableName}`, { database: this.glueDatabase, tableName: props.glueTableName, bucket: props.bucket, partitionKeys: props.partitionKeys, dataFormat: props.dataFormat, columns: props.columns }); ``` I need to explicitly set the `compressionType` of this table to `"none"`. I see there is a way to do it via the console, where we go to `Edit Table`, and then fill in the compressionType under `Table properties`. Can anyone point me to the prop to use if I want to do it using cdk?

I am trying to integrate EMR with Lake Formation. The EMR cluster has been created with Default Roles for both EMR and EC2 instances. In addition to the default permissions created in those Roles, I have also provided full access on Lake formation and Glue to the Default roles . I selected the default Service linked Role for all s3 buckets registered with this Lake formation. After creating Jupyter notebook, which uses the above created cluster, I tried running spark.sql ("show databases") and spark.sql("use <database>") to get the following error **An error was encountered: org.apache.spark.sql.catalyst.analysis.AccessControlException: Unable to verify existence of default database: com.amazonaws.services.glue.model.AccessDeniedException: Insufficient Lake Formation permission(s) on default (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: .....; Proxy: null);** I think i have tried everything with permissions and not able to understand the root cause for the error. Appreciate any pointers which would help understand / resolve the error. Many Thanks.

I have Delta Lake tables (using Symlink text input format) catalogued in Glue, stored in a S3 bucket, with all its resources tagged with LakeFormation Tags (for tag-based governance). The problem is that, although the users can see the database, tables, and metadata within Athena's catalogue, they cannot perform queries against the specific tables because of "Permission denied on S3 path" errors. LakeFormation has the data location registered for the datalake bucket, with AWSServiceRoleForLakeFormationDataAccess role. And this role has IAM permissions automatically added to the resources: ``` LakeFormationDataAccessServiceRolePolicy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets" ], "Resource": [ "arn:aws:s3:::*" ] } ] } ``` and ``` LakeFormationDataAccessPolicyForS3 { "Version": "2012-10-17", "Statement": [ { "Sid": "LakeFormationDataAccessPermissionsForS3", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::MYBUCKET/*" ] }, { "Sid": "LakeFormationDataAccessPermissionsForS3ListBucket", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::MYBUCKET" ] } ] } ``` I have also tried registering the data location with a role with Admin permissions (Action "*" and Resources "*"), but even so the same error is thrown. Looking through the CloudTrail logs, I found that LakeFormation passes custom policies to the role when running AssumeRole: ``` "policy": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Action\": [\"s3:GetObject\"],\n \"Effect\": \"Allow\",\n \"Resource\": [\"arn:aws:s3:::MYBUCKET\",\n \"Condition\": {\"ForAnyValue:StringLike\":{\"s3:prefix\":[\"MYTABLE/_symlink_format_manifest\",\"MYTABLE/_symlink_format_manifest/*\"]}}\n },\n {\n \"Action\": [\"kms:Decrypt\"],\n \"Effect\": \"Allow\",\n \"Resource\": [\"*\"],\n \"Condition\": {\"StringEquals\":{\"kms:ViaService\":[\"s3.us-east-2.amazonaws.com\"]}}\n } ]\n}" } ``` This seems like a malformatted json string that is being passed to the assumed role. Can this be causing the errors I'm having? And does anyone have had this issue before? PS: I have manually removed ACL control over the S3 bucket and objects. Still same behavior. The error is not shown if I remove the data location, and Athena ignores Lake Formation.

I have an AWS Data Lake that is ready to be used at the moment. My use case for the Data Lake is to be able to, ingest data from different API connectors (coming from other data vendors and service providers). There are at least three different services such as CRM tools, data vendors and etc which I need to ingest data into AWS Data Lake. All these services provide an API which can be used to connect to the data lake. The challenge is however to ensure that the process of ingesting/transforming and loading (ETL), must be automated and doesn't need manual interventions. Which means once connected, with every new data point that the vendors provide, it must automatically be loaded into the data lake. My question is, what AWS services do I need to use to accomplish this ? (1) Building an API connector from different data providers to Data lake ? (2) Be able to do some ETL (or transform) the data before loading it into the Data Lake. Potentially this ETL service can sit between the API and the Data lake. Thanks,

I ran an Lake Formation BluePrint and result was as follows: | method to run | result | | --- | --- | | by hand | COMPLETED | | scheduled | IMPORT FAILED(NoCredentialsError: Unable to locate credentials) | The error log of ETL Job for workflow said "botocore.exceptions.NoCredentialsError: Unable to locate credentials". Why scheduled job failed? Is there any solution? The BluePrint's detail as follows: * Blueprint type: Database snapshot * Import target's data format: Parquet * Import frequency: Monthly * Day of the month: 1 * start time: 10:00 * Policies which the IAM role(use by this Blueprint) has: * AWS managed policy * AWSGlueServiceRole * inline policies ``` 1st { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "lakeformation:GetDataAccess", "Resource": "*" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:ListBucket", "s3:DeleteObject" ], "Resource": [ "{{DATALAKE BUCKET ARN}}/*", "{{DATALAKE BUCKET ARN}}" ] } ] } 2nd { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "{{Glue Connect to RDS ARN}}" } ] } ```

I've granted data permissions to a user through the use of LF-Tags (instead of Named Data Catalog resources). It works fine. How/Where can I view and revoke these permissions ? I'm thinking about some kind of display where I can see UserA <-> (Tag B, Value C). I'm an admin of the lake formation. In console it only seems possible to view data permissions only when granted using Named Data Catalog resources. By going to Menu -> Permissions -> Datalake permissions I can't find a way to filter by lf-tags. I've searched documentation but only found how to view permissions granted using Named Data Catalog resources (https://docs.aws.amazon.com/lake-formation/latest/dg/viewing-permissions.html)

I have configured a data lake using Lake Formation and Glue to populate and upsert the database and tables. This upsert includes assigning LF Tags to each table and column (for LF Tag-based governance). I'm trying to query the table using Athena, with an administrator level user account, plus Lake formation permissions on LF Tags, datalake administrator, database creator, and permissions to the s3 data location resource. However, when I query a ``` select count(*) from table ``` on Athena, I'm thrown this error: ``` Insufficient Lake Formation permission(s): Illegal permission combination (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 6280c520-8891-4461-8a7e-2e6bbb5a2fec; Proxy: null) This query ran against the "raw" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 2f3f0e09-0cda-4e07-a390-80689a66c022 ``` I've tried adding all possible permissions to access the data, but still can't run a query using Athena. Can someone spot what I'm missing out? Thanks! Denis

Service Name: AWS Glue Situation: When I search for table in Glue Data Catalog by input a part of table name in web UI search box, the result is not showed correctly. For example; | Name | Database | Location | | --- | --- | --- | | mqtt_aapl | mydb | mylocation | | mqtt_goog | mydb| mylocation | | mqtt_tesla | mydb | mylocation | | kafka_stock | myrealtime | mylocation| And I typed "mqtt" , it returns inconsistent result 1) Show only top of list e.g. only "mqtt_aapl" 2) Show all tables in catalog including "kafka_stock" 3) Show nothing Expectation: 1) When I typed "mqtt", it should return 3 items of list "mqtt_aapl", "mqtt_goog" and "mqtt_tesla" 2) When I typed "stock" it should return only "kafka_stock" Please help to figure out how I can deal with this situation.

I have gotten the following error when trying to assign a administrator to LakeFormation. The error seems to be internal to AWS: com.amazon.coral.service.InternalFailure This happened at 20:40 GMT-5 zone time

Hi, I am signed in with a userId which has IAM Administrator access and DataLake Administrator access. The userId also has permissions to table columns set up in Lake Formation. However when creating a Data Filter on a table using console, I get following error message. > Error creating data cell filter: data-filter-1. Not authorized to create filter. Any suggestions to resolve the issue will be greatly appreciated.

Are governed tables insert/append only? Is it possible to update data already in the table? https://aws.amazon.com/blogs/big-data/part-3-effective-data-lakes-using-aws-lake-formation-part-3-using-acid-transactions-on-governed-tables/ has a good example of inserting data, but it doesn't look possible to define a primary key to be able to dedupilcate data if you did another insert on the same row or a way to do an update. Thanks

Is it possible to assign data permissions to an AWS SSO user by using their federated user arn? If so, please can you advise of the format. It's possible to assign an AWS SSO created role (permission set) data permission in Lake Formation as it is available as a drop down from IAM, but the user is not available in that list. The top right drop down gives is labelled as a federated user in the pattern of: <IAM_ROLE_FROM_SSO>/<USER> but baking that into an arn like: arn:aws:iam:ACCOUNTID:federated-user/<IAM_ROLE_FROM_SSO>/<USER> is unfortunately not recognised by Lake Formation ui. Could anyone provide any pointers how we could give single users data permissions when they are AWS SSO users please?

We have a BI feature where a web app which uses non-aws authentication queries Athena for data which is hive partitioned by customer. Currently any BI query gets modified to filter data down to just the partition that the user has access to, but we would like to migrate to using the new row level security with lake formation. Since the backend is currently running under a single IAM role with access to all data, what is the best way to go forward with this? Would we need to have a process that creates a LF filter for each customer/table combo, then create a IAM role for each customer and then grant the LF filters to each role and use STS to assume that customer role when invoking athena? Is there an easier way with STS that can handle the filters and grants a little more dynamically without having to statically define each filter?

``` 2021-06-23 13:26:36,962 INFO \[Thread-4] util.FileSchemeWrapper (FileSchemeWrapper.scala:executeWithQualifiedScheme(76)): Should Bypass LF Cred Fetching: false ``` Job keeps running and has this log flushed the log stream. And I already made sure that the rule which run this job is registered in Data Permission of Lake Formation. Not sure what I'm missing here. Thanks!

When defining blueprints in AWS Lake Formation, can we specify a particular snapshot? Does Lake Formation always uses the recent snapshot by default?

Hi, AWS Glue Crawlers with CSV and XML Classifiers and works well with files encoded in UTF-8 but not with file encoded in UTF-16. Public documentation does not clarify this point: - Do Glue crawler and classifier support UTF-16? - Is there please an available documentation on supported encodings with Glue crawlers and classifiers? Best regards, Nicolas.

Hi, I am trying to catalog some data from S3 as part of Lake Formation setup. I am using a crawler to read a CSV file from an S3 bucket. The crawler is set up to use the default AWSGlueServiceRoleDefault IAM role, which has full access to S3. However, the job keeps failing with the following error: ERROR : Insufficient Lake Formation permission(s) on s3://<<bucket-name>>/<<file-name>> (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 6ca688e0-1b6d-11ea-9499-cb57d11186a3). For more information, see Setting up IAM Permissions in the Developer Guide (http://docs.aws.amazon.com/glue/latest/dg/getting-started-access.html) As per the documentation, using AWSGlueServiceRoleDefault should be sufficient to execute the crawler. Any ideas why this is failing?

Hi We are currently investigating Lake Formation and it looks promising. There are a few things which we can't figure out to do. Either because we don't know how or because they are not yet developed for Lake Formation 1) User defined groups We cannot see our IAM groups in the grant permission to data access. We have a large group of analysts divided into different divisions. We would like to give a specific access to each division, but not having the overhead of doing this for each analyst. I can only find the "everyone" group, which doesn't suit our needs 2) User defined blueprints Currently the number of blueprints is limited to databases and cloudtrail-logs. It would be a nice feature to be able to create your own blueprints in order to recreate userdefined datalake ingestions. I don't know if anyone has some workarounds for these issues or there is a wish-list somewhere to propose new features Best and Thanks