Using Lake Formation tag-based control to automate permissions management for data mesh architectures

Introduction

Organizations that implement data mesh architectures on AWS typically separate data ownership across multiple accounts. Domain teams that represent business units, such as sales, marketing, or finance, operate in dedicated AWS accounts called producer accounts where they create and manage data products. A central governance account maintains security policies and access controls across domains.

A critical challenge emerges in this architecture. Lake Formation requires explicit permissions grants on every database and table. When you create a table in a producer account, Lake Formation doesn't automatically grant access to data users. Without explicit permissions grants, queries against newly created tables fail with "Insufficient Lake Formation permission(s)" errors.

In traditional single-account architecture, you can manage this challenge through manual permissions grants. However, manual permissions don't scale in data mesh architectures with multiple account layers that include the producer account, data user account, and a central governance account. Each permission request requires manual review and approval through the Lake Formation console, AWS Command Line Interface (AWS CLI), or API operation. The manual process creates delays for data users and increases workload for governance teams.

Through customer engagements with enterprise organizations that implement data mesh architectures on AWS, Lake Formation permissions management at scale presents a recurring operational challenge. AWS Enterprise Support collaborates with customers that are building data mesh architectures to implement scalable permissions management solutions.

During architecture reviews and operational assessments, organizations consistently ask the following questions:

• “How can we enable domain teams to create data products without waiting for manual permissions grants?”

• “How do we maintain centralized governance policies while distributing data ownership across multiple accounts?”

• “Can Lake Formation permissions management scale to hundreds of tables per day across dozens of accounts?”

This article explains how to use LF-TBAC to automate permissions management across producer accounts.

Solution overview

The solution consists of the following components to automate permissions grants.

Cross-account role configuration

The foundation establishes trust relationships between producer accounts and the central governance account. In each producer account, you configure the following three AWS Identity and Access Management (IAM) roles.

Service-linked role Applications and data pipelines use the service-linked role to create databases and tables. This role receives `SUPER_USER permission on the Amazon S3 Tables catalog.

Data engineer role The domain teams that query their own data use the data engineer role in the producer account to query tables through Amazon Athena. This role receives tag-based permissions that automatically grant access to tables that match specific LF-Tag expressions. The same mechanism supports data user roles in separate accounts for cross-account access.

Lake Formation administrator role The central governance account assumes the Lake Formation administrator role. This role has full Lake Formation permissions to create LF-Tags, apply LF-Tags to resources, and grant permissions to user roles. The role's trust policy allows the AWS Lambda execution role in the central governance account to use AWS Security Token Service (STS) to assume the role.

In the central governance account, you configure only one IAM role.

Cross-account management role The cross-account management role is a Lambda execution role that can assume the Lake Formation administrator role in a producer account. This role allows centralized governance logic to execute across producer accounts.

LF-Tag based permission policies

Instead of naming individual resources in permissions grants, LF-Tag based permissions policies grant permissions to resources that match LF-Tag expressions. Common examples of LT-Tag keys include Environment with values such as Production, Staging, and Development, and Department with values such as Sales, Marketing, and Analytics.

An LF-Tag based permissions policy specifies the following values:

• Principal: The IAM role that's receiving permissions, for example, data-engineer-role.

• Resource Type: DATABASE or TABLE.

• Tag Expression: The combination of LF-Tags that resources must have, for example Environment=Production and Department=sales.

• Permissions: The Lake Formation permissions to grant, for example SELECT and DESCRIBE.

For example, an LF-Tag based policy might state: "Grant SELECT and DESCRIBE permissions on any TABLE tagged with `Environment=Production` AND `Department=sales` to the data-engineer-role."

Automated resource tagging and permissions grants

The solution uses the following Lambda functions that work together to provide automation. The first function automatically tags newly created tables. The second function automatically grants permissions when a governance team creates a named LF-Tag expression.

Automatic tagging when a service-linked role creates tables

When a service-linked role creates an AWS Glue database or S3 table in a producer account, the API operation is logged as a management event in AWS CloudTrail. CloudTrail forwards the event to Amazon EventBridge in the producer account with the AWS API Call via CloudTrail detail-type. An EventBridge rule in the producer account filters for events and triggers the lf_auto_tagger Lambda function in the central governance account. The Lambda function assumes the Lake Formation administrator role in the producer account and applies LF-Tags based on the namespace pattern. For example, a table in namespace sales_prod automatically receives the Environment=Production and Department=sales tags. A table in namespace analytics_dev receives Environment=Development and Department=analytics.

Because CloudTrail events can take several minutes to propagate to EventBridge, tagging typically completes within 5 minutes of table creation.

Note: This solution applies to S3 tables that the role creates through Athena SQL CREATE TABLE statements when it logs events with the glue.amazonaws.com event source in CloudTrail. You configure the EventBridge rule to detect the AWS Glue API events. S3 tables events aren't currently delivered to EventBridge. To support CLI or SDK table creation, you must further configure the rule.

Automatic permissions grants when you create expressions

Named LF-Tag expressions let governance teams define reusable tag combinations with descriptive names. When a governance team member identifies a new access pattern, they create a named LF-Tag expression in the producer account. For example, they might enable the sales analytics team to query production sales data. CloudTrail logs the CreateLFTagExpression API operation and forwards it to EventBridge in the producer account. An EventBridge rule filters for CreateLFTagExpression events and triggers the lf_expression_grant Lambda function in the central governance account.

The lf_expression_grant Lambda function uses AWS STS to assume the Lake Formation administrator role in the producer account. After the function assumes the administrator role, the Lambda function reads the expression details from the event payload. Based on the Department tag value in the expression, the function determines the data user role that requires permissions. For example, if the expression contains Department=sales, then the Lambda function grants permissions to the sales-analyst-role role.

Automatic permission evaluation

After the Lambda function applies LF-Tags to a newly created table, Lake Formation evaluates the tag-based permissions policies in the producer account. If the table's tags match a policy expression, then Lake Formation automatically grants the specified permissions to the principals in the policy. This evaluation happens immediately after the function applies tags, with no additional API operations or manual intervention required.

Prerequisites

You must meet the following requirements:

• You have at least one producer account for domains and a central governance account.
  Note: You can use more than two accounts.

• You activated CloudTrail in the producer account.

• You activated Amazon S3 Tables integration in the producer account.

• You have IAM permissions to create Lambda functions, IAM roles, EventBridge rules, and Lake Formation resources.

Solution implementation

To implement the solution, you can use AWS CLI commands that create the required resources in both the producer and central governance accounts. For more information about implementation, see the sample-lake-formation-permission-management repository on the GitHub website. The repository includes Lambda function code, EventBridge rule patterns, and IAM policy templates that you can adapt to your environment.

Conclusion

LF-TBAC provides a scalable solution to manage permissions across data mesh architectures. When you grant permissions to tag expressions instead of individual resources, you no longer need to manually grant permissions. LF-TBAC lets you maintain centralized governance across multiple producer accounts.

The innovation combines tag-based access control with event triggered automation. Governance teams can create LF-Tag expressions to automatically grant permissions to data user roles. When you create tables, they're automatically tagged based on their namespace. Lake Formation then grants permissions based on the tags. This approach scales to hundreds of tables each day across dozens of producer accounts without additional governance capacity.

As your data mesh architecture evolves, you can extend tag-based access control to solve additional governance requirements:

• Data classification: Add LF-Tags for data sensitivity levels, including Public, Internal, Confidential, and Restricted. Automatically enforce appropriate access controls based on classification.

• Compliance reporting: Query Lake Formation permissions by tag to generate compliance reports that show the roles that have access to specific categories of data across producer accounts.

AWS Enterprise Support can help organizations implement LF-TBAC in data mesh architecture. AWS Support Engineers and Technical Account Managers (TAMs) provide guidance on multi-account strategies and share best practices for Lake Formation governance. They also offer hands-on support to optimize AWS data services. To implement this pattern in your environment, contact your TAM or see AWS Support for available plans and offerings.

About the authors

Sana Aneja

Sana is a Sr. TAM at AWS located in Ney Jersey. She focuses on supporting Independent Software Vendor (ISV) customers, and guides them to migrate, optimize, and navigate their journey in the AWS Cloud. When not at work, Sana likes to spend time with family, listen to music, travel, and explore new places.

Benjamin Herrera

Benjamin is a Cloud Support Engineer at AWS. He assists clients with problem solving, design, optimization, and implementation of Data Engineering solutions aligned with industry’s best practices.