Managing FSxN S3 Access Points in Centralized VPC Endpoint Architectures

Overview

When using Amazon FSx for NetApp ONTAP (FSxN) S3 Access Points with NetworkOrigin: VPC in environments with centralized VPC endpoints and Route 53 Resolver forwarding, you may get AccessDenied even though all IAM and resource policies appear correct. This article explains why and how to fix it.

Symptoms

An error occurred (AccessDenied) when calling the ListObjectsV2 operation:
User: arn:aws:sts::111111111111:assumed-role/MyEC2Role/i-0abc123def456789
is not authorized to perform: s3:ListBucket on resource:
"arn:aws:s3:us-east-1:111111111111:accesspoint/my-fsxn-s3-accesspoint"
with an explicit deny in a resource-based policy

This occurs when:

• The S3 Access Point has NetworkOrigin: VPC pointing to the local workload VPC
• The environment uses centralized VPC endpoints with Route 53 Resolver forwarding
• IAM role permissions, access point policy, SCPs/RCPs, and VPC endpoint policies are all correct

Root Cause

In hub-and-spoke architectures, a Route 53 Resolver forwarding rule (e.g., for *.amazonaws.com or *.<region>.amazonaws.com) redirects DNS from spoke VPCs to centralized resolvers in a hub account. These resolvers return the private IPs of a shared S3 Interface Endpoint in the hub VPC.

When the FSxN S3 Access Point is created with NetworkOrigin: VPC referencing the local workload VPC (VPC-A), but traffic actually flows through the shared Interface Endpoint in the hub VPC (VPC-B), S3 sees a VPC mismatch:

Traffic Flow (see architecture diagram)

1. ① DNS query - The EC2 instance resolves the S3 Access Point DNS. A Route 53 Resolver forwarding rule intercepts the query and sends it to centralized DNS resolvers in the hub account.
2. ② Resolver → Interface Endpoint - The centralized resolver returns private IPs of the shared S3 Interface Endpoint in the hub VPC (VPC-B).
3. ③ Private IP resolution - The EC2 instance receives private IPs (10.x.x.x) belonging to the hub VPC's Interface Endpoint.
4. ④ S3 traffic routed through hub - The EC2 instance sends the S3 API call to these private IPs, so the request enters S3 through the Interface Endpoint in VPC-B (hub), not VPC-A (workload).
5. ⑤ S3 VPC-origin evaluation - S3 receives the request and checks the access point's NetworkOrigin: VPC constraint. It sees the request arrived from VPC-B.
6. ⑥ AccessDenied - VPC-B ≠ VPC-A → S3 denies the request with "explicit deny in a resource-based policy." The error is returned to the workload account.
7. ⑦ CloudTrail - The denied event is logged in the workload (spoke) account's CloudTrail, with vpcEndpointId and vpcEndpointAccountId pointing to the hub account's endpoint - the key diagnostic evidence.

How to Identify This Issue Using CloudTrail

CloudTrail is the fastest way to confirm this pattern. Look at the S3 event for the denied request and check two critical fields:

{
    "eventSource": "s3.amazonaws.com",
    "eventName": "ListObjects",
    "sourceIPAddress": "10.x.x.x",
    "userIdentity": {
        "accountId": "111111111111",
        "arn": "arn:aws:sts::111111111111:assumed-role/MyEC2Role/i-0abc123def456789"
    },
    "vpcEndpointId": "vpce-0abc1234def56789",
    "vpcEndpointAccountId": "999999999999",
    "recipientAccountId": "111111111111"
}

Key indicators:

CloudTrail Field	What to Look For
vpcEndpointId	If this is an Interface Endpoint you don't recognize from the workload account, traffic is flowing through a centralized endpoint
vpcEndpointAccountId	If this is a different account than the workload account, traffic is crossing account boundaries through a shared endpoint
sourceIPAddress	A private IP (10.x.x.x, 172.x.x.x) confirms traffic is routed through VPC infrastructure, not the public internet
recipientAccountId	This is the account where the event was logged - typically the workload account

Important Note: When vpcEndpointAccountId ≠ recipientAccountId, your S3 traffic is flowing through a VPC endpoint owned by another account. If your access point has NetworkOrigin: VPC pointing to the local workload VPC, but traffic arrives through a different VPC in a different account - that's the mismatch causing the deny.

Diagnosis Steps

1. Confirm the Access Point Configuration

aws s3control get-access-point \
  --account-id 111111111111 \
  --name my-fsxn-s3-accesspoint \
  --region us-east-1

Check NetworkOrigin: VPC and note the VpcConfiguration.VpcId.

2. Check DNS Resolution from the EC2 Instance

nslookup <access-point-alias>.s3-accesspoint.us-east-1.amazonaws.com

If this returns private IPs and there's no S3 Interface Endpoint in the local account/VPC, DNS is being forwarded to a centralized endpoint.

3. Confirm Route 53 Resolver Forwarding

aws route53resolver list-resolver-rules --region us-east-1

Look for forwarding rules targeting amazonaws.com or <region>.amazonaws.com that point to IPs in a hub VPC.

4. Check CloudTrail

Look for the denied S3 event and compare vpcEndpointAccountId against your workload account ID, as described above.

Resolution

Option 1: Use the Central Hub VPC (Recommended)

Recreate the FSxN S3 Access Point with NetworkOrigin: VPC using the central hub VPC ID (where the shared Interface Endpoint resides) instead of the local workload VPC ID. This aligns the VPC-origin constraint with the actual traffic path and works within the existing centralized networking architecture.

Option 2: Use NetworkOrigin: Internet

If VPC-origin restriction isn't required, recreate the access point with NetworkOrigin: Internet. This removes the VPC constraint and allows traffic through any path.

Option 3: Create a Local S3 Interface Endpoint

Create an S3 Interface Endpoint directly in the workload VPC with private DNS enabled. This overrides the centralized DNS forwarding so traffic stays within the local VPC. Note: Coordinate with your networking team - this may conflict with the centralized architecture.

How AWS TAMs Help with Complex Cross-Service Issues

This issue spanned S3, FSxN, VPC Endpoints, Route 53 Resolver, and organizational policies across multiple accounts and teams. It's a pattern where the error message ("explicit deny in a resource-based policy") gives limited visibility into the actual root cause.

AWS Technical Account Managers (TAMs) help enterprise customers navigate these situations by:

• Systematically eliminating candidates - methodically testing each layer (IAM, resource policies, SCPs/RCPs, VPC endpoints, DNS, network routing) to isolate the root cause
• Reproducing in isolated environments - building test environments that replicate customer architectures to validate hypotheses without impacting production
• Bridging cross-service knowledge - understanding how services interact in enterprise architectures (centralized endpoints, DNS forwarding, organizational policies)
• Coordinating across teams - translating technical findings into actionable guidance for application, security, and networking teams who each own different pieces of the architecture
• Codifying learnings - helping organizations turn the fix into internal standards (e.g., "always use the hub VPC ID for VPC-origin access points") to prevent recurrence

Learn more about AWS Enterprise Support and Technical Account Managers.

Related Resources

• Managing access with S3 Access Points
• Amazon FSx for NetApp ONTAP - Accessing data through S3
• VPC endpoints for Amazon S3
• Route 53 Resolver forwarding rules