Some Applications Timeout When Accessing S3 Through VPC Endpoint While Others Work

If the S3 bucket is in the same region as the VPC, I would recommend creating a VPC gateway endpoint instead of a VPC interface endpoint for S3. Gateway endpoints are entirely free of charge, as opposed to interface endpoints that incur both fixed per-hour fees for each AZ in which an elastic network interface (ENI) is provisioned and per-gigabyte fees for all traffic passing through it.

Gateway endpoints act as IP routers, so they don't use security groups. However, if you need to restrict access to S3 through the gateway endpoint, they do support VPC endpoint policies. With an endpoint policy, you can control which principals (IAM roles or users) are allowed to connect to S3 through the endpoint, which buckets or accounts/organisations owning them are permitted, or you can enforce encryption settings or any other aspects for S3 access that can be controlled by IAM policies.

When you associate the appropriate VPC route tables with your gateway endpoint, all traffic to S3 in the region will be transparently routed through the gateway endpoint. Nothing needs to be separately configured regarding DNS configuration, security groups, or application settings.

The issue you're experiencing with inconsistent access to your S3 bucket through the VPC endpoint could be related to a few factors:

1. DNS Resolution: When you enable the 'Enable Private DNS Name' option for an S3 interface endpoint, it overrides the public DNS resolution for S3 endpoints within that VPC. This means that all S3 requests should be routed through the VPC endpoint. However, if DNS settings are not properly configured in your VPC, some applications might still attempt to use the public S3 endpoint, leading to timeouts.

2. Endpoint Type: You mentioned using an interface VPC endpoint for S3. However, for S3, it's generally recommended to use a gateway endpoint instead. Gateway endpoints are specifically designed for S3 and DynamoDB, and they're more cost-effective and easier to set up. If you're using an interface endpoint, it might cause some unexpected behavior.

3. Region Specificity: If your applications are not explicitly specifying the region when making S3 requests, they might default to us-east-1. This could cause issues if your VPC endpoint is in a different region.

4. SDK Configuration: While you mentioned that all applications are using the AWS SDK with default configurations, it's possible that some applications are overriding these defaults or using different versions of the SDK that handle VPC endpoints differently.

To resolve these issues and ensure consistent behavior:

1. Verify that DNS hostnames and DNS resolution are enabled for your VPC.

2. Consider switching to a gateway endpoint for S3 instead of an interface endpoint.

3. Ensure all applications explicitly specify the correct region when making S3 requests.

4. Review the SDK configurations across all applications to ensure they're consistent and properly configured to use VPC endpoints.

5. Check that the VPC endpoint policy allows access to the specific S3 bucket (company-finance-reports) for all necessary actions.

6. Verify that the route tables for your subnets have the correct routes to the S3 endpoint.

7. If using custom endpoint URLs, ensure they're correctly formatted and consistently used across all applications.

By addressing these points, you should be able to achieve consistent access to your S3 bucket through the VPC endpoint across all your applications, meeting your security team's requirements.
Sources
Accessing s3 from a Lambda within a vpc | AWS re:Post
Controlling access from VPC endpoints with bucket policies - Amazon Simple Storage Service
S3 endpoint doesn't work | AWS re:Post
Gateway endpoints for Amazon S3 - Amazon Virtual Private Cloud

Since you are getting consistent failures from certain applications, something in your setting is blocking connectivity. Since you are confident with S3 bucket policy and application security groups, the only other thing to check is your endpoint. Check the following for your S3 interface

• Security Groups: check if it has necessary inbound rules
• Subnets: check if the subnet has NACL blocking traffic and has the correct routing policy in route table
• Policy: confirm it's allowing application's IAM role access