Unanswered Questions tagged with Amazon Simple Storage Service

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Policies applied on organization trail logs bucket created by AWS Tower

Hello, We just setup AWS Tower on our organization. Everything ran smoothly but we detected a strange policy applied by AWS Tower on the bucket responsible to aggregate Cloudtrail trails from all of our organization. This bucket is located on the Log Archive account of Tower architecture. The policy is : ``` { "Sid": "AWSBucketDeliveryForOrganizationTrail", "Effect": "Allow", "Principal": { "Service": "cloudtrail.amazonaws.com" }, "Action": "s3:PutObject", "Resource": [ "arn:aws:s3:::CLOUDTRAIL_BUCKET/ORGANIZATION_ID/AWSLogs/ORGANIZATION_ID/*" ] } ``` This policy allows `cloudtrail` service to push objects on the provided path. Out of curiosity, we tried to configure a Cloudtrail trail located on non-related AWS account (by non-related I mean an AWS account that doesn't belong to the AWS organization) to use this S3 path to push data on. And it worked. Is there any reason why this policy doesn't have a `condition` field to restrict access to accounts that belong to the organization like : ``` "Condition": { "StringEquals": { "aws:PrincipalOrgID": [ "ORGANIZATION_ID" ]} } } ``` Our Tower landing zone version is 3.0. This version enabled Organization-based trail instead of Account-based trails, so I think this policy exists since this version. I know there are some non easily guessable variables (like the Org ID and the bucket name) in the process, but as a compliance tool, AWS Tower should restrict access to the organization itself as it's restricted to it by design. Thanks for your time
0
answers
1
votes
36
views
asked 13 days ago

S3 – file extension and metadata for compressed files

I store various files in an S3 bucket which I'd like to compress. Some using Gzip and some using Brotli. For the Gzip case, I set `Content-Encoding` as `gzip` and for the Brotli case, I set it to `br`. The files have the corresponding suffixes, i.e. `.gz` for Gzip-compressed file and `.br` for Brotli-compressed file. The problem is that when I download the files using Amazon S3 console, both types of files are correctly decompressed, but only the Gzip-compressed files have their suffix removed. E.g. when I download `file1.json.gz` (which has `Content-Type` set to `application/json` and `Content-Encoding` set to `gzip`), it gets decompressed and saved as `file1.json`. However, when I download `file2.json.br` (with the `Content-Type` set to `application/json` and `Content-Encoding` set to `br`), the file gets decompressed but another `.json` suffix is added so the file is saved as `file2.json.json`. I tried to also set `Content-Disposition` to contain `attachment; filename="file2.json"` but this doesn't help. So, I have a couple of questions: - What's the correct way how to store the compressed files in S3 to achieve a consistent handling? According to [`PutObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_RequestSyntax) API it seems, that `Content-Encoding` is what specified that files has been compressed using a specific algorithm and that it needs to be decompressed when accessed by the client, so it seems that the file extension (e.g. `.br`) is not needed. However, some services, e.g. [Athena](https://docs.aws.amazon.com/athena/latest/ug/compression-formats.html) explicitely state that they need the files to have proper extension to be treated like a compressed files. - Is Gzip-compressed file handled differently than other types (e.g. Brotli)? And if so, why and is that browser or S3 which initiates this different handling?
0
answers
0
votes
13
views
asked 20 days ago

Multipart upload with aws S3 + checksums

I am trying to implement browser multipart upload to a S3 bucket. I should be able to pause and play the upload and also I'll like to automatically generate the checksums as I'm uploading. I have tried several approaches and I've been hitting a wall. Some of the approaches I've tried. * Using the amplify S3 upload, this works well, but has the caveat that I can't generate the checksums automatically, to generate the checksums, I run a lambda function after file upload, the caveat is for large files, the lambda function times out. Also, I'll like to avoid going this route as I believe It's quite computationally expensive. * Using https://blog.logrocket.com/multipart-uploads-s3-node-js-react/. This is also similar to the above, the caveat is when I add the checksum algorithm to the upload part query, I get a **checksum type mismatch occurred, expected checksum type sha256, actual checksum type: null site:stackoverflow.com s3**. After a lot of googling, I'm not sure I can compute the checksums using presigned url. * and the current approach is to do away with the presigned url and send the chunked data to the lambda functions which then sends to the bucket. Since I'm managing everything with amplify, I run into some problems with API gateway(multipart/form-data). I have set the gateway to accept binary data and followed other fixes I found online but I’m stuck on **execution failed due to configuration error unable to transform request**. How do I fix the above error and what will be the ideal approach to implement the functionalities(multipart file upload to support resumable uploads and checksum computation)
0
answers
0
votes
24
views
asked 25 days ago

Mounting a file system to Github actions.

I am attempting to shift a workflow into the cloud. So that I can keep costs down I am using Github actions to do some Mac specific stuff - build macOS install packages. This is done using a tool - autopkg. Autopkg caches the application download and package between runs. Unfortunately this cache is too large for Github and can include files too big for Github actions. Package building has to happen on a Mac. Since the next step is to do some uploading of the packages to multiple sites and run some Python to process th built packages and this can run on a small Linux EC2 instance it seems the logical solution is to provide a file system from AWS that autopkg can use as a cache and mount it on every Github action run. I have been tearing my hair out attempting this with either S3 and S3fs or EFS and can't seem to wrap my head around how all the bits hang together. For testing I tried the mount native on my Mac and I tried it in amazonlinux and Debian Docker containers. I'm figuring the solution will be using NFS or efs-utils to mount an EFS volume but I can't get it working. In a Debian container using efs-utils I got close but it seems I can't get the DNS name to resolve. The amazonlinux Docker container was too basic to get efs-utils to work. I also got the aws command line tool installed but it runs in to the same DNS resolution problems. I tried connecting the underlying Mac to an AWS VPN in the same VPC as the file system. still had the same DNS problems. Any help would be appreciated. I've just updated the question with some more stuff I have tried.
0
answers
0
votes
12
views
asked a month ago

Using Cognito and Cloudfront to control access to user files on S3

Hi, I'm putting together a media viewer website for myself to learn how AWS works. My first step was to host a webpage (index.html) on S3, and have this webpage allow for image/video uploads to a folder in my bucket using the AWS Javascript SDK (v2), and having the mediaviewer on the web page access these files directly through http. I have lambda functions that convert media formats appropriately, and hold metadata in DynamoDB that can be queried by the website using the javascript SDK. This all works fine. Now, I'd like to make it a bit more secure, and support users who login, individual user directories within the buckets, and control access to the media files so users can only view their own files. So the steps I used to do this were the following: 1. Create a user pool and identity pool in Cognito. 2. Add a google sign in button, and enable user pool sign in with the google button... To do this, Google requires the webpage to be served via https (not http). 3. Since S3 can't serve files via https, I put the S3 bucket behind cloudfront. 4. Modify my bucket to have a user directory, and subdirectories for each cognito identityid. Modify the access policies so that users can only read/write to their individual subdirectory, and can only read/write to a subset of DynamoDB based on their identity ID. The webpage uses AWS Javascript SDK calls to login with cognito, upload to S3, access dynamodb. It all appears to work well, and seems to give me secure user access control. 5. Now, the hole... I want the media viewer portion of my app to access the images/media via https:// links, and not via the javascript sdk. The way its currently configured, https access goes through cloudfront, and cloudfront has access to all the files in the S3 bucket. I'm trying to figure out how to make an https request via cloudfront (along with a cognito token), and then have cloudfront inspect the token, determine the identity ID of the user, and only serve contents for that user if he is logged in. Does this require lambda@edge or is there an easier way? I don't want to use signed urls, because I anticipate having a single user view hundreds of urls at a single time (in a gallery view), and figure generations signed urls will slow things down too much. 6. In the future, I may want to enable sharing of files... Could I enable that by having an entry in DynamoDB for every file, and have cloudfront check if the user is allowed to view the file, before serving it? Would this be part of the lambd@edge function? Thanks
0
answers
0
votes
31
views
rrrpdx
asked a month ago