Questions tagged with Amazon Simple Storage Service

We had scheduled, daily auto back-up of our SqlServer Db on RDS, to S3 bucket for at least last 6 years. It was working fine since then. Suddenly it stopped working, which means we don't see any Backup in our S3 bucket since 24th March. Up on diagnosing the problem, we realized that it is failing since then and the error is STEP1 exec msdb.dbo.rds_restore_database @restore_db_name='RestoreDbFromS3', @s3_arn_to_restore_from='arn:aws:s3:::awsbucketName/SqlServerDb.bak'; STEP 2 exec msdb.dbo.rds_task_status @task_id=7; Response indicates Error with following Task Description [2022-05-28 12:51:22.030] Task execution has started. [2022-05-28 12:51:22.237] Aborted the task because of a task failure or an overlap with your preferred backup window for RDS automated backup. [2022-05-28 12:51:22.240] Task has been aborted [2022-05-28 12:51:22.240] The provided token has expired. We studied a lot to identify the root cause and solution but could not find anything accurately relevant. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/SQLServer.Procedural.Importing.html#SQLServer.Procedural.Importing.Native.Troubleshooting above link shows troubleshooting options as per the error responses, but this does not include the error response that we are getting. Note: between 25th & 26th March, our aws instance was suspended for couple of hours due to delayed payment of monthly invoice. we restored the same quickly. Everything on the same aws account is working fine since then, but we just found out that db backup service has impacted as we see the last successful backup available in S3 bucket is of dated 24th March. We suspect that some token has expired up on account suspension, but are unable to identify which one and how to restore the same back to normal. Help, Assistance and Guidance would be much appreciated.

I found here: https://aws.amazon.com/blogs/mt/configuration-history-configuration-snapshot-files-aws-config/ " AWS Config delivers three types of configuration files to the S3 bucket: Configuration history (A configuration history is a collection of the configuration items for a given resource over any time period. ) Configuration snapshot OversizedChangeNotification" However, in this docs: https://docs.aws.amazon.com/ja_jp/config/latest/developerguide/delete-config-data-with-retention-period.html It only said that retention period delete the "ConfigurationItems" (A configuration item represents a point-in-time view of the various attributes of a supported AWS resource that exists in your account. ) In this docs: https://docs.aws.amazon.com/config/latest/developerguide/config-concepts.html#config-history: "The components of a configuration item include metadata, attributes, relationships, current configuration, and related events. AWS Config creates a configuration item whenever it detects a change to a resource type that it is recording. " I wonder that: Is ConfigurationItems a subset of Configuration history? Is the things that saved to S3 equal to ConfigurationItems? If not, where is ConfigurationItems stored? And if things stored in S3, is ConfigurationItems deleted or become damaged? I am setting AWS S3 lifcycle is expire objects in 300 days and AWS Config retention period is 7 years. Therefore, I am wondering what is the relationship between those 2? Because S3 lifecycle period is 300 days, will AWS Config data is deleted in 300 days? Thank you so much!

I have an Amplify Hosting frontend that uses a rewrite redirect (200) to redirect to an S3 bucket. That worked out fine but then I created a CloudFront distribution and since then, the rewrite gives a timeout after 30 seconds serving a 504 error. That is why I disabled and then deleted the CloudFront distribution. However the timeout and 504 error (coming from CloudFront) is still there. Any ideas how to fully remove this CloudFront forwarding or do I have to delete the bucket/frontend?

When downloading large files (e.g. a few TB of data) via S3 CLI, the session token expires beyond 36 hours. I understand from [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html) that it's not possible to extend or refresh temporary credentials. I would like to know if s3 sync can resume the same downloading job with a new set of temporary credentials.

Hi, the current pipeline I am implementing involves a Kinesis Data Stream -> Kinesis Delivery Stream -> S3 route, however when setting the buffer intervals (Both s3 and transformation) to near 5 minutes, data still rapidly appears within the S3 bucket. For my purposes I need the data for every 5 minutes to be combined into a single large file to be pushed into S3, which I was under the assumption Kinesis delivery streams should handle under the hood when taking in from a Kinesis data stream. I'd appreciate any help in pointing out where my implementation could potentially be going astray :).

Hi, How do I delete groups of versioned files in S3? I have thousands of files and do not want to have to select to delete each file. I have added a Lifecycle rule but it does not seem to be deleting the versions. Thanks,

SES – email receiving + S3 action + encrypted S3 bucket = FAIL hi there, Much like https://stackoverflow.com/questions/53666393/how-to-integrate-amazon-ses-with-encrypted-bucket I'm having difficulty attempting to use SES email receiving where the email is put in an S3 bucket with encryption rules and default encryption settings applied. When creating the rule I receive the following message: > Unknown error > > Request failed with unknown error, ensure that you have the necessary permissions and that all fields have correct values. > > Server-side message: Could not write to bucket: BUCKET_NAME_HERE Thought / comments appreciated given the details in the documentation are not exactly transparent on the topic.

Hi I'm using my new Synology NAS and its app Glacier to backup my files to AWS Glacier. Everything is fine. Glacier vaults are in place and files are in there. I was curious to see how Synology use S3 Glacier. So, I tried to find the corresponding S3 bucket used by Synology App.... but there is no bucket ... Question : How can I have all my files in Amazon Glacier and not having a S3 bucket with my files ? Thanks

I connected s3 and cloudfront using oai. So, since I connected using oai, I couldn't access the s3 domain using the original domain. In my opinion, the distribution domain is accessed using oai and the original domain, and the original domain seems to be a rest api to access the s3 domain. I wonder if my thinking is correct.

Have created a DMS task to replicate data from MongoDB to S3 as parquet files. The updates, inserts, deletes are working fine and replicating in target, but the new columns added to source are not reflecting in the S3 file generated from DMS. The file contains the same schema as source when DMS ran a full load What all settings should I do in the task to reflect these alter statements of adding column to target?

I'm unit testing the process using [S3's SelectObjectContent](https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#S3.SelectObjectContent). Replace SelectObjectContent with a mock of [s3iface](https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3iface/). But I don't know how to create a mock response to get the `e.Payload`. Please tell me how to get `e.Payload` by the SelectObjectContent mock. **process** ``` resp, err := svc.SelectObjectContent(params) if err != nil { return err } defer resp.EventStream.Close() for event := range resp.EventStream.Events() { switch e := event.(type) { case *s3.RecordsEvent: fmt.Println(string(e.Payload)) } } ``` ** Mock ** ``` type mockS3Client struct { s3iface.S3API } func (m *mockS3Client) SelectObjectContent(input *s3.SelectObjectContentInput) (*s3.SelectObjectContentOutput, error) { // mock response/functionality } ```

I have a flask application that needs to access some configuration files stored in s3 and I don't want to stored these in my git repo as there are sensitive information. I want to use boto3 to read in these files into the ec2 instance. **I need to know how I will be able to access those files on s3 using boto3**. The **ec2 instance is authorized to access the s3 bucket** and since the application will be running on the ec2 (that is authorized) is it safe to say that any code run on that instance will also have these access. Note that I am going to run these script to read in these "**config files**" from my **codedeploy "appspec.yml"**. In summary will my codeploy instance be able to use the authorization on my ec2 instance to access the said s3 bucket or **I need to give the codedeploy instance access too**? I am aware I can add my **access keys** in my **environment** on my computer to run this script but how does it work on codedeploy?

For example, 1. https://d2u87yfqb01lll.cloudfront.net/test/ Returns AccessDenied error. But the URL below is done responding successfully. https://d2u87yfqb01lll.cloudfront.net/test/index.html 2. https://d12dcz4jiki7qg.cloudfront.net/%e2%80%aabitcoin-straight-up-gave-me-the-middle-finger-today%e2%80%ac-reddit-bitcoin/ Returns NoSuchKey error. But the URL below is done responding successfully. https://d12dcz4jiki7qg.cloudfront.net/%e2%80%aabitcoin-straight-up-gave-me-the-middle-finger-today%e2%80%ac-reddit-bitcoin/index.html Both are setup correctly. I enbaled S3 hosting for the contents(index.html as the default page is setup). Made the bucket and all the objects under the bucket public access. ACL are both enabled. Cloudfront has default root object. Please let me know which steps I might have missed. Many thanks.

Hi , I created data delivery stream and create IAM role by default as proposed by AWS but still I got error "Access was denied. Ensure that the trust policy for the provided IAM role allows Firehose to assume the role, and the access policy allows access to the S3 bucket." While the IAM role is defined by AWS auto, what should I do now? If I need something to add what should I add, kindly need a brief help?

I created a S3 Lifecycle rule to delete expired object, and uploaded S3 object with aws cli `aws s3 cp hello-unsigned.txt s3://bucket/ --no-signed-request --endpoint-url https://bucket.s3-vpce/`. The S3 object was uploaded successfully, but didn't show any information about `expiration rule` and `expiration date` in `Management configurations` block. I uploaded another S3 object using `aws s3 cp hello-signed.txt s3://bucket/ --endpoint-url https://bucket.s3-vpce/`. This S3 object was applied Lifecycle rule. Is this a bug or normal behavior about unsigned S3 object? Or I misconfiguration on any steps?

I am trying to use boto3 to connect to an S3 bucket from a container that is hosted on an EC2 instance within ECS. All of this is hosted on AWS GovCloud, which only allows private hosted zones in route53. Instead I have set up GoDaddy DNS to resolve my elastic load balancer URI and to manage the Amazon Certificate for https connections to the application. When I try to run the command `python manage.py collectstatic --no-input` which should upload/update any new/modified files to S3 as part of the application set up the connection times out. According to [this article](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html#vpc-endpoints-policies-s3) > If you're using the Amazon DNS servers, you must enable both DNS hostnames and DNS resolution for your VPC. If you're using your own DNS server, ensure that requests to Amazon S3 resolve correctly to the IP addresses maintained by AWS. How do I go about setting up a private hosted zone to manage resolution of the S3 connection since I am using by own DNS server?

Hi all! I have a Django website which is deployed on AWS Lambda. All the static/media is stored in the S3 bucket. I managed to serve static from S3 and it works fine, however, when trying to upload media through admin (I was trying to add an article with a pic attached to it), I get a message "Endpoint request timed out". Here is my AWS and storage configuration: **ukraine101.aws.utils.py** ``` from storages.backends.s3boto3 import S3Boto3Storage StaticRootS3BotoStorage = lambda: S3Boto3Storage(location='static') MediaRootS3BotoStorage = lambda: S3Boto3Storage(location='media') ``` **settings.py** ``` STATICFILES_DIRS = [BASE_DIR / "static"] STATIC_URL = 'https://<my-bucket-name>.s3.amazonaws.com/' MEDIA_URL = 'https://<my-bucket-name>.s3.amazonaws.com/media/' MEDIA_ROOT = MEDIA_URL DEFAULT_FILE_STORAGE = 'ukraine101.aws.utils.MediaRootS3BotoStorage' STATICFILES_STORAGE = 'ukraine101.aws.utils.StaticRootS3BotoStorage' AWS_STORAGE_BUCKET_NAME = '<my-bucket-name>' AWS_S3_REGION_NAME = 'us-east-1' AWS_ACCESS_KEY_ID = '<my-key-i-dont-show>' AWS_SECRET_ACCESS_KEY = '<my-secret-key-i-dont-show>' AWS_S3_SIGNATURE_VERSION = 's3v4' AWS_S3_FILE_OVERWRITE = False AWS_DEFAULT_ACL = None AWS_S3_VERIFY = True AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME STATICFILES_LOCATION = 'static' ``` **My Article model:** ``` class Article(models.Model): title = models.CharField(max_length=250, ) summary = models.TextField(blank=False, null=False, ) image = models.ImageField(blank=False, null=False, upload_to='articles/', ) text = RichTextField(blank=False, null=False, ) category = models.ForeignKey(Category, null=True, blank=True, default='', on_delete=models.SET_DEFAULT) featured = models.BooleanField(default=False) date_created = models.DateField(auto_now_add=True) slug = AutoSlugField(populate_from='title') related_book = models.ForeignKey(Book, null=True, blank=True, default='', on_delete=models.SET_DEFAULT) def get_absolute_url(self): return reverse("articles:article-detail", kwargs={"slug": self.slug}) def get_comments(self): return Comment.objects.filter(article=self.id) author = models.ForeignKey(User, null=True, blank=True, default='', on_delete=models.SET_DEFAULT) ``` **AWS bucket policy:** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicRead", "Effect": "Allow", "Principal": "*", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectAcl" ], "Resource": "arn:aws:s3:::<my-bucket-name>/*" } ] } ``` **CORS:** ``` [ { "AllowedHeaders": [ "*" ], "AllowedMethods": [ "GET", "POST", "PUT", "HEAD" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [], "MaxAgeSeconds": 3000 } ] ``` **User permissions policies (there are two attached): ** Policy 1: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets" ], "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads", "s3:ListBucketVersions" ], "Resource": "arn:aws:s3:::<my-bucket-name>" }, { "Effect": "Allow", "Action": [ "s3:*Object*", "s3:ListMultipartUploadParts", "s3:AbortMultipartUpload" ], "Resource": "arn:aws:s3:::<my-bucket-name>/*" } ] } ``` Policy 2: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:*", "s3-object-lambda:*" ], "Resource": [ "arn:aws:s3:::<my-bucket-name>", "arn:aws:s3:::<my-bucket-name>/*" ] } ] } ``` Please, if someone knows what can be wrong and why this timeout is happening, help me.

Hello, As the title says, when I use SSE-KMS, S3 can directly access the KMS key I specify without authorizing S3 to assume any role, why is that?

I have a verified domain set up in SES that saves received emails to S3 and I do some processing on it from there. This has been working completely as expected for a few weeks (at very low volume) but as of this afternoon it has taken up to 50 minutes for an email to arrive in S3. I have been testing at a slightly higher frequency but not more than once per 10 minutes. Is there anything that could cause this or is SES just slow today? Thanks in advance

I have an s3 life cycle policy using intelligent tiering where objects go to archive after 90 days and then transition to deep glacier after the next 90. And then remain in deep glacier until the retention policy hits at 305 total days. I'd like to remove the deep glacier rule and let the stuff thats currently on deep glacier age out. But if I remove the deep glacier transition, I'm concerned that all objects already on deep glacier will then hit the archive rule since technically they are over 90 days old, after which all those older objects will transition from deep glacier back to the archive storage tier. So the question is, what's the behavior: Will only objects turning 90 days old transition to the archive tier and all deep glacier objects can age out, or will ALL objects currently in deep glacier transition back to archive?

When creating a Ground Truth labelling job, some of the files in S3 are not displaying in the browser. The files (.png image and .flac audio file) are available in S3 and are able to be seen / heard when downloaded. Any ideas on what is wrong?

Hi, We have some buckets (have been around for a while, approx 200GB+ data) and we want to **turn on** encryption-at-rest using SSE-S3 (the most "transparent" way) https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-encryption.html The S3 buckets are mounted to our Linux VMs using S3FS https://github.com/s3fs-fuse/s3fs-fuse which has support for this (seems fairly transparent) So, it seems like the way this works is that you can only enable this on files on a go-forward basis so the older files that already exist will not be in encrypted-at-rest (which is ok, we can backfill this later) Has anybody tried to do this before using this combo? If we mount the bucket using s3fs with `-o use_sse` option, what will happen as the files will be half-and-half? Will it "just work"? s3fs will be mounted with `-o use_sse` and it will be able to handle files that are BOTH the old way (not encrypted-at-rest) and the newer files (encrypted-at-rest) ... we can then start backfilling the older files and we have time or will this fail catastrophically the minute we mount the s3 bucket :( Is the solution to just start a new bucket and do the SSE-S3 and then just start moving the files over (we have done this before in terms of having code in our application check for a file in multiple buckets before giving up) Of course, we will test all this stuff, just wanted to ask a quick question in case we are worried about this too much and if this is a "no big deal" or "be very careful" Thanks!

Hi everyone. I am using node to programmatically create segments in pinpoint.. sdk of createImportJob method needs parameter S3 ("S3 file localed")..however I would like to upload directly without uploading to S3 first like console interface does.. Is there some option? Thanks

I'm hosting a static website via S3 buckets and CloudFront. I've set it up so that the data is stored in the www bucket and the other, non-www, bucket routes to it. Typing in the ULR with the www works perfectly. The issue I'm running into is that when I type in the URL without the www, it routes to another site with a different, but similar URL. It's been like this for about a week and nothing I do seems to change it. I've redone the hosting zones and checked multiple times and none of it seems to work. I've done something like this before with other sites I've set up so I know what I'm doing works. I'm last as to how to go about fixing this so any help would be very much appreciated.

Hi, we have a CodePipeline that we use to deploy our static website. We have it configured to be triggered whenever a zip file is dropped in a specific S3 location, following this: https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-s3deploy.html The problem we have is that we publish some documents to the website, which after some time will have to be deleted from it. With the current behaviour, we will push a new zip file without those files that have to be deleted, but since the pipeline will not delete those files, they will still be available. e.g. 1. We will deploy our website with a new job post. When the user goes to https://www.my-website.com/careers/senior-developer it will access the file `careers/senior-developer.html` in our bucket. 2. A few weeks later, redeploy the website removing that job post. Because the pipeline doesn't delete any files, a user could still go to https://www.my-website.com/careers/senior-developer and still access the file. What's the recommended way of dealing with this issue? Thanks in advance

IHAC who is doing scoping with an Architecture using DMS and SCT. I had a few questions I was hoping you can get answered for me. 1. Does AWS DMS support data validation with Cassandra as a source? I don’t see it here - https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html#CHAP_BestPractices.DataValidation but I do see Cassandra as a valid source target here https://aws.amazon.com/about-aws/whats-new/2018/09/aws-dms-aws-sct-now-support-the-migration-of-apache-cassandra-databases/ 2. Does AWS DMS support ongoing replication with Cassandra as a source? Reading the docs it looks like if I wanted to extract data from Cassandra and write to s3 (Using DMS) then post process that data into a different format (Like json) and write to a different S3 bucket, I could so by attaching a Lamba to the original S3 event from the DMS extract and drop. Can you confirm my understanding? 3. How is incremental data loaded ongoing after initial load from Cassandra (with DMS)? In the docs it looks like its stored in s3 in csv form. Does it write 1 csv per source table and keep appending or updating the existing csv? does it create 1 csv per row, per batch...etc? I’m wondering how the event in step 3 would be triggered if I did want to continuously post process updates as they come in in real time and covert source data from Cassandra into Json data I store on s3.

# Update I added a VPC gateway endpoint for S3 in the same region (US East 1). I selected the route table for it that the lambda uses. But still, the bug persists. Below I've included details regarding my network configuration. The lambda is located in the "api" subnet. ## Network Configuration 1 VPC 4 subnets: * public &nbsp;&nbsp;&nbsp;&nbsp;IPv4 CIDR: 10.0.0.0/24 &nbsp;&nbsp;&nbsp;&nbsp;route table: public &nbsp;&nbsp;&nbsp;&nbsp;Network ACL: public * private &nbsp;&nbsp;&nbsp;&nbsp;IPv4 CIDR: 10.0.1.0/24 &nbsp;&nbsp;&nbsp;&nbsp;route table: private &nbsp;&nbsp;&nbsp;&nbsp;Network ACL: private * api &nbsp;&nbsp;&nbsp;&nbsp;IPv4 CIDR: 10.0.4.0/24 &nbsp;&nbsp;&nbsp;&nbsp;route table: api &nbsp;&nbsp;&nbsp;&nbsp;Network ACL: api * private2-required &nbsp;&nbsp;&nbsp;&nbsp;IPv4 CIDR: 10.0.2.0/24 &nbsp;&nbsp;&nbsp;&nbsp;route table: public &nbsp;&nbsp;&nbsp;&nbsp;Network ACL: - 3 route tables: * public &nbsp;&nbsp;&nbsp;&nbsp;Destination: 10.0.0.0/16 &nbsp;&nbsp;&nbsp;&nbsp;Target: local &nbsp;&nbsp;&nbsp;&nbsp;Destination: 0.0.0.0/0 &nbsp;&nbsp;&nbsp;&nbsp;Target: igw-xxxxxxx &nbsp;&nbsp;&nbsp;&nbsp;Destination: ::/0 &nbsp;&nbsp;&nbsp;&nbsp;Target: igw-xxxxxxxx * private &nbsp;&nbsp;&nbsp;&nbsp;Destination: 10.0.0.0/16 &nbsp;&nbsp;&nbsp;&nbsp;Target: local * api &nbsp;&nbsp;&nbsp;&nbsp;Destination: 10.0.0.0/16 &nbsp;&nbsp;&nbsp;&nbsp;Target: local &nbsp;&nbsp;&nbsp;&nbsp;Destination: 0.0.0.0/0 &nbsp;&nbsp;&nbsp;&nbsp;Target: nat-xxxxxxxx &nbsp;&nbsp;&nbsp;&nbsp;Destination: pl-xxxxxxxx &nbsp;&nbsp;&nbsp;&nbsp;Target: vpce-xxxxxxxx &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(VPC S3 endpoint) 4 network ACLs * public &nbsp;&nbsp;&nbsp;&nbsp;inbound rules: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;All traffic (allow) &nbsp;&nbsp;&nbsp;&nbsp;outbound rules: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;All traffic (allow) * private &nbsp;&nbsp;&nbsp;&nbsp;inbound rules: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;100: PostgreSQL TCP 5432 10.0.0.48/32 (allow) &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;101: PostgreSQL TCP 5432 10.0.4.0/24 (allow) &nbsp;&nbsp;&nbsp;&nbsp;outbound rules: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;100: Custom TCP TCP 32768-65535 10.0.0.48/32 (allow) &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;101: Custom TCP TCP 1024-65535 10.0.4.0/24 (allow) * api &nbsp;&nbsp;&nbsp;&nbsp;inbound rules: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;All traffic (allow) &nbsp;&nbsp;&nbsp;&nbsp;outbound rules: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;All traffic (allow) * \- &nbsp;&nbsp;&nbsp;&nbsp;inbound rules: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;All traffic (allow) &nbsp;&nbsp;&nbsp;&nbsp;outbound rules: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;All traffic (allow) # Update # I increased the timeout of the lambda to 5 minutes, and the timeout of the PUT request to the S3 bucket to 5 minutes as well. Before this the request itself would timeout, but now I'm actually getting a response back from S3. It is a 400 Bad Request response. The error code is `RequestTimeout`. And the message in the payload of the response is "Your socket connection to the server was not read from or written to within the timeout period." This exact same code works 100% of the time for a small payload (on the order of 1KB), but apparently for payloads on the order of 1MB it starts breaking. There is no logic in _my code_ that does anything differently based on the size of the payload. I've read similar issues that suggest the issue is with the wrong number of bytes being provided in the "content-length" header, but I've never provided a value for that header. Furthermore, the lambda works flawlessly when executed in my local environment. The problem definitely appears to be a networking one. At first glance it might seem like this is just an issue with the lambda being able to interact with services outside of the VPC, but that's not the case because the lambda _does_ work exactly as expected for smaller file sizes (<1KB). So it's not that it flat out can't communicate with S3. Scratching my head here... # Original # I use S3 to host images for an application. In my local testing environment the images upload at an acceptable speed. However, when I run the same exact code from an AWS Lambda (in my VPC), the speeds are untenably slow. I've concluded this because I've tested with smaller images (< 1KB) and they work 100% of the time without making any changes to the code. Then I use 1MB sized payloads and they fail 98% percent of the time. I know the request to S3 is the issue because of logs made from within the Lambda that indicate the execution reaches the upload request, but — almost — never successfully passes it (times out).

Hello All, Today I was trying to copy a directory from one location to another, and was using the following command to execute my copy. aws s3 s3://bucketname/directory/ s3://bucketname/directory/subdirectory --recursive The copy took overnight to complete because it was 16.4TB in size, but when I got into work the next day, it was done, or at least it had completed. But when I do a compare between the two locations I get the following bucketname/directory/ 103,690 objects - 16.4TB bucketname/directory/subdirectory/ 103,650 - 16.4TB So there is a 40 object difference between the source location and the destination location. I tried using the following command to copy over the files that were missing aws s3 sync s3://bucketname/directory/ s3://bucket/directory/subdirectory/ which returned no results. It sat for a while maybe like 2 minutes or so, and then just returned to the next line. I am at my wits end trying to copy of the missing objects, and my boss thinks that I lost the data, so I need to figure out a way to get the difference between the source and destination copied over. If anyone could help me with this, I would REALY appreciate it. I am a newbie with AWS, so I may not understand everything that I am told, but I will try anything to get this resolved. I am doing all the commands through an EC2 instance that I am ssh into, and then use AWS CLI commands. Thanks to anyone who might be able to help me. Take care, -Tired & Frustrated :)

I'm wondering if there's a way to make the S3 path unguessable. Let's suppose I have an S3 path like this: https://s3-bucket.com/{singer_id}/album/song/song.mp3, this file will be served through CloudFront, so the path will be: https://cloundfront-dist-id.com/{singer_id}/album/song/song.mp3?signature=... (I'm using signed URLs). My question is : it is possible to make the /{singer_id}/album/song/song.mp3 not guessable by hashing it using for example Lambda or Lambda@Edge function so the client will see a url like this https://cloundfront-dist-id.com/some_hash?signature= ? Thanks in advance. https://stackoverflow.com/questions/70885356/non-guessable-cloudfront-url I am also facing issue. Question may arise why need of hash because signed url are secure. For my side, I need such url with s3 path hidden. I am using same AWS bucket for retrieving image for internal use without signed url and sharing that file to others using signed url. Internal USe CDN without signed url after CNAMe https://data.example.com/{singer_id}/album/song/song.mp3 Signed url https://secured-data.example.com/{singer_id}/album/song/song.mp3?signature=. &Expires == Since both using same AWS bucket and if someone guesses in signed url then access content https://data.example.com/{singer_id}/album/song/song.mp3?signature=. &Expires . File opens . In this scenario, I want to hide {singer_id}/album/song/song.mp3 to some new value and file is displayed under new name

Hello Every one, I'm starting with AWS by trying to build a personal blog with downloadable content from S3 (PDF and images mostly for now) . I'm concerned at this point by data transfer costs when users download this content. Is my concern legitimate ? is it better to avoid downloadable content in this case if it will incur unpredictable costs ? Thanks in advance

Is there an AWS service that can codelessly, serverlessly download an external file via HTTPS to an S3 bucket? Is there a way I can serverlessly, codelessly download a file from a given external HTTPS URL and save the downloaded file to a given S3 bucket? Is there a way I can do this with Step Functions? Is my only option to drop the codeless requirement and use Lambda?

I am trying to make an S3 Backup using AWS Backup. The error message I'm getting is (I have deliberately changed the bucket name and account number) ``` Unable to perform s3:PutBucketNotification on my-bucket-name-123 The backup job failed to create a recovery point for your resource arn:aws:s3:::my-bucket-name-123 due to missing permissions on role arn:aws:iam::123456789000:role/service-role/AWSBackupDefaultServiceRole. ``` I have attached the inline policies described in the [documentation](https://docs.aws.amazon.com/aws-backup/latest/devguide/s3-backups.html) to AWSBackupDefaultServiceRole (note: the role also contains the AWS managed policy AWSBackupServiceRolePolicyForBackup as well as the following) ``` { "Version":"2012-10-17", "Statement":[ { "Sid":"S3BucketBackupPermissions", "Action":[ "s3:GetInventoryConfiguration", "s3:PutInventoryConfiguration", "s3:ListBucketVersions", "s3:ListBucket", "s3:GetBucketVersioning", "s3:GetBucketNotification", "s3:PutBucketNotification", "s3:GetBucketLocation", "s3:GetBucketTagging" ], "Effect":"Allow", "Resource":[ "arn:aws:s3:::*" ] }, { "Sid":"S3ObjectBackupPermissions", "Action":[ "s3:GetObjectAcl", "s3:GetObject", "s3:GetObjectVersionTagging", "s3:GetObjectVersionAcl", "s3:GetObjectTagging", "s3:GetObjectVersion" ], "Effect":"Allow", "Resource":[ "arn:aws:s3:::*/*" ] }, { "Sid":"S3GlobalPermissions", "Action":[ "s3:ListAllMyBuckets" ], "Effect":"Allow", "Resource":[ "*" ] }, { "Sid":"KMSBackupPermissions", "Action":[ "kms:Decrypt", "kms:DescribeKey" ], "Effect":"Allow", "Resource":"*", "Condition":{ "StringLike":{ "kms:ViaService":"s3.*.amazonaws.com" } } }, { "Sid":"EventsPermissions", "Action":[ "events:DescribeRule", "events:EnableRule", "events:PutRule", "events:DeleteRule", "events:PutTargets", "events:RemoveTargets", "events:ListTargetsByRule", "events:DisableRule" ], "Effect":"Allow", "Resource":"arn:aws:events:*:*:rule/AwsBackupManagedRule*" }, { "Sid":"EventsMetricsGlobalPermissions", "Action":[ "cloudwatch:GetMetricData", "events:ListRules" ], "Effect":"Allow", "Resource":"*" } ] } ``` This to me, looks correct and it not should be giving that error. Is there a bug? Or is there a step which is not described in the documentation? I would really appreciate some help. Many thanks ``` ```

I am a newbie so pardon my ignorance. I am writing a very simple step function state machine that uses the AWS SDK to retrieve a file from S3. Every time I run it the task that gets the file from S3 fails with an "S3.InvalidContent" error with "Failed to convert 'Body' to string" as the cause. The full definition of my state machine is: ``` { "Comment": "A description of my state machine", "StartAt": "GetAudioFile", "States": { "GetAudioFile": { "Type": "Task", "Parameters": { "Bucket": "11123", "Key": "test.wav" }, "Resource": "arn:aws:states:::aws-sdk:s3:getObject", "End": true } } } ``` The full text of the TaskFailed event is: ``` { "resourceType": "aws-sdk:s3", "resource": "getObject", "error": "S3.InvalidContent", "cause": "Failed to convert 'Body' to string" } ``` The full text of the CloudWatch log entry with the error is: ``` { "id": "5", "type": "TaskFailed", "details": { "cause": "Failed to convert 'Body' to string", "error": "S3.InvalidContent", "resource": "getObject", "resourceType": "aws-sdk:s3" }, "previous_event_id": "4", "event_timestamp": "1651894187569", "execution_arn": "arn:aws:states:us-east-1:601423303632:execution:test:44ae6102-b544-3cfa-e186-181cdf331493" } ``` 1. What am I doing wrong? 2. How do I fix it? 3. What additional information do you need from me? 4. Most importantly, where can I find answers to these stupid questions so I don't have to post these stupid questions on re:Post again? (I have spent nearly a day scouring AWS docs and Googling without finding anything.)

I'm trying to use the AWS S3 API to perform a multi-part upload with Signed URLs. This will allow us to send a request to the server (which is configured with the correct credentials), and then return a pre-signed URL to the client (which will not have credentials configured). The client should then be able to complete the request, computing subsequent signatures as appropriate. This appears to be possible as per the AWS S3 documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html Signature Calculations for the Authorization Header: Transferring Payload in Multiple Chunks (Chunked Upload) (AWS Signature Version 4) - Amazon Simple Storage Service - AWS Documentation As described in the Overview, when authenticating requests using the Authorization header, you have an option of uploading the payload in chunks. You can send data in fixed size or variable size chunks. This section describes the signature calculation process in chunked upload, how you create the chunk body, and how the delayed signing works where you first upload the chunk, and send its ... docs.aws.amazon.com The main caveat here is that it seems to need the Content-Length​ up front, but we won't know the value of that as we'll be streaming the value. Is there a way for us to use signed URLs to do multipart upload without knowing the length of the blob to be uploaded beforehand?

Hi There, I have a scenario where the Cost and Usage report has been used for a while, however the corresponding S3 bucket was accidentally deleted last month. The bucket is now restored however there is a need to regenerate the April report in CSV, is there any way to recreate this or is it lost due to the bucket deletion? Thanks

What is the best way to sync my DynamoDB tables to S3, so that I can perform serverless 'big data' queries using Athena? The data must be kept in sync without any intervention. The frequency of sync would depend on the cost, ideally daily but perhaps weekly. I have had this question a long time. I will cover what I have considered, and why I don't like the options. 1) AWS Glue Elastic Views. Sounds like this will do the job with no code, but it was announced 18 months ago and there have been no updates since. Its not generally available, and there is not information on when it might be. 2) Use dynamodb native backup following this blog https://aws.amazon.com/blogs/aws/new-export-amazon-dynamodb-table-data-to-data-lake-amazon-s3/. I actually already use this method for 'one-off' data transfers that I kick-off manually and then configure in Athena. I have two issues with this option. The first is that, to my knowledge, the export cannot be scheduled natively. The blog suggests using the CLI to kick off exports, and I assume the writer intends that the CLI would need scheduling on a cron job somewhere. I don't run any servers for this. I imagine I could do it via a scheduled Lambda with an SDK. The second issue is that the export path in S3 always includes a unique export ID. This means I can't configure the Athena table to point to a static location for the data and just switch over the new data after a scheduled export. Perhaps I could write another lambda to move the data around to a static location after the export has finished, but it seems a shame to have to do so much work and I've not seen that covered anywhere before. 3) I can use data pipeline as described in https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBPipeline.html. This post is more about backing data up than making it accessible to Athena. I feel like this use case must be so common, and yet none of the ideas I've seen online are really complete. I was wondering if anyone had any ideas or experiences that would be useful here?

I'd like to request to S3 as a cognito certification qualification. S3 is using sdk Cognito is using amplify. Use an angular typescript. I would like to replace the secret key with the cognito authentication information when creating S3. I want to access s3 with the user I received from Auth.signIn, but the credentials are missing. I need your help. ``` public signIn(user: IUser): Promise<any> { return Auth.signIn(user.email, user.password).then((user) => { AWS.config.region = 'ap-northeast-2'; AWS.config.credentials = new AWS.CognitoIdentityCredentials({ IdentityPoolId: 'ap-northeast-2:aaaaaaaa-bbbb-dddd-eeee-ffffffff', }); const userSession = Auth.userSession(user); const idToken = userSession['__zone_symbol__value']['idToken']['jwtToken']; AWS.config.region = 'ap-northeast-2'; AWS.config.credentials = new AWS.CognitoIdentityCredentials({ IdentityPoolId: 'ap-northeast-2:aaaaaaaa-bbbb-dddd-eeee-ffffffff', RoleArn: 'arn:aws:iam::111111111111:role/Cognito_role', Logins: { CognitoIdentityPool: 'ap-northeast-2:aaaaaaaa-bbbb-dddd-eeee-ffffffff', idToken: idToken, }, })); const s3 = new AWS.S3({ apiVersion: '2012-10-17', region: 'ap-northeast-2', params: { Bucket: 'Bucketname', }, }); s3.config.credentials.sessionToken = user.signInUserSession['accessToken']['jwtToken']; s3.listObjects(function (err, data) { if (err) { return alert( 'There was an error: ' + err.message ); } else { console.log('***********s3List***********', data); } }); } ``` bucket policy ``` { "Version": "2012-10-17", "Id": "Policy", "Statement": [ { "Sid": "AllowIPmix", "Effect": "Allow", "Principal": "*", "Action": "*", "Resource": "arn:aws:s3:::s3name/*", } ] } ``` cognito Role Policies - AmazonS3FullAccess ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:*", ], "Resource": "*" } ] } ```

Hello, So I am using AWS Pinpoint both programatically and via the AWS Console. To make email templates I use the AWS Console. I have several images in an S3 bucket and I hide those images behind a CloudFront distribution so that they are accessible only via that distribution while the bucket is private. I have an image in the email template and in the `src` of the image I put the URL of the image behind the distribution. Outside the template, I am able to see the image on that URL. Also, before sending the email template, the images are properly displayed on the template but once I send it to an email and I open the email, I see no images, only te alternative text I have put. Why is that? How can I fix it? EDIT: I found out the problem exists only in Outlook. Do you have any idea why that is?

I don't have an account to fill out a tech support ticket, but this isn't tech support. It feels like a bug: ``` Access to XMLHttpRequest at 'https://us-east-1.console.aws.amazon.com/s3/proxy' from origin 'https://s3.console.aws.amazon.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. ``` A lot of these errors for all of my buckets. On the main list of buckets I am getting all 400 responses from the `Access` column for all buckets. It shows '**Unknown Error**': ``` Request URL: https://us-east-1.console.aws.amazon.com/s3/s3control-proxy Request Method: POST Status Code: 400 Referrer Policy: no-referrer-when-downgrade ```

I am using a Wordpress plugin called Media Cloud, which allows me to upload files into my CMS, which pass through to an S3 bucket. I'm not sure whether this is happening at the plugin level or the S3 level, but sometimes, when the PDF arrives in my bucket, the metadata shows the Content-Type as image/png, and not application/pdf. This means that the PDF won't open in a browser, because the browser is trying to treat it as an image. Does anyone know why this might happen, and if there is a rule I can set to force all pdfs to have the PDF metadata?

My customer allows users to upload files via multipart upload to S3. We're using the PHP SDK to create the multipart upload. How is it possible with S3 multipart uploads to limit the maximum filesize? Please suggest a way to implement this, the documentation didn't seem to provide any insight into this.

I configured AWS Backup in CDK to enable continuous backups for s3 buckets with this configuration : - backup rule : with `enableContinuousBackup: true` and `deleteAfter 35 days` - backup selection : with `resources` array having the ARN of the bucket directly set and roles setup following the docs of aws : https://docs.aws.amazon.com/aws-backup/latest/devguide/s3-backups.html Later I deleted the stack in CDK and ,as expected, all the resources were deleted except for the vault that was orphaned. The problem happens when trying to delete the recovery points inside the vault, I get back the status as `Expired` with a message `Insufficient permission to delete recovery point`. - I am logged in as a user with AdministratorAccess - I changed the access policy of the vault to allow anyone to delete the vault / recovery point - even when logged as the root of the account, I still get the same message. --- - For reference, this is aws managed policy attached to my user : `AdministratorAccess` , it Allows (325 of 325 services) including AWS Backup obviously. - Here's the vault access policy that I set : ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [ "backup:DeleteBackupVault", "backup:DeleteBackupVaultAccessPolicy", "backup:DeleteRecoveryPoint", "backup:StartCopyJob", "backup:StartRestoreJob", "backup:UpdateRecoveryPointLifecycle" ], "Resource": "*" } ] } ``` Any ideas what I'm missing here ? **Update ** : - A full week after creating the backup recovery point, and still unable to delete it. - I tried deleting it from the AWS CLI but no luck. - I tried suspending the versioning for the bucket in question and tried, but no luck too.

we are fetching my data from pentaho , through snowflake, loading to s3. the data in pentaho is accurate ,but while it travels to s3 bucket , for a column, some portion of the data is randomly divided and send to next row, the data doesn't any new line (\n), for delimiter we use | as a symbol

Is it possible to configure a default storage class for a S3 bucket other than Standard. For example can I configure the default storage class of a bucket to Intelligent Tiering so that any objects which are PUT to the bucket will be stored in Intelligent Tiering unless explicitly set otherwise in the initial PUT? This would save the transition cost of each object.

is it possible to save a file , where the lambda is running. i am using a javascript library to manipulate some text , same code below . I am using a thirdparty async function to get some data and as the returned array can be quite big, i think it is failing , when i try to upload it to s3. how can i save it to a file in /tmp/file.txt or where the lambda is running , so that i can debug to see at least the data is being returned and then is it possible to upload the file to and s3 and then delete it locally? ``` thirdparty.func() .then(function(data){ //data returned is an array data.forEach(function(val){ //write val to a local file }) }) .catch(e=>{console.log(e)}); ```

### Quick Summary: If objects are put into a bucket owned by "Account A" from a different account ("Account B"), you cannot access files via S3 static website (http) from "Account A" (bucket owner). This is true regardless of the bucket policy granting GetObject on all objects, and regardless of if bucket-owner-full-control ACL is enabled on the object. - If trying to download a file from Account A via S3 API (console/cli), it works fine. - If trying to download a file from Account A via S3 static website (http), S3 responds HTTP 403 Forbidden if the file was uploaded by Account B. Files uploaded by Account A download fine. - Disabling Object ACL's fixes the problem but is not feasible (explained below) ### OVERVIEW I have a unique setup where I need to publish files to an S3 bucket from an account that does not own the bucket. The upload actions work fine. My problem is that I cannot access files from the bucket-owner account over the S3 static website *if the files were published from another account* (403 Forbidden response). **The problem only exists if the files were pushed to S3 FROM a different account.** Because the issue is only for those files, the problem seems like it would be in the Object Ownership ACL configuration. I've confirmed I can access other files (that weren't uploaded by the other acct) in the bucket through the S3 static website endpoint, so I know my bucket policy and VPC endpoint config is correct. If I completely disable Object ACL's completely **it works fine**, however I cannot do that because of two issues: - Ansible does not support publishing files to buckets with ACL's disabled. (Disabling ACL is a relatively new S3 feature and Ansible doesn't support it) - The primary utility I'm using to publish files (Aptly) also doesn't support publishing to buckets with ACL's disabled. (Disabling ACL is a relatively new S3 feature and Aptly doesn't support it) Because of these above constraints, I must use Object ACL's enabled on the bucket. I've tried both settings "Object Writer" and "Bucket owner preferred", neither are working. All files are uploaded with the `bucket-owner-full-control` object ACL. SCREENSHOT: https://i.stack.imgur.com/G1FxK.png As mentioned, disabling ACL fixes everything, but since my client tools (Ansible and Aptly) cannot upload to S3 without an ACL set, ACL's must remain enabled. SCREENSHOT: https://i.stack.imgur.com/NcKOd.png ### ENVIRONMENT EXPLAINED: - Bucket `test-bucket-a` is in "Account A", it's not a "private" bucket but it does not allow public access. Access is granted via policies (snippet below). - Bucket objects (files) are pushed to `test-bucket-a` from an "Account B" role. - Access from "Account B" to put files into the bucket is granted via policies (not shown here). Files upload without issue. - Objects are given the `bucket-owner-full-control` ACL when uploading. - I have verified that the ACL's look correct and both "Account A" and "Account B" have object access. (screenshot at bottom of question) - I am trying to access the files from the bucket-owner account (Account A) over the S3 static website access (over http). I can access files that were not uploaded by "Account B" but files uploaded by "Account B" return 403 Forbidden I am using VPC Endpoint to access (files cannot be public facing), and this is added to the bucket policy. All the needed routes and endpoint config are in-place. I know my policy config is good because everything works perfectly for files uploaded within the same account or if I disable object ACL. ``` { "Sid": "AllowGetThroughVPCEndpoint", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::test-bucket-a/*", "Condition": { "StringEquals": { "aws:sourceVpce": "vpce-0bfb94<scrubbed>" } } }, ``` **Here is an example of how this file is uploaded using Ansible:** Reminder: the role doing the uploading is NOT part of the bucket owner account. ``` - name: "publish gpg pubkey to s3 from Account B" aws_s3: bucket: "test-bucket-a" object: "/files/pubkey.gpg" src: "/home/file/pubkey.gpg" mode: "put" permission: "bucket-owner-full-control" ``` **Some key troubleshooting notes:** - From "Account A" when logged into the console, **I can download the file.** This is very strange and shows that API requests to GetObject are working. Does the S3 website config follow some different rule structure?? - From "Account A" when accessing the file from an HTTP endpoint (S3 website) it returns **HTTP 403 Forbidden** - I have tried deleting and re-uploading the file multiple times. - I have tried manually setting object ACL via the aws cli (ex: `aws s3api put-object-acl --acl bucket-owner-full-control ...`) - When viewing the "object" ACL, I have confirmed that both "Account A" and "Account B" have access. See below screenshot. Note that it confirms the object owner is an external account. SCREENSHOT: https://i.stack.imgur.com/TCYvv.png

I have a S3 bucket configured in AWS lightsail behind a AWS lightsail distribution (generic version from Cloudfront). I have my bucket setup as "individual objects can be public". My problem is that the S3 objects in Lightsail are only accessible on the internet when I switch the permission to "Public access", not when is in "Private" mode. I want to avoid switching the permission to "Public" because then the end user could bypass the CDN and go directly to S3, assuming ofc that the bucket name was known, in my case I'm using custom domain. I know that full version of Cloud front has the option of use Origin Access Identity, which will allow all S3 bucket objects to be private and still be accessible. Is there something similar on Lightsail distribution?

When we enable FIPS, client side libs disables MD5SUM. This [page](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html) lists condition in which etag is md5sum. My question is in case of FIPS S3 [endpoints](https://aws.amazon.com/compliance/fips/), what is Etag, is it md5sum ? how client can understand if it is something else (like sha)

I am trying to delete files form my bucket as two .csv files will be pushed in my bucket daily and i want to delete both files after 14 or 16 hours, and next day the same activity will be repeated as 2 new files will be pushed and then deleted. i think it only delete the files which get older then 24 hours if i am not wrong! any suggestion please!!

I have a website through S3 with a Cloudfront distribution in front of it. The title of the tab for the website homepage has the domain title, but the title of every page just says "Document." How can I change this so that each page of my website shows the domain title in the browser tab title?

Hi team, I create a bucket with CDK and set the property **eventBridgeEnabled to true** ``` const bucket = new Bucket( this, "myBucket", { versioned: false, publicReadAccess: false, blockPublicAccess: BlockPublicAccess.BLOCK_ALL, **eventBridgeEnabled: true**, } ); ``` but when I go to the** s3 console** and check the properties tab/ EventBridge it's still **showing "off" instead of "on"** is there anything missing? Thank you.

Hi team, I want to add a glue workflow as a target to my rule ``` import * as events from "@aws-cdk/aws-events"; const rule = new events.Rule( this, "object_created_event", { description: "description here....", ruleName: "newObj", enabled: true, eventPattern: { source: ["aws.s3"], detailType: ["Object Created"], detail: { bucket: { name: ["test_bucket"], }, }, }, } ); rule.addTarget(new targets.....); ``` I can't find glue workflow as a target on the targets list so that I can add it as a target to my rule. is there any other way to do it other than using ..addTarget method? thank you.

I set up an s3 bucket that collects inventory data from multiple AWS accounts using the Systems Manager "Resource Data Sync". I was able to set up the Data Syncs to feed into the single bucket without issue and the Glue crawler was created automatically. Now that I'm trying to query the data in Athena, I noticed there is an issue with how the Crawler is parsing the data in the bucket. The folder "AWS:InstanceInformation" is not being turned into a table. Instead, it is turning all of the "region=us-east-1/" and "test.json" sub-items into tables which are, obviously, not queryable. To illustrate further, each of the following paths is being turned into it's own table. * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=12345679012/region=us-east-1 * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=12345679012/test.json * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=23456790123/region=us-east-1 * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=23456790123/test.json * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=34567901234/region=us-east-1 * s3://resource-data-sync-bucket/AWS:InstanceInformation/accountid=34567901234/test.json This is ONLY happening with the "AWS:InstanceInformation" folder. All of the other folders (e.g. "AWS:DetailedInstanceInformation") are being properly turned into tables. Since all of this data was populated automatically, I'm assuming that we are dealing with a bug? Is there anything I can do to fix this?

Hi team, I want to enable eventBridge notification on my S3 bucket via CDK, I saw it was added as a property to the bucket props as part of this PR : https://github.com/aws/aws-cdk/pull/18150 but I can't find it on the props ``` import { BlockPublicAccess, Bucket} from "@aws-cdk/aws-s3"; const myS3Bucket= new Bucket( this, "myS3Bucket", { versioned: false, publicReadAccess: false, blockPublicAccess: BlockPublicAccess.BLOCK_ALL, eventBridgeEnabled: true, -- error here ..... } ); ``` I have this error : `eventBridgeEnabled: boolean; }' is not assignable to parameter of type 'BucketProps'.` I even tried with CfnBucket but had same error ``` const bucket = new CfnBucket(this, 'MyEventBridgeBucket', { eventBridgeEnabled: true, }); ``` `Argument of type '{ eventBridgeEnabled: boolean; }' is not assignable to parameter of type 'CfnBucketProps'. Object literal may only specify known properties, and 'eventBridgeEnabled' does not exist in type 'CfnBucketProps'` I want to simply to add it in my first declaration so I can keep those properties : ``` versioned: false, publicReadAccess: false, blockPublicAccess: BlockPublicAccess.BLOCK_ALL, ``` how can I enable eventBridge on my S3 bucket via CDK ? Thank you!

On the Workspaces Web portal config I've enabled file transfer permissions for upload/download and would like to share the attached user storage space wider - can't find the s3 bucket? If it's ephemeral, is there anyway to make it an s3 location?

According to the [official document](https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html), I try to scale write operations by writing to multiple prefixes, i got nothing, but the same limits(3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second). The official document seems misleading or incorrect. Q: How can i achieve a s3 request rate above the limit of 3500 PUT per second? Or, how can i make the prefix partitioned ? PS：I got a solution in[ the link ](https://aws.amazon.com/premiumsupport/knowledge-center/s3-request-limit-avoid-throttling/), but how to "gradually scale up" as described in it ?

After running a simple select query in Athena, I get an error - **HIVE_UNKNOWN_ERROR: Error creating an instance of com.facebook.presto.hive.lakeformation.CachingLakeFormationCredentialsProvider**. Athena query is supposed to run agains a LF database within the same S3 bucket (data source and query destination are split into two folders inside the bucket). Any idea what could be the issue?

Hi Experts, The question is right in title. I would like to know about an approach that could be best in terms of performance and preserving the empty folders for renaming folders. I've tried CLI but somehow it keeps the empty folders intact and doesn't move them over to the new directory. Any help would be appreciated.

Hi, I am using AWS Lamba to read data from web and store it in dynamoDB at periodic intervals during the day. Post which, I take the daily dump from DynamoDB and get a CSV file is S3 for daily reporting. This had been working perfectly fine and I was getting data for all batches during the day. But off late my S3 dump seems to have gaps and I am missing data for different batches. And these missing batches are different for different days. I am not getting any error message - so, just wanted to get some guidance on how to go about debugging this. Below are stats for my dyanmoDB table: Item count: 235,267 Table size: 65.6 megabytes Average item size: 279.03 bytes Provisioned read capacity units: 1 Provisioned write capacity units: 1 Also below is snapshot of key operational metrics for the table: https://ibb.co/TPhWqBj https://ibb.co/jVgRS93 Regards, Dbeings

With SDK v1, the protocol could be set via `ClientConfiguration`: ``` AmazonS3ClientBuilder.standard() .withClientConfiguration(new ClientConfiguration().withProtocol(Protocol.HTTP)) ... .build(); ``` I'm unable to find a similar configuration in `S3ClientBuilder`, `SdkHttpClient.Builder`, or a variety of other classes I've looked through (e.g., `S3Configuration`, `SdkClientOption`, etc). My current plan is to use`ExecutionInterceptor.modifyHttpRequest` to force the protocol in `SdkHttpRequest` but I'm wondering if there's something I've overlooked.

I'm trying to integrate with a service (Ninja Forms File Uploads) that can connect to an S3 bucket to store files. The service requires a "bucket name" as part of its configuration, which works great for both the original bucket's name and the access point's alias I had to make as part of the Object Lambda Access Point setup process. However, it does not seem to work with ARNs, which appears to be the only way I can refer to the Object Lambda Access Point I made. Is there any way to refer to it by an alias?

I have been trying to use the Go SDK to put a lifecycle configuration for a bucket but I keep getting Malformed XML errors. This is the relevant section of the code: ```var filter types.LifecycleRuleFilter rule := types.LifecycleRule{ Status: types.ExpirationStatusDisabled, ID: aws.String("testid2"), NoncurrentVersionExpiration: &types.NoncurrentVersionExpiration{ NewerNoncurrentVersions: 4, NoncurrentDays: 1, }, Filter: filter, } bucketConfig := types.BucketLifecycleConfiguration{ Rules: []types.LifecycleRule{rule}, } putBucketConfigInput := &s3.PutBucketLifecycleConfigurationInput{ Bucket: aws.String("beer-sample-default-default-test5"), LifecycleConfiguration: &bucketConfig, } bucketLifecycleOutput, err := client.PutBucketLifecycleConfiguration(aws.BackgroundContext(), putBucketConfigInput, func(*s3.Options) {}) ``` Would love to get some idea on where I'm going wrong. Thanks!

my website users stream and download large video files (static content) from my AWS S3 bucket. I know If I use CloudFront then 1st TB transfer is free. But my traffic is 10-20 TB per month and I want to minimize this cost. Please share recommended best practices for reducing costs.

Hey all, I'm getting this error when trying to create an s3logging bucket using the aws cli following the instructions found here: https://docs.aws.amazon.com/cli/latest/reference/s3api/create-bucket.html: ``` aws s3api create-bucket \ --bucket s3logs-us-east-1 \ --region us-east-1 \ --object-ownership BucketOwnerEnforced To see help text, you can run: aws help aws <command> help aws <command> <subcommand> help usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters] Unknown options: --object-ownership, BucketOwnerEnforced ``` This is pretty frustrating as I'm LITERALLY just copying and pasting the example found in the documentation and changing the s3 bucket name. This is my aws cli version: ``` aws --version aws-cli/1.19.1 Python/3.9.7 Linux/5.16.15-76051615-generic botocore/1.20.0 ``` Any idea what's going wrong here?

I need to provide an answer to the question, "describe how data is destroyed when no longer needed" for compliance purposes relative to S3 data upon intentional, permanent deletion.

Can one directly invoke a step function directly from a s3 object notification ? for example , when object is created. also, does it work similar to lambda , when hooked up to the s3 object notification. for each object created , will it start a new instance of step function or can we make it such that all of the object notification goes to the same instance of step function, and as i understand step function are state machines , so say i have two states enabled and disabled. when object is created, state machine/step function will move to enable state , and for any other s3 object created notification that comes after the first one , as the step function is already in a enabled state , it won't do anything unless it is in a disabled state. once the files are processed it will move to disabled state.

Hi, I'd like to access a specific file which I know the key of through the AWS Cli. Before using MRAP, I simply had to do `aws s3 cp s3://my-bucket/my-path/file.png ./file.png` to download a specific file from within my private subnet (through a VPC Endpoint for S3). Now that I've setup MRAP, I'd like to know if there is an easy way to achieve the same goal using AWS CLI or if I need to implement my own CLI for this specific purpose. Thanks in advance! (I couldn't tag this question properly, there are no tag for awscli or mrap)

Hi, All of a sudden our site wasn't saving images to the S3 bucket and also getting Access Denied from Cloudfront. We have a WordPress site and aren't using Plugins to save to our S3 bucket. Does Amazon change permissions over time? As its a very strange issue that images just stopped uploading..

I've been trying to setup through AWS Management Console an S3 event to send notification to SQS for newly-created objects but I'm always getting an error. **Unknown Error** An unexpected error occurred. ***Unable to validate the following destination configurations*** NOTE: Both S3 and SQS are in the same region.

Some csv data sources that I've updated into s3 are showing an object url (It has ...s3.us-west-1.amazonaws.com... in the url). When trying to paste the full path in the browser, I get this error: <Error> <Code>AccessDenied</Code> <Message>Access Denied</Message> <RequestId>Q7W20ZWNAHAP2VXM</RequestId> <HostId>sc6hPDs7qlwkZ8wozFhWenhx8tFx1VCz7nktdM87ieQF7yzy+2C2skVqDzrryiQps//BA+HHy2A=</HostId> </Error>

What is the best DB service in AWS to store XML data? I do not want to convert to JSON. Need to retrieve XML and display on UI when needed. Size may not be more than 200 KB max. Today it is stored in oracle as BLOB/CLOB which we want to redesign.

I've been working on a python script (python version 3.9.6) using boto3 to upload files to my s3 bucket. I want to back up files to AWS as my offsite backup, plus I want to learn more about AWS since I'm a programmer. The problem is my PC freezes when I'm in the middle of uploading it. I have been trying to debug whether or not it's a Windows issue, but I recently changed my configuration to upload chunks in 1000mb sizes and it froze after 15 minutes. Previously, I was using 200mb chunks. My files are rather large so I want to use large chunks. How do I inow if this is a windows issue, isp issue, or boto3 issue or my issue with coding my script? I have successfully uploaded many small files to my S3 bucket, so I don't think it's my script. By the way, I should add that I'm also working with my ISP (T-Mobile home internet) because I do lose internet connectivity several times a day. I get the usual endpoint connection error when that happens. I don't know if my ISP issue can cause my PC to freeze up though. Also, I use half the thread available to me on my PC.

In Redshift, How do I insert into an existing table (table name: 'container_tv') from this s3 file: https://###-west-1.amazonaws.com/###/###/###.csv

I'm trying to configure the AWS S3 service to download the included files in a bucket using Unity for mobile. I downloaded the SDK package and I got it installed. From AWS console I set up a IAM policy and roles for unauth users I created a Cognito IdentityPool and got the relative id I set up the S3 bucket and its policy using the generator, including the **arn:aws:iam::{id}:role/{cognito unauth role}** and the resource **arn:aws:s3:::{bucket name}/***. In code I set credentials and region and create CognitoAWSCredentials (C# used) ```C# _credentials = new CognitoAWSCredentials(IdentityPoolId, _CognitoIdentityRegion); ``` then I create the client: ```C# _s3Client = new AmazonS3Client(_credentials, RegionEndpoint.EUCentral1); // the region is the same in _CognitoIdentityRegion ``` I then try to use the s3Client to get my files (in bucketname subfolders) ``` private void GetAWSObject(string S3BucketName, string folder, string sampleFileName, IAmazonS3 s3Client) { string message = string.Format("fetching {0} from bucket {1}", sampleFileName, S3BucketName); Debug.LogWarning(message); s3Client.GetObjectAsync(S3BucketName, folder + "/" + sampleFileName, (responseObj) => { var response = responseObj.Response; if (response.ResponseStream != null) { string path = Application.persistentDataPath + "/" + folder + "/" + sampleFileName; Debug.LogWarning("\nDownload path AWS: " + path); using (var fs = System.IO.File.Create(path)) { byte[] buffer = new byte[81920]; int count; while ((count = response.ResponseStream.Read(buffer, 0, buffer.Length)) != 0) fs.Write(buffer, 0, count); fs.Flush(); } } else { Debug.LogWarning("-----> response.ResponseStream is null"); } }); } ``` At this point I cannot debug into the Async method, I don't get any kind of error, I don't get any file downloaded and I even cannot check is connection to AWS S3 has worked in some part of the script. What am I doing wrong? Thanks for help a lot!

I am currently using GroundTruth Labeling, and have created a manifest file successfully. Right now, I am encountering the error "NetworkingError: Network Failure - The S3 bucket you entered in Input dataset location cannot be reached. Either the bucket does not exist, or you do not have permission to access it. If the bucket does not exist, update Input dataset location with a new S3 URI. If the bucket exists, give the IAM entity you are using to create this labeling job permission to read and write to this S3 bucket, and try your request again." I have made the contents of the S3 bucket and the bucket itself public to everybody, so I was wondering why this error occurs.

I am currently creating a GroundTruth Labeling job, and am following the tutorial https://www.youtube.com/watch?v=_FPI6KjDlCI&t=210s . I have created the same bucket ground-truth-example-labeling-job and uploaded jpg files within the bucket. Within this tutorial, under Select S3 bucket or resource, they were able to go within the S3 Bucket and access the jpg files inside. However, I am able to go inside the ground-truth-example-labeling-job bucket, but no jpg files are visible for me to select. The entire bucket is empty with nothing to select. Is this a permissions settings problem?

Hello, I (account #A) have given access to an external account (account #B) in an S3 bucket with the canonical ID. However, and when I try to download a file to an EC2 bucket, it's still producing the error: ``` fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden ``` I'm trying to follow the instructions at https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/. I think that points 2 and 3 have been taken care of (although I used the console). The issue is that for point 1, I'm starting from https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-with-s3-actions.html and the JSON policy reads: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "statement1", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::***********:user/********" }, "Action": [ "s3:GetObject" ], "Resource": "arn:aws:s3:::cmaq-database/*" } ] } ``` However, the console is not accepting it claiming: '*This policy contains the following error: Has prohibited field Principal*." Thanks.

Hi team, I have an s3 bucket that accepts 20 CSV files through SFTP transfer, I want to move those 20 files to another s3 bucket for treatment only and only if ALL 20 files have been loaded on the first bucket. what is the best option to do that transfer under the condition that all 20 files are completely loaded in the first bucket to trigger the transfer to the second bucket? If I use lambda that listens for s3 PutObject Event, there is no risk of parallel invocation if 20 events are generated at the same time? Thank you.

I'm getting the following error when I run a Glue Crawler on an S3 bucket: > ERROR : Not all read errors will be logged. com.amazonaws.services.s3.model.AmazonS3Exception: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access. (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; The S3 bucket has default "Amazon S3-managed keys (SSE-S3)" encryption enabled, not a CMK from KMS. The Glue Crawler has an IAM role with the managed policy AWSGlueServiceRole and a policy that was created by the wizard: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::<bucket-name>/*" ] } ] } ``` This previous post also has a 403 and was solved by adding GetObject permissions to the specific bucket, which I already have. I'm also not using a VPC endpoint, which is one of the possible problems listed on [this AWS help article](https://aws.amazon.com/premiumsupport/knowledge-center/glue-403-access-denied-error/). And i do NOT have Requester Pays turned on. It's all in the same AWS account. The default private bucket settings are checked, and there's no bucket policy.

Hello, I have given public access to everyone in an S3 bucket (it's a temporal thing and the only object is a simple file). However, and when I try to download the file (using an external account for testing) with: ``` $ aws s3 cp --no-sign-request s3://BUCKET/OBJECT /home/ubuntu ``` I'm still getting the error: ``` fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden ``` In the permissions tab (for the bucket), it has Access: Public, Block all public access: Off, ACL Everyone (public access): List (Objects) and Read (Bucket ACL). I can list the bucket content but not make a copy. What else do I need to do to enable downloading the objects? Thanks.

Hi Team, I have a glue job that reads CSV files from S3 and injects data to Mysql RDS, I want to move all files from s3 to glacier once they're treated by ETL glue. I would like to know what is the best architecture to move CSV files from s3 to glacier upon glue job success? is it better to do it from within the glue script at the end of the script? or to do it through another process? also, any code example will be helpful :) Thank you :)

I have a use-case where I have to read DynamoDB table data and convert this into csv file and write into a s3 bucket using AWS lambda function. DynamoDB tables data is bit low around 2-3 MB so I want to read entire table every week. I am new to AWS and services and want to how to read table without triggers and how to schedule lambda function so it can run every week? Please share the steps to accomplish this task. Any help would be appreciated.

I am looking for the solution of exporting Cloudwatch logs to S3 continuously where there is log updates from application. I am looking for a automated log export from cloudwatch to S3.

Hi Team, I'm trying to read Parquet files in S3, but I get the following error. Please help. I'm not sure if the data inside the parquet file is corrupt or I'm unable to read the file due to datatype mismatch. Any help would be much appreciated. df = spark.read.parquet("s3://xxxxxxx/edo_sms_replica_us_stg/event_t/TESTFILES/LOAD00000CAD.parquet") An error was encountered: Invalid status code '404' from http://ip-xx.xx.xx..awscorp.siriusxm.com:8998/sessions/168 with error payload: {"msg":"Session '168' not found."}

I export data from an Aurora Postgres instance to S3 via the `aws_s3.query_export_to_s3` function. The destination bucket does not have default encryption enabled. When I try to download one of the files I get the following error: > The ciphertext refers to a customer mast3r key that does not exist, does not exist in this region, or you are not allowed to access. Note: I had to change the word mast3r because this forum doesn't allow me to post it as it is a "non-inclusive" word... The reasons seems to be that the files got encrypted with the AWS managed RDS key which has the following policy: ``` { "Version": "2012-10-17", "Id": "auto-rds-2", "Statement": [ { "Sid": "Allow access through RDS for all principals in the account that are authorized to use RDS", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:CreateGrant", "kms:ListGrants", "kms:DescribeKey" ], "Resource": "*", "Condition": { "StringEquals": { "kms:CallerAccount": "123456789", "kms:ViaService": "rds.eu-central-1.amazonaws.com" } } }, { "Sid": "Allow direct access to key metadata to the account", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123456789:root" }, "Action": [ "kms:Describe*", "kms:Get*", "kms:List*", "kms:RevokeGrant" ], "Resource": "*" } ] } ``` I assume that the access doesn't work because of the `ViaService` condition when trying to decrypt the file via S3. I tried to access to files with the root user instead of an IAM user and it works. Is there any way to get access with an IAM user? As far as I know, you cannot modify the policy of an AWS managed key. I also don't understand why the root user can decrypt the file as the policy doesn't explicitly grant decrypt permissions to it other than the permissions when called from RDS.

Hi everyone, **i got **a s3 bucket which continuously generates csv-data, every minute or so. I now want to visualize this big-data via my grafana instance. First **i tried **to do that with aws athena and a glue crawler. But this solution is not very suitable after every query inside my grafana instance generates high costs inside my s3 bucket for tier2 get requests. **Now the idea is** to store the data inside a timestream database and use the timestream-plugin inside grafana to visualize my data. **The question is, if anyone ever used timestream as primary database for this kind of scenario (or are Databases like InfluxDB more suitable). Also whats the easiest way to write the data automatically to my timestreamdb ? ** Greetings Wig

Hello, Can anyone help me below case? I wanted my bucket to access from specific IPs only, otherwise deny. I set up S3 bucket policy as follow: ``` { "Version": "2012-10-17", "Id": "S3PolicyId1", "Statement": [ { "Sid": "IPAllow", "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": [ "arn:aws:s3:::DOC-EXAMPLE-BUCKET", "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*" ], "Condition": { "NotIpAddress": { "aws:SourceIp": "x.x.x.x" }, "Bool":{ "aws:ViaAWSService":"false" } } } ] } ``` For S3 replication, I configured S3 Replication Rule as per AWS Docs by setting policies and attaching to IAM role as follow: ``` { "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Principal":{ "Service":"s3.amazonaws.com" }, "Action":"sts:AssumeRole" } ] } ``` ``` { "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Action":[ "s3:GetReplicationConfiguration", "s3:ListBucket" ], "Resource":[ "arn:aws:s3:::SourceBucket" ] }, { "Effect":"Allow", "Action":[ "s3:GetObjectVersionForReplication", "s3:GetObjectVersionAcl", "s3:GetObjectVersionTagging" ], "Resource":[ "arn:aws:s3:::SourceBucket/*" ] }, { "Effect":"Allow", "Action":[ "s3:ReplicateObject", "s3:ReplicateDelete", "s3:ReplicateTags" ], "Resource":"arn:aws:s3:::DestinationBucket/*" } ] } ``` Without bucket policy, objects are replicated smoothly. Once I add the bucket policy, replication is failed every time. I have no idea. Regards, Ohnmar

Trying to export the VM with the following command ``` aws ec2 export-image \ --image-id ami-02c7de82603c5f32e \ --disk-image-format VMDK \ --s3-export-location S3Bucket=auditor-updates,S3Prefix=export/ ``` And getting the following error: `An error occurred (InvalidParameter) when calling the ExportImage operation: Access denied to the bucket auditor-updates` The bucket and the folder definitely exist, typo is impossible. I launched the command under the AWS CLI credentials corresponding to the DevOps role, however, I also tried root access just to make sure actual account permissions are not the reason.

We are creating S3 objects with KMS managed keys with CSE. I can upload and download the objects via java SDK by building the S3 client using the code below. ` > AmazonS3EncryptionV2 s3Encryption = AmazonS3EncryptionClientV2Builder.standard().withKmsClient(kmsClient) .withEncryptionMaterialsProvider(new KMSEncryptionMaterialsProvider(keyAlias)) .withCryptoConfiguration(new CryptoConfigurationV2().withCryptoMode(CryptoMode.AuthenticatedEncryption)) .build(); ` Key rotation is enable for the used keys. Will I be able to decrypt the old objects in S3 bucket if the key is rotated and key material is updated? How does KMS know which material to use to decrypt the data key when I have some objects created with before KMS key rotation and some after the rotation ? Is the version of key stored as part of the object metadata? I looked at the metadata of the generated objects and I don't see any metadata that references the version of the key material. Am I missing something about the key management and how they should be used with S3?

I have created one S3 bucket, and i deployed my code in that bucket and i try to open that code in chrome with bucketname as url but am getting site can't be reached what is issue ? Can any help me ?