Questions tagged with Amazon Simple Storage Service

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

How to get access to s3 for .NET SDK with the same credentials used for awscli?

I am on a federated account that only allows for 60 minutes access tokens. This makes using AWS difficult since I have to constantly relog in with MFA, even for the AWS CLI on my machine. I'm fairly certain that any programmatic secret access key and token I generate would be useless after an hour. I am writing a .NET program (.NET framework 4.8) that will run on a EC2 instance to read and write from an S3 bucket. As per the documentation example, they give this example to initalize the AmazonS3Client: ``` // Before running this app: // - Credentials must be specified in an AWS profile. If you use a profile other than // the [default] profile, also set the AWS_PROFILE environment variable. // - An AWS Region must be specified either in the [default] profile // or by setting the AWS_REGION environment variable. var s3client = new AmazonS3Client(); ``` I've looked into SecretManager and ParameterStore, but that would matter if the programmatic access keys go inactive after an hour. Perhaps there is another way to give the program access to S3 and the SDK... If I cannot use access keys and tokens stored in a file, could I use the IAM access that awscli uses? For example, I can type into powershell `aws s3 ls s3://mybucket` to list and read files from s3 to the ec2 instance. Could the .NET SDK use the same credentials to access the S3 bucket?
1
answers
0
votes
19
views
asked 9 days ago

How to deal with multiple duplicate keys (Fn::Sub) in a aws cloudformation template?

I have a policy that is being made in a cloudformation template. I want to add two resources to the policy, they end up being `arn::bucket` and `arn::bucket/*`. The issue is that the `arn` is a parameter and I get the error: `[cfn-lint] E0000: Duplicate resource found "Fn::Sub" (line 161)`. I understand that it doesn't like the duplicates. ``` "RolePolicies": { "Type": "AWS::IAM::Policy", "Properties": { "PolicyName": "GetGEBucketPutCustomerBucket", "PolicyDocument": { "Statement": [ { "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetObjectAttributes", "s3:GetObjectTagging", "s3:ListBucket", "s3:DeleteObject" ], "Effect": "Allow", "Resource": { "Fn::Sub": [ "${arn}/*", { "arn": { "Ref": "CustomerS3BucketARN" } } ], "Fn::Sub": [ "${arn}", { "arn": { "Ref": "CustomerS3BucketARN" } } ] } } ] }, "Roles": [ { "Ref": "InstanceRole" } ] }, "Metadata": { "AWS::CloudFormation::Designer": { "id": "a713fcc6-95c8-423f-a5b8-0020a81e5ce4" } } } ``` However, this cloudformation is allowed to run, but produces errors. When viewing the policy in IAM console window after create, I see that both of the resources were not created. ![IAM Console](/media/postImages/original/IM-C-6juMgR12vBi6kAOuH5Q) IAM policy editor gives me this error. `Ln 1, Col 0Missing Version: We recommend that you specify the Version element to help you with debugging permission issues.` since the resource than ends with `/*` wasn't created by cloud formation.
1
answers
0
votes
28
views
asked 9 days ago

Pandas overflow error when reading large .txt file from S3

I have a large (>10gb) `txt` file on S3 that I trying to load into pandas. I have tried a couple different approaches, but haven't been able to successfully load / read the data. ``` Python 3.8.12 ``` As for the memory, I am running this on an instance with 32 Gib of RAM, so I don't think it's a memory issue. import pandas as pd import boto3 # AWS credentials import boto3 aws_id = 'xxxx' aws_secret = 'xxxx' Client = boto3.client( 's3', aws_access_key_id=aws_id, aws_secret_access_key=aws_secret ) # Read data from S3 result = Client.get_object(Bucket="bucket-1", Key = "file1.txt") print(result) # To Pandas DataFrame df_loan = pd.read_csv(io.BytesIO(result['Body'].read()), sep="\t", dtype='object', header=None) Traceback: --------------------------------------------------------------------------- OverflowError Traceback (most recent call last) /tmp/ipykernel_4239/3915305936.py in <cell line: 2>() 1 # To Pandas DataFrame ----> 2 df_loan = pd.read_csv(io.BytesIO(result['Body'].read()), sep="\t", dtype='object', header=None) ~/anaconda3/envs/python3/lib/python3.8/site-packages/botocore/response.py in read(self, amt) 97 """ 98 try: ---> 99 chunk = self._raw_stream.read(amt) 100 except URLLib3ReadTimeoutError as e: 101 # TODO: the url will be None as urllib3 isn't setting it yet ~/anaconda3/envs/python3/lib/python3.8/site-packages/urllib3/response.py in read(self, amt, decode_content, cache_content) 513 if amt is None: 514 # cStringIO doesn't like amt=None --> 515 data = self._fp.read() if not fp_closed else b"" 516 flush_decoder = True 517 else: ~/anaconda3/envs/python3/lib/python3.8/http/client.py in read(self, amt) 470 else: 471 try: --> 472 s = self._safe_read(self.length) 473 except IncompleteRead: 474 self._close_conn() ~/anaconda3/envs/python3/lib/python3.8/http/client.py in _safe_read(self, amt) 611 IncompleteRead exception can be used to detect the problem. 612 """ --> 613 data = self.fp.read(amt) 614 if len(data) < amt: 615 raise IncompleteRead(data, amt-len(data)) ~/anaconda3/envs/python3/lib/python3.8/socket.py in readinto(self, b) 667 while True: 668 try: --> 669 return self._sock.recv_into(b) 670 except timeout: 671 self._timeout_occurred = True ~/anaconda3/envs/python3/lib/python3.8/ssl.py in recv_into(self, buffer, nbytes, flags) 1239 "non-zero flags not allowed in calls to recv_into() on %s" % 1240 self.__class__) -> 1241 return self.read(nbytes, buffer) 1242 else: 1243 return super().recv_into(buffer, nbytes, flags) ~/anaconda3/envs/python3/lib/python3.8/ssl.py in read(self, len, buffer) 1097 try: 1098 if buffer is not None: -> 1099 return self._sslobj.read(len, buffer) 1100 else: 1101 return self._sslobj.read(len) OverflowError: signed integer is greater than maximum
1
answers
0
votes
10
views
asked 11 days ago