By using AWS re:Post, you agree to the Terms of Use

Questions tagged with Serverless

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

How to upload dynamically generated file to S3 bucket using lambda with PHP?

Using [bref][1] as PHP serverless framework. Problem: - Unable to upload the dynamically generated file to S3 bucket from Lambda using PHP. In CloudWatch, there is no any error. Code: - serverless.yml ``` service: lambdaToS3 provider: name: aws region: ap-southeast-2 runtime: provided.al2 #stage: prod profile: default iam: role: statements: - Effect: Allow Action: - 's3:GetObject' - 's3:PutObject' - 's3:GetObjectAcl' - 's3:PutObjectAcl' Resource: - 'arn:aws:s3:::*/*' plugins: - ./vendor/bref/bref functions: lambdaToS3: handler: index.php description: '' layers: - ${bref:layer.php-74} events: - httpApi: '*' # Exclude files from deployment package: patterns: - '!tests/**' - '!tmp/**' ``` index.php `comments are output when executing a file using HTTP GateWay` In this file, I am generating a simple text file in `/tmp/` folder and trying to upload it to the S3 bucket with two different attributes, `SourceFile` or `Body`. None of them working. ``` <?php declare(strict_types=1); ini_set('display_errors', "1"); ini_set('display_startup_errors', "1"); error_reporting(E_ALL); require __DIR__ . '/vendor/autoload.php'; use Aws\S3\S3Client; use Bref\Logger\StderrLogger; $logger = new StderrLogger(); $s3client = new S3Client([ 'scheme' => 'http', 'version' => '2006-03-01', 'region' => 'ap-southeast-2', 'output' => 'JSON' ]); $bucket = 'bucketname'; $file = "/tmp/newfile.txt"; $myfile = fopen($file, "w") or die("Unable to open file!"); $txt = "John Doe\n"; fwrite($myfile, $txt); $txt = "Jane Doe\n"; fwrite($myfile, $txt); fclose($myfile); echo "<pre>"; print_r("File Exist: ". file_exists($file)); echo "</pre>"; // 1 echo "<pre>"; print_r(" ================ "); echo "</pre>"; echo "<pre>"; print_r(file_get_contents($file)); echo "</pre>"; // we can get content using this method. echo "<pre>"; print_r(" ================ "); echo "</pre>"; // Put on S3 using SourceFile $result = $s3client->putObject([ 'Bucket' => $bucket, 'Key' => "AWS_LAMBDA_S3.txt", 'SourceFile' => $file, 'ACL' => 'public-read', ]); echo "<pre>"; print_r("File Upload: " . json_encode($result)); echo "</pre>"; // Put on S3 using Body $myfile = fopen($file, "rb"); $result = $s3client->putObject([ 'Bucket' => $bucket, 'Key' => "AWS_LAMBDA_S3_BODY.txt", 'Body' => $myfile, 'ACL' => 'public-read', ]); echo "<pre>"; print_r("File Upload: " . json_encode($result)); echo "</pre>"; exit; ``` Regarding policy: Once the function deploy to AWS Lambda, we got a following policy for the role ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:CreateLogGroup" ], "Resource": "arn:aws:logs:ap-southeast-2:xxxx:log-group:/aws/lambda/lambdaToS3-dev*:*" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObjectAcl", "s3:PutObjectAcl" ], "Resource": "*" }, { "Sid": "VisualEditor2", "Effect": "Allow", "Action": [ "sqs:DeleteMessage", "sqs:ReceiveMessage", "sqs:GetQueueAttributes", "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:ap-southeast-2:xxxx:log-group:/aws/lambda/lambdaToS3-dev*:*:*", "arn:aws:sqs:ap-southeast-2:xxxx:xxxx" ] } ] } ``` [1]: https://bref.sh
0
answers
0
votes
16
views
asked 6 days ago

AWS lambda function not able to resolve or connect to ipv6 only domain

I implemented a AWS lambda function which shall pass an Alexa custom skill event to my domain for processing, see code below. const https = require('https'); exports.handler = (event, context, callback) => { var options = { hostname: '<my.domain.com>', path: '/<mypath>', port: 443, method: 'POST', rejectUnauthorized: false, headers: { 'Content-Type': 'application/json', 'Authorization': '<my base64 user:password>' } }; const req = https.request(options, (res) => { let body = ''; console.log('Status:', res.statusCode); console.log('Headers:', JSON.stringify(res.headers)); res.setEncoding('utf8'); res.on('data', (chunk) => { body += chunk; }); res.on('end', () => { console.log('Successfully processed HTTPS response'); body = JSON.parse(body); callback(null, body); }); }); req.on('error', callback); req.write(JSON.stringify(event)); req.end(); }; The function runs serverless, not connected to a VPC. The domain <my.domain.com> resolves to an IPv6 address and I am able to connect to my host for example from an internet instance using curl and receive the expected answers. curl -i -k -v -X POST -d testcase.json -u <user:password> https://<my.domain.com>:<my port>/<my path> In AWS I implemented a test case and run it. The test returned the error ENOTFOUND from function getaddrinfo trying to resolve my domain, see execution result below. Test Event Name Test0001 Response { "errorType": "Error", "errorMessage": "getaddrinfo ENOTFOUND <my.domain.com>", "trace": [ "Error: getaddrinfo ENOTFOUND <my.domain.com>", " at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)" ] } Function Logs LOGS Name: cloudwatch_lambda_agent State: Subscribed Types: [platform] EXTENSION Name: cloudwatch_lambda_agent State: Ready Events: [SHUTDOWN,INVOKE] START RequestId: 78314f37-e991-4d3d-b4f2-03da64bf91b7 Version: $LATEST 2022-09-24T04:59:06.966Z 78314f37-e991-4d3d-b4f2-03da64bf91b7 ERROR Invoke Error {"errorType":"Error","errorMessage":"getaddrinfo ENOTFOUND <my.domain.com>","code":"ENOTFOUND","errno":-3008,"syscall":"getaddrinfo","hostname":"<my.domain.com>","stack":["Error: getaddrinfo ENOTFOUND <my.domain.com>"," at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)"]} END RequestId: 78314f37-e991-4d3d-b4f2-03da64bf91b7 REPORT RequestId: 78314f37-e991-4d3d-b4f2-03da64bf91b7 Duration: 425.43 ms Billed Duration: 426 ms Memory Size: 128 MB Max Memory Used: 76 MB Init Duration: 248.14 ms During my investigation I found the hint to add option „family: 6,“. Using this option the test case resolves the domain now to the correct ipv6 address, but returns then EAFNOSUPPORT trying to connect to the address, see execution result below. Request ID 78314f37-e991-4d3d-b4f2-03da64bf91b7 Test Event Name Test0001 Response { "errorType": "Error", "errorMessage": "connect EAFNOSUPPORT xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:443 - Local (undefined:undefined)", "trace": [ "Error: connect EAFNOSUPPORT xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:443 - Local (undefined:undefined)", " at internalConnect (node:net:953:16)", " at defaultTriggerAsyncIdScope (node:internal/async_hooks:465:18)", " at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1097:9)", " at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:73:8)" ] } Function Logs LOGS Name: cloudwatch_lambda_agent State: Subscribed Types: [platform] EXTENSION Name: cloudwatch_lambda_agent State: Ready Events: [INVOKE,SHUTDOWN] START RequestId: f3493148-071f-466d-94c7-d29a0d715640 Version: $LATEST 2022-09-24T05:06:52.877Z f3493148-071f-466d-94c7-d29a0d715640 ERROR Invoke Error {"errorType":"Error","errorMessage":"connect EAFNOSUPPORT xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:443 - Local (undefined:undefined)","code":"EAFNOSUPPORT","errno":-97,"syscall":"connect","address":"xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx","port":443,"stack":["Error: connect EAFNOSUPPORT xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:443 - Local (undefined:undefined)"," at internalConnect (node:net:953:16)"," at defaultTriggerAsyncIdScope (node:internal/async_hooks:465:18)"," at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1097:9)"," at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:73:8)"]} END RequestId: f3493148-071f-466d-94c7-d29a0d715640 REPORT RequestId: f3493148-071f-466d-94c7-d29a0d715640 Duration: 447.45 ms Billed Duration: 448 ms Memory Size: 128 MB Max Memory Used: 76 MB Init Duration: 231.52 ms Request ID f3493148-071f-466d-94c7-d29a0d715640 Any further investigation was not successful. I assume it is an issue using IPv6, but I am not able to solve it. Any help is appreciated. Thank you in advance. Joachim
2
answers
0
votes
25
views
asked 9 days ago

Can someone explain this "connect EAFNOSUPPORT" Lambda error?

Hello, I have deployed a Node JS API to Lambda which works fine on my local development machine. However when the Lambda is triggered it runs until it hits a call to an external API(authorize.net). I receive the following error that I'm not quite sure the meaning of. Did a search and came up empty. There are other external API calls in the function that work if I circumvent this particular problematic call. Your help would be greatly appreciated. ``` 2022-09-22T16:51:59.090Z 29f689fb-1ae5-4b26-bd21-69ce6bab6ae1 INFO { "message": "connect EAFNOSUPPORT ::1:80 - Local (undefined:undefined)", "name": "Error", "stack": "Error: connect EAFNOSUPPORT ::1:80 - Local (undefined:undefined)\n at internalConnect (node:net:953:16)\n at defaultTriggerAsyncIdScope (node:internal/async_hooks:465:18)\n at node:net:1044:9\n at processTicksAndRejections (node:internal/process/task_queues:78:11)", "config": { "transitional": { "silentJSONParsing": true, "forcedJSONParsing": true, "clarifyTimeoutError": false }, "transformRequest": [ null ], "transformResponse": [ null ], "timeout": 0, "xsrfCookieName": "XSRF-TOKEN", "xsrfHeaderName": "X-XSRF-TOKEN", "maxContentLength": -1, "maxBodyLength": -1, "env": {}, "headers": { "Accept": "application/json, text/plain, */*", "Content-Type": "application/json", "User-Agent": "axios/0.27.2", "Content-Length": 143 }, "method": "post", "url": "/", "data": "{\"getTransactionDetailsRequest\":{\"merchantAuthentication\":{\"name\":\"[my-login-id]\",\"transactionKey\":\"[my-transaction-key]\"},\"transId\":\"60201341313\"}}" }, "code": "EAFNOSUPPORT", "status": null } ```
2
answers
0
votes
75
views
asked 10 days ago

Lambda random long execution while running QLDB query

I have a lambda triggered by a SQS FIFO queue when there are messages on this queue. Basically this lambda is getting the message from the queue and connecting to QLDB through a VPC endpoint in order to run a simple SELECT query and a subsequent INSERT query. The table selected by the query has a index for the field used in the where condition. Flow (all the services are running "inside" a VPC): `SQS -> Lambda -> VPC interface endpoint -> QLDB` Query SELECT: `SELECT FIELD1, FIELD2 FROM TABLE1 WHERE FIELD3 = "ABCDE"` Query INSERT: `INSERT INTO TABLE1 .....` This lambda is using a shared connection/session on QLDB and this is how I'm connecting to it: ``` import { QldbDriver, RetryConfig } from 'amazon-qldb-driver-nodejs' let driverQldb: QldbDriver const ledgerName = 'MyLedger' export function connectQLDB(): QldbDriver { if ( !driverQldb ) { const retryLimit = 4 const retryConfig = new RetryConfig(retryLimit) const maxConcurrentTransactions = 1500 driverQldb = new QldbDriver(ledgerName, {}, maxConcurrentTransactions, retryConfig) } return driverQldb } ``` When I run a load test that simulates around 200 requests/messages per second to that lambda in a time interval of 15 minutes, I'm starting facing a random long execution for that lambda while running the queries on QLDB (mainly the SELECT query). Sometimes the same query retrieves data around 100ms and sometimes it takes more than 40 seconds which results in lambda timeouts. I have changed lambda timeout to 1 minute but this is not the best approch and sometimes it is not enough too. The VPC endpoint metrics are showing around 250 active connections and 1000 new connections during this load test execution. Is there any QLDB metric that could help to identify the root cause of this behavior? Could it be related to some QLDB limitation (like the 1500 active sessions described here: https://docs.aws.amazon.com/qldb/latest/developerguide/limits.html#limits.default) or something related to concurrency read/write iops?
1
answers
0
votes
53
views
asked 13 days ago

API Gateway - How to accept Authorization with Bearer keyword - HTTP API

API Gateway HTTP usign Cognito requires JWT token to be included in Authorization Header. This is a problem when testing against Swagger Editor, which includes "Bearer" keyboard in Authorization Header. Is there a way to configure API Gateway to accept JWT with `Bearer` keyword? **OpenAPI Schema:** ``` securitySchemes: AwsOAuth2: type: oauth2 flows: implicit: authorizationUrl: https://auth.ourdomain.com/login scopes: aws.cognito.signin.user.admin: Gives you access to all the User Pool APIs that can be accessed using access tokens alone email: Grants access to the email and email_verified claims. This scope can only be requested with the openid scope. openid: Returns all user attributes in the ID token that are readable by the client. The ID token is not returned if the openid scope is not requested by the client. phone: Grants access to the phone_number and phone_number_verified claims. This scope can only be requested with the openid scope. profile: Grants access to all user attributes that are readable by the client. This scope can only be requested with the openid scope. x-amazon-apigateway-authorizer: identitySource: "$request.header.Authorization" jwtConfiguration: audience: - "xxxxxxxx" issuer: "https://cognito-idp.eu-west-1.amazonaws.com/eu-west-1_xxxxxxx" type: "jwt" security: - AwsOAuth2: [] ``` Generates following curl request in OpenAPI Swagger Editor: ``` curl -X 'GET' \ 'https://api.ourdomain.com/0.5/app-user/heyho' \ -H 'accept: application/json' \ -H 'Authorization: Bearer eyJraWQiOiJ1aVcwc3Exxxxxxxxxxxx' ``` Problem is, that this gets rejected by API Gateway HTTP when integrating with Cognito. It requires header like this (without Bearer): ``` -H 'Authorization: eyJraWQiOiJ1aVcwc3Exxxxxxxxxxxx' ```
1
answers
0
votes
45
views
asked 13 days ago

Latency in GET requests

Hello. I wrote a code in Python that extracts data from the FTX exchange using their API. I am running the code in an AWS instance (free plan, t2.micro), located very closeby to the servers of the exchange. The code is essentially an infinite loop. At each step, it sends 3 'GET' requests, elaborates the response, and then goes to the next step. For the first few hundred iterations, the latency (defined below at the end of the post) for each block of three requests is of the order of 0.3seconds. After some time, it starts to grow up, reaching values from 2 to 5 seconds. In my local computer, located in the US, the latency is pretty constant at 1 second. There are no ratelimits in the FTX API for `GET` requests, so I should not expect any limit from the server. Is AWS limiting the rate of `GET` requests that I can make? I am trying to understand the origin of this extra-latency. To do so, I have monitored the https data traffic with `tcpdump` and I have modified the python script so that it stops as soon as it experiences a latency > 2 seconds. In this way, I can isolate the last packets in the tcpdump output and try to understand the origin of the delay. However, I really don't know how to read the output (I uploaded it here https://pastebin.com/tAhcicPU). Can anyone help me to understand the origin of the latency? 104.18.33.31.443 is the IP of FTX server 172.31.9.8 is the IP of the machine where my code runs **Definition of latency used here**: I post the relevant part of the code where I compute the latency ``` latency=0 for pair in pairList: # pairList = ['BTC/USD','ETH/BTC','ETH/USD'] api=requests.get(f'https://ftx.com/api/markets/{pair}/orderbook?depth={20}') latency+=api.elapsed.total_seconds() return latency ``` So, it is the total sum of the latency returned by the requests.get for each request.
1
answers
0
votes
18
views
asked 13 days ago