- Newest
- Most votes
- Most comments
Hi Matvey,
I understand that when you deploy Py lambda using SAM from command line with "sam build" it creates a .aws-sam/build with the code in it, and also .aws-sam/deps/900e2021-84ce-4093-a603-5ef231a0f4d3 with all dependencies in it as well. When using "sam package", it packages only the code from the build directory and dependencies from [1] separately. Both files are then being uploaded into S3 but only once the code is deployed. When doing the same action in PyCharm it does not seperate them i.e. all tree is packaged and deployed as one hence the Lambda is functional. This seems that the the two packages Lambda is installation is geared towards a layered approach whereas in the second part is a virtual environment deployed separately. Also you would essentially like to understand the cost of using Lambda layer as an approach both from a monetary perspective as well as a runtime execution level. Please feel free to correct if there is any misunderstanding.
Let us first have a look at the cost of using Lambda from both monetary and runtime execution level. From a cost level. "There is no cost for storing the layer -- or using the layer. However, the layer storage does take up space in your total Lambda storage" [2]. In terms of performance let us take two identical lambda functions written in python3.8 using one without layer (directly packaging the function with its dependencies) and one without packaging it dependencies meaning using a layer instead. This test will use the requests library version "2.22.0". [3] and the code [4] will be in Python3.8 both tests will be conducted with cold starts. [5] The code is designed to ping 2 websites at random and get a response. Please note Lambda is a highly distributed system with various components so test results may change when testing on your side.
TEST 1 - Lambda function without layer
Duration: 78.77 ms Memory Size: 128 MB Max Memory Used: 49 MB Init Duration: 404.33 ms
TEST 2 - Lambda function with layer
Duration: 448.31 ms Memory Size: 128 MB Max Memory Used: 49 MB Init Duration: 207.40 ms
It is quite evident that TEST 1 produced the less duration time so that can conclude using the above experiment that yes adding a layer does give a trade off in duration. However there are advantages to using Lambda layers as outlined here. [6]
As for the part about "Now, when I use "sam package", it packages only the code from the build directory and dependencies from 900e2021-84ce-4093-a603-5ef231a0f4d3 separately." I tested this using SAM but keeping my directory structure as the image below. I ran the following command inside the package "sam-app-sam-test":
a. sam package --s3-bucket<bucket_name> --output-template-file packaged-template.yaml b. sam deploy --template-file packaged-template.yaml --stack-name <stack_name> --capabilities CAPABILITY_IAM (stack name came from using helloworldexample when running "sam init" and choosing helloworld example in ZIP format)
The above gave me a single packge in S3 named a2457d556ccffadd639b45c7604684f2. When I uploaded it in Lambda I got the following (refer to image below).
Hence, I am not sure how the original package looked for you when using "sam build" and "sam package". Additionally, as I don't have access and knowledge of how the PyCharm IDE works under the hood with SAM I am unable to comment on why PyCharm doesn't seperate the packages. Hence, I kindly request that you please create a case to AWS in order to facilitate a detailed response. This will allow us to understand the PyCharm directory structure and how it works with SAM as well as how the directory structure looks from the command line. I you want SAM to use a layer here is structure. [7] The structure I am using is as follow. [8]
As always, you are welcome to add any questions related to this re:Post.
References:
[1] 900e2021-84ce-4093-a603-5ef231a0f4d3
[2] https://repost.aws/questions/QUoSWSKhamS1-ZvDWao3drKQ/lambda-layer-costs
[3] https://pypi.org/project/requests/2.22.0/
[4]
import random
import requests
def lambda_handler(event, context):
websites = [
"http://example.com ",
"http://amazon.com ",
# Add more websites as needed
]
random_website = random.choice(websites)
try:
response = requests.get(random_website)
return {
'statusCode': response.status_code,
'body': f'Response from {random_website}: {response.text}'
}
except Exception as e:
return {
'statusCode': 500,
'body': f'Error pinging {random_website}: {str(e)}'
}
[5] https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/
[6] https://docs.aws.amazon.com/lambda/latest/dg/chapter-layers.html
[7] https://docs.aws.amazon.com/lambda/latest/dg/layers-sam.html
[8]
AWSTemplateFormatVersion: '2010-09-09' Transform: AWS::Serverless-2016-10-31 Description: 'sam-app-sam-test
Sample SAM Template for sam-app-sam-test
' Globals: Function: Timeout: 3 MemorySize: 128 Resources: HelloWorldFunction: Type: AWS::Serverless::Function Properties: CodeUri: s3://sampacakgeapptest/a2457d556ccffadd639b45c7604684f2 Handler: app.lambda_handler Runtime: python3.7 Events: HelloWorld: Type: Api Properties: Path: /hello Method: get Metadata: SamResourceId: HelloWorldFunction Outputs: HelloWorldApi: Description: API Gateway endpoint URL for Prod stage for Hello World function Value: Fn::Sub: https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/ HelloWorldFunction: Description: Hello World Lambda Function ARN Value: Fn::GetAtt: - HelloWorldFunction - Arn HelloWorldFunctionIamRole: Description: Implicit IAM Role created for Hello World function Value: Fn::GetAtt: - HelloWorldFunctionRole - Arn
You have the deps folder because you probably used sam sync. When using it, sam packages all dependencies in a layer so that function changes will be faster as the dependencies are packages in a different place.
You do not need to use sam package, sam deploy does it automatically as part of the deployment process.
Hi,
I have tried to package and deploy the way you described with the majority of my parameters coming from the samconfig.toml file and also from the command line after. It ended up with incomplete installation and lambda not being able to run. Here is my config:
version = 0.1
[default]
[default.global.parameters]
stack_name = "jwt-auth-20-1"
profile = "abcd"
region="eu-west-1"
s3_bucket = "abcd"
capabilities = "CAPABILITY_IAM"
template_file = "#####/JTW-authorizer/template.yaml"
# resolve_s3 = true
[default.build.parameters]
cached = false
parallel = true
[default.validate.parameters]
lint = true
[default.deploy.parameters]
confirm_changeset = false
[default.package.parameters]
[default.sync.parameters]
watch = false
[default.local_start_api.parameters]
warm_containers = "EAGER"
[default.local_start_lambda.parameters]
warm_containers = "EAGER"
Resulted in the following YAML:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 'JTW-authorizer
Globals:
Function:
Timeout: 3
MemorySize: 128
Resources:
JWTFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: s3://abcd/e6b2c55e2d49e6608e98515aff35fbdf
Handler: lambda_function.lambda_handler
Runtime: python3.8
Architectures:
- x86_64
Environment:
Variables:
PRINCIPAL: 12345678
SECRET_KEY: 12323434534
Metadata:
SamResourceId: JWTFunction
Outputs:
LambdaAuthFunction:
Description: JWT Lambda Function ARN
Value:
Fn::GetAtt:
- JWTFunction
- Arn
The log:
mteplov@scorpio:~/PycharmProjects/JTW-authorizer$ sam package --force-upload --debug
2023-08-21 15:17:08,517 | Config file location: JTW-authorizer/samconfig.toml
2023-08-21 15:17:08,519 | Loading configuration values from [default.['package'].parameters] (env.command_name.section) in config file at 'JTW-authorizer/samconfig.toml'...
2023-08-21 15:17:08,519 | Configuration values successfully loaded.
2023-08-21 15:17:08,520 | Configuration values are: {'stack_name': 'jwt-auth-20-1', 'profile': 's3-cloudformation-push', 'region': 'eu-west-1', 's3_bucket': 'abcd', 'capabilities': 'CAPABILITY_IAM', 'template_file': 'JTW-authorizer/template.yaml'}
2023-08-21 15:17:08,522 | Using SAM Template at JTW-authorizer/template.yaml
2023-08-21 15:17:08,550 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-08-21 15:17:08,552 | Using config file: samconfig.toml, config environment: default
2023-08-21 15:17:08,552 | Expand command line arguments to:
2023-08-21 15:17:08,553 | --template_file=JTW-authorizer/template.yaml --force_upload --s3_bucket=abcd
2023-08-21 15:17:08,567 | No Parameters detected in the template
2023-08-21 15:17:08,583 | There is no customer defined id or cdk path defined for resource JWTFunction, so we will use the resource logical id as the resource id
2023-08-21 15:17:08,584 | 0 stacks found in the template
2023-08-21 15:17:08,584 | No Parameters detected in the template
2023-08-21 15:17:08,592 | There is no customer defined id or cdk path defined for resource JWTFunction, so we will use the resource logical id as the resource id
2023-08-21 15:17:08,743 | There is no customer defined id or cdk path defined for resource JWTFunction, so we will use the resource logical id as the resource id
2023-08-21 15:17:08,744 | Sam customer defined id is more priority than other IDs. Customer defined id for resource JWTFunction is JWTFunction
Uploading to e6b2c55e2d49e6608e98515aff35fbdf 3003 / 3003 (100.00%)
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: 'JTW-authorizer
'
Globals:
Function:
Timeout: 3
MemorySize: 128
Resources:
JWTFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: s3://abcd/e6b2c55e2d49e6608e98515aff35fbdf
Handler: lambda_function.lambda_handler
Runtime: python3.8
Architectures:
- x86_64
Environment:
Variables:
PRINCIPAL: 12345678
Metadata:
SamResourceId: JWTFunction
Outputs:
LambdaAuthFunction:
Description: JWT Lambda Function ARN
Value:
Fn::GetAtt:
- JWTFunction
- Arn
2023-08-21 15:17:09,007 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-08-21 15:17:09,010 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-08-21 15:17:09,012 | Unable to find Click Context for getting session_id.
mteplov@scorpio:~/PycharmProjects/JTW-authorizer$ sam deploy --debug
2023-08-21 15:21:51,513 | Config file location: /JTW-authorizer/samconfig.toml
2023-08-21 15:21:51,515 | Loading configuration values from [default.['deploy'].parameters] (env.command_name.section) in config file at '/JTW-authorizer/samconfig.toml'...
2023-08-21 15:21:51,515 | Configuration values successfully loaded.
2023-08-21 15:21:51,516 | Configuration values are: {'stack_name': 'jwt-auth-20-1', 'profile': 'abcd, 'region': 'eu-west-1', 's3_bucket': 'abcd, 'capabilities': 'CAPABILITY_IAM', , 'template_file': 'JTW-authorizer/template.yaml', 'confirm_changeset': False}
2023-08-21 15:21:51,518 | Using SAM Template at JTW-authorizer/template.yaml
2023-08-21 15:21:51,535 | Using config file: samconfig.toml, config environment: default
2023-08-21 15:21:51,536 | Expand command line arguments to:
2023-08-21 15:21:51,536 | --template_file=/JTW-authorizer/template.yaml --fail_on_empty_changeset --on_failure=ROLLBACK --stack_name=jwt-auth-20-1 --s3_bucket=abcd --capabilities=['CAPABILITY_IAM']
2023-08-21 15:21:51,668 | No Parameters detected in the template
2023-08-21 15:21:51,677 | There is no customer defined id or cdk path defined for resource JWTFunction, so we will use the resource logical id as the resource id
2023-08-21 15:21:51,678 | 0 stacks found in the template
2023-08-21 15:21:51,678 | No Parameters detected in the template
2023-08-21 15:21:51,686 | There is no customer defined id or cdk path defined for resource JWTFunction, so we will use the resource logical id as the resource id
2023-08-21 15:21:51,755 | There is no customer defined id or cdk path defined for resource JWTFunction, so we will use the resource logical id as the resource id
2023-08-21 15:21:51,756 | Sam customer defined id is more priority than other IDs. Customer defined id for resource JWTFunction is JWTFunction
2023-08-21 15:21:51,961 | File with same data already exists at e6b2c55e2d49e6608e98515aff35fbdf, skipping upload
Deploying with following values
===============================
Stack name : jwt-auth-20-1
Region : eu-west-1
Confirm changeset : False
Disable rollback : False
Deployment s3 bucket : abcd
Capabilities : ["CAPABILITY_IAM"]
Parameter overrides : {}
Signing Profiles : {}
Initiating deployment
=====================
2023-08-21 15:21:51,986 | No Parameters detected in the template
2023-08-21 15:21:51,993 | Sam customer defined id is more priority than other IDs. Customer defined id for resource JWTFunction is JWTFunction
2023-08-21 15:21:51,994 | 0 stacks found in the template
2023-08-21 15:21:51,994 | No Parameters detected in the template
2023-08-21 15:21:52,001 | Sam customer defined id is more priority than other IDs. Customer defined id for resource JWTFunction is JWTFunction
2023-08-21 15:21:52,002 | 1 resources found in the stack
Uploading to d595a846bbc48202695c19e4edc7a209.template 726 / 726 (100.00%)
Waiting for changeset to be created..
CloudFormation stack changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType Replacement
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Also, I don't see any layer with the dependencies being created - only insights and code guru code are in.
Any ideas?
Regards
Interestingly enough, via PyCharm, deployment also has stopped with the new template.yaml generated - also it only deployed the code. I recovered from the GIT, and from what I see, there is one line real difference: via SAM CLI from the generated template, it was: Properties: CodeUri: s3://abcd/e6b2c55e2d49e6608e98515aff35fbdf in Pycharm it is: Properties: CodeUri: jwt_handler/
I guess, somehow it actually deploys directly and not via S3. Can this be?
Relevant content
- Accepted Answerasked 10 months ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 9 months ago
- AWS OFFICIALUpdated 3 years ago
