Datadog Forwarder Lambda function is failing while deploying through CloudFormation Template even though the value for ReservedConcurrency is increased from 100 to 1000.

1

Hi AWS, I am trying to integrate AWS with Datadog using CloudFormation method but while doing so my Forwarder Lambda function is failing which is responsible for collecting logs, metrics etc.

I am attaching the template for the Forwarder function which is integrated into the main template as Nested Stack and also pasting the error for the same:

AWSTemplateFormatVersion: "2010-09-09"
Description: Pushes logs, metrics and traces from AWS to Datadog.
Mappings:
  Constants:
    DdForwarder:
      Version: 3.59.0
      LayerVersion: 24
Parameters:
  DdApiKey:
    Type: String
    NoEcho: true
    Default: ""
    Description: The Datadog API key, which can be found from the APIs page (/account/settings#api). It will be stored in AWS Secrets Manager securely. If DdApiKeySecretArn is also set, this value is ignored.
  DdApiKeySecretArn:
    Type: String
    AllowedPattern: "arn:.*:secretsmanager:.*"
    Default: "arn:aws:secretsmanager:DEFAULT"
    Description: The ARN of the secret storing the Datadog API key, if you already have it stored in Secrets Manager. You must store the secret as a plaintext, rather than a key-value pair.
  DdSite:
    Type: String
    Default: datadoghq.com
    Description: Define your Datadog Site to send data to. Possible values are `datadoghq.com`, `datadoghq.eu`, `us3.datadoghq.com`, `us5.datadoghq.com` and `ddog-gov.com`.
    AllowedPattern: .+
    ConstraintDescription: DdSite is required
  FunctionName:
    Type: String
    Default: DatadogForwarder
    Description: The Datadog Forwarder Lambda function name. DO NOT change when updating an existing CloudFormation stack, otherwise the current forwarder function will be replaced and all the triggers will be lost.
  MemorySize:
    Type: Number
    Default: 1024
    MinValue: 128
    MaxValue: 3008
    Description: Memory size for the Datadog Forwarder Lambda function
  Timeout:
    Type: Number
    Default: 120
    Description: Timeout for the Datadog Forwarder Lambda function
  TagsCacheTTLSeconds:
    Type: Number
    Default: 300
    Description: TTL (in seconds) for the Datadog tags cache
  ReservedConcurrency:
    Type: Number
    Default: 100
    Description: Reserved concurrency for the Datadog Forwarder Lambda function
  LogRetentionInDays:
    Type: Number
    Default: 90
    Description: CloudWatch log retention for logs generated by the Datadog Forwarder Lambda function
  SourceZipUrl:
    Type: String
    Default: ""
    Description: DO NOT CHANGE unless you know what you are doing. Override the default location of the function source code.
  InstallAsLayer:
    Type: String
    Default: true
    Description: Whether to use the layer-based installation flow. Set to false to use our legacy installation flow, which installs a second function that copies the forwarder code from Github to an S3 bucket. Defaults to true.
    AllowedValues:
      - true
      - false
  LayerARN:
    Type: String
    Default: ""
    Description: ARN for the layer containing the forwarder code. If empty, the script will use the version of the layer the forwarder was published with.
  DdTags:
    Type: String
    Default: ""
    Description: Add custom tags to forwarded logs, comma-delimited string, no trailing comma, e.g., env:prod,stack:classic
  DdFetchLambdaTags:
    Type: String
    Default: true
    AllowedValues:
      - true
      - false
    Description: Let the forwarder fetch Lambda tags using GetResources API calls and apply them to logs, metrics and traces. If set to true, permission tag:GetResources will be automatically added to the Lambda execution IAM role. The tags are cached in memory so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The forwarder increments the aws.lambda.enhanced.get_resources_api_calls metric for each API call made.
  DdFetchLogGroupTags:
    Type: String
    Default: true
    AllowedValues:
      - true
      - false
    Description: Let the forwarder fetch Log Group tags using ListTagsLogGroup and apply them to logs, metrics and traces. If set to true, permission logs:ListTagsLogGroup will be automatically added to the Lambda execution IAM role. The tags are cached in memory so that they'll only be fetched when the function cold starts or when the TTL (1 hour) expires. The forwarder increments the aws.lambda.enhanced.list_tags_log_group_api_call metric for each API call made.
  DdUseTcp:
    Type: String
    Default: false
    AllowedValues:
      - true
      - false
    Description: By default, the forwarder sends logs using HTTPS through the port 443. To send logs over an SSL encrypted TCP connection, set this parameter to true.
  DdNoSsl:
    Type: String
    Default: false
    AllowedValues:
      - true
      - false
    Description: Disable SSL when forwarding logs, set to true when forwarding logs through a proxy.
  DdUrl:
    Type: String
    Default: ""
    Description: The endpoint URL to forward the logs to, useful for forwarding logs through a proxy
  DdPort:
    Type: String
    Default: ""
    Description: The endpoint port to forward the logs to, useful for forwarding logs through a proxy
  DdSkipSslValidation:
    Type: String
    Default: false
    AllowedValues:
      - true
      - false
    Description: Send logs over HTTPS, while NOT validating the certificate provided by the endpoint. This will still encrypt the traffic between the forwarder and the log intake endpoint, but will not verify if the destination SSL certificate is valid.
  RedactIp:
    Type: String
    Default: false
    AllowedValues:
      - true
      - false
    Description: Replace text matching \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} with xxx.xxx.xxx.xxx
  RedactEmail:
    Type: String
    Default: false
    AllowedValues:
      - true
      - false
    Description: Replace text matching [a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+ with xxxxx@xxxxx.com
  DdScrubbingRule:
    Type: String
    Default: ""
    Description: Replace text matching the supplied regular expression with xxxxx (default) or DdScrubbingRuleReplacement (if supplied). Log scrubbing rule is applied to the full JSON-formatted log, including any metadata that is automatically added by the Lambda function. Each instance of a pattern match is replaced until no more matches are found in each log. Note, using inefficient regular expression, such as `.*`, may slow down the Lambda function.
  DdScrubbingRuleReplacement:
    Type: String
    Default: ""
    Description: Replace text matching DdScrubbingRule with the supplied text
  ExcludeAtMatch:
    Type: String
    Default: ""
    Description: DO NOT send logs matching the supplied regular expression. If a log matches both the ExcludeAtMatch and IncludeAtMatch, it is excluded. Filtering rules are applied to the full JSON-formatted log, including any metadata that is automatically added by the function. Note, using inefficient regular expression, such as `.*`, may slow down the Lambda function.
  IncludeAtMatch:
    Type: String
    Default: ""
    Description: Only send logs matching the supplied regular expression and not excluded by ExcludeAtMatch. Note, using inefficient regular expression, such as `.*`, may slow down the Lambda function.
  DdMultilineLogRegexPattern:
    Type: String
    Default: ""
    Description: Use the supplied regular expression to detect for a new log line for multiline logs from S3, e.g., use expression "\d{2}\/\d{2}\/\d{4}" for multiline logs beginning with pattern "11/10/2014".
  DdForwardLog:
    Type: String
    Default: true
    AllowedValues:
      - true
      - false
    Description: Set to false to disable log forwarding, while continuing to forward other observability data, such as metrics and traces from Lambda functions.
  DdUseCompression:
    Type: String
    Default: true
    AllowedValues:
      - true
      - false
    Description: Set to false to disable log compression. Only valid when sending logs over HTTP.
  DdUsePrivateLink:
    Type: String
    Default: false
    AllowedValues:
      - true
      - false
    Description: DEPRECATED, DO NOT CHANGE. See README.md for details. Set to true to deploy the Forwarder to a VPC and send logs, metrics, and traces via AWS PrivateLink. When set to true, must also set VPCSecurityGroupIds and VPCSubnetIds.
  DdUseVPC:
    Type: String
    Default: false
    AllowedValues:
      - true
      - false
    Description: Set to true to deploy the Forwarder to a VPC and send logs, metrics, and traces via a proxy. When set to true, must also set VPCSecurityGroupIds and VPCSubnetIds.
  DdHttpProxyURL:
    Type: String
    Default: ""
    Description: "Sets the standard web proxy environment variables HTTP_PROXY and HTTPS_PROXY. These are the url endpoints your proxy server exposes. Don't use this in combination with AWS Private Link. Make sure to also set DdSkipSslValidation to true."
  DdNoProxy:
    Type: String
    Default: ""
    Description: "Sets the standard web proxy environment variable NO_PROXY. It is a comma-separated list of domain names that should be excluded from the web proxy."
  VPCSecurityGroupIds:
    Type: CommaDelimitedList
    Default: ""
    Description: Comma separated list of VPC Security Group Ids. Used when DdUsePrivateLink or DdUseVPC is enabled.
  VPCSubnetIds:
    Type: CommaDelimitedList
    Default: ""
    Description: Comma separated list of VPC Subnet Ids. Used when DdUsePrivateLink or DdUseVPC is enabled.
  DdCompressionLevel:
    Type: Number
    Default: 6
    AllowedValues: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    Description: Set the compression level from 0 (no compression) to 9 (best compression) when sending logs.
  DdMaxWorkers:
    Type: Number
    Default: 20
    Description: Set the max number of workers sending logs concurrently.
  PermissionsBoundaryArn:
    Type: String
    Default: ""
    Description: ARN for the Permissions Boundary Policy
  AdditionalTargetLambdaArns:
    Type: CommaDelimitedList
    Default: ""
    Description: Comma separated list of lambda ARNs that get invoked asynchronously with the same input event
  DdApiUrl:
    Type: String
    Default: ""
    Description: The endpoint URL to forward the metrics to, useful for forwarding metrics through a proxy
  DdTraceIntakeUrl:
    Type: String
    Default: ""
    Description: The endpoint URL to forward the traces to, useful for forwarding traces through a proxy
  DdForwarderBucketName:
    Type: String
    Default: ""
    Description: The name of the forwarder bucket to create. If not provided, AWS will generate a unique name.
Conditions:
  IsAWSChina:
    Fn::Equals:
      - Ref: AWS::Partition
      - "aws-cn"
  IsGovCloud:
    Fn::Equals:
      - Ref: AWS::Partition
      - "aws-us-gov"
  UseZipCopier:
    Fn::Or:
      - Condition: IsAWSChina
      - Fn::And:
          - Fn::Equals: [!Ref InstallAsLayer, "false"]
          - Fn::Not:
              - Condition: SetLayerARN
  CreateDdApiKeySecret:
    Fn::Equals:
      - Ref: DdApiKeySecretArn
      - "arn:aws:secretsmanager:DEFAULT"
  SetFunctionName:
    Fn::Not:
      - Fn::Equals:
          - Ref: FunctionName
          - "DatadogForwarder"
  SetSourceZipUrl:
    Fn::Not:
      - Fn::Equals:
          - Ref: SourceZipUrl
          - ""
  SetS3SourceZip:
    Fn::Equals:
      - !Select [0, !Split ["/", !Ref SourceZipUrl]]
      - "s3:"
  SetDdTags:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdTags
          - ""
  SetDdUseTcp:
    Fn::Equals:
      - Ref: DdUseTcp
      - true
  SetDdNoSsl:
    Fn::Equals:
      - Ref: DdNoSsl
      - true
  SetDdUrl:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdUrl
          - ""
  SetDdPort:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdPort
          - ""
  SetRedactIp:
    Fn::Equals:
      - Ref: RedactIp
      - true
  SetRedactEmail:
    Fn::Equals:
      - Ref: RedactEmail
      - true
  SetDdScrubbingRule:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdScrubbingRule
          - ""
  SetDdScrubbingRuleReplacement:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdScrubbingRuleReplacement
          - ""
  SetExcludeAtMatch:
    Fn::Not:
      - Fn::Equals:
          - Ref: ExcludeAtMatch
          - ""
  SetIncludeAtMatch:
    Fn::Not:
      - Fn::Equals:
          - Ref: IncludeAtMatch
          - ""
  SetDdMultilineLogRegexPattern:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdMultilineLogRegexPattern
          - ""
  SetDdSkipSslValidation:
    Fn::Equals:
      - Ref: DdSkipSslValidation
      - true
  SetDdFetchLambdaTags:
    Fn::Equals:
      - Ref: DdFetchLambdaTags
      - true
  SetDdFetchLogGroupTags:
    Fn::Equals:
      - Ref: DdFetchLogGroupTags
      - true
  CreateS3BucketForTags:
    Fn::Or:
      - Fn::Equals:
          - Ref: DdFetchLogGroupTags
          - true
      - Fn::Equals:
          - Ref: DdFetchLambdaTags
          - true
  SetDdUsePrivateLink:
    Fn::Equals:
      - Ref: DdUsePrivateLink
      - true
  SetDdUseVPC:
    Fn::Equals:
      - Ref: DdUseVPC
      - true
  SetDdHttpProxyURL:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdHttpProxyURL
          - ""
  SetDdNoProxy:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdNoProxy
          - ""
  SetLayerARN:
    Fn::Not:
      - Fn::Equals:
          - Ref: LayerARN
          - ""
  UseVPC:
    Fn::Or:
      - Condition: SetDdUsePrivateLink
      - Condition: SetDdUseVPC
  SetDdForwardLog:
    Fn::Equals:
      - Ref: DdForwardLog
      - false
  SetDdUseCompression:
    Fn::Equals:
      - Ref: DdUseCompression
      - false
  SetDdCompressionLevel:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdCompressionLevel
          - 6
  SetDdMaxWorkers:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdMaxWorkers
          - 20
  SetPermissionsBoundary:
    Fn::Not:
      - Fn::Equals:
          - Ref: PermissionsBoundaryArn
          - ""
  SetAdditionalTargetLambdas:
    Fn::Not:
      - Fn::Equals:
          - Fn::Join: ["", !Ref AdditionalTargetLambdaArns]
          - ""
  SetDdApiUrl:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdApiUrl
          - ""
  SetDdTraceIntakeUrl:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdTraceIntakeUrl
          - ""
  SetDdForwarderBucketName:
    Fn::Not:
      - Fn::Equals:
          - Ref: DdForwarderBucketName
          - ""
Rules:
  MustSetDdApiKey:
    Assertions:
      - Assert:
          Fn::Or:
            - Fn::Not:
                - Fn::Equals:
                    - Ref: DdApiKey
                    - ""
            - Fn::Not:
                - Fn::Equals:
                    - Ref: DdApiKeySecretArn
                    - "arn:aws:s

Error is: Resource handler returned message: "Specified ReservedConcurrentExecutions for function decreases account's UnreservedConcurrentExecution below its minimum value of [10]. (Service: Lambda, Status Code: 400, Request ID: e1f10e3a-7b20-4d80-ad8c-0d58299bfd58)" (RequestToken: 10e1a046-9b8c-b47c-a925-f13a6614ed1a, HandlerErrorCode: InvalidRequest)

profile picture
asked 2 years ago256 views
1 Answer
0

Can you please change default value of parameter ReservedConcurrency from 100 to lower value like 10. You could also try checking your account quota limits and availability

AWS
EXPERT
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions