By using AWS re:Post, you agree to the Terms of Use
/DevOps/

Questions tagged with DevOps

Sort by most recent
  • 1
  • 90 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

AWs trigger EventBatchingCondition/BatchWindow is not optional

Hi team, I have a glue workflow : trigger (type = "EVENT") => trigger a glue job (to take data from S3 and push them to MySQL RDS) I configured the glue Triggering criteria to kickoff the glue job after 5 events were received. in the console it says : > Specify the number of events received or maximum elapsed time before firing this trigger. > Time delay in seconds (optional) on AWS documentation it says also it's not required : ``` BatchWindow Window of time in seconds after which EventBridge event trigger fires. Window starts when first event is received. Type: Integer Valid Range: Minimum value of 1. Maximum value of 900. Required: No ``` So I want only my trigger to be triggered only and only after 5 events are received and not depending on: Time delay in seconds (optional). actually, the Time delay in seconds (optional) is set to 900 by default and my job is started after 900s even if there are no 5 events received. that's not the behaviour we want. We want ONLY the job to be started after x events are received. I tried via the console to edit the trigger and remove the 900s for the Time delay in seconds (optional) input but I can't save it until I put a value on it. it says it's optional but it doesn't seem to be. is there a workaround to make the trigger not take account of Time delay in seconds (optional)? and only be launched when it received x events and nothing else. right now the behaviour I have is that my job is triggered after 900s, we want to eliminate this case and let the job be triggered only and only if there is x event received and nothing else. how can I make the Time delay in seconds (optional) input optional, because now the console forces me to put a value in there? thank you.
1
answers
0
votes
6
views
asked 22 days ago

Fail to start an EC2 task on ECS

Hi there i am trying to start a task which uses gpu on my instance. EC2 is already added to a cluster but it failed to start, here is the error: ``` status: STOPPED (CannotStartContainerError: Error response from dae) Details Status reason CannotStartContainerError: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr Network bindings - not configured ``` ec2: setup ``` Type: AWS::EC2::Instance Properties: IamInstanceProfile: !Ref InstanceProfile ImageId: ami-0d5564ca7e0b414a9 InstanceType: g4dn.xlarge KeyName: tmp-key SubnetId: !Ref PrivateSubnetOne SecurityGroupIds: - !Ref ContainerSecurityGroup UserData: Fn::Base64: !Sub | #!/bin/bash echo ECS_CLUSTER=traffic-data-cluster >> /etc/ecs/ecs.config echo ECS_ENABLED_GPU_SUPPORT=true >> /etc/ecs/ecs.config ``` Dockerfile ``` FROM nvidia/cuda:11.6.0-base-ubuntu20.04 ENV NVIDIA_VISIBLE_DEVICES all ENV NVIDIA_DRIVER_CAPABILITIES compute,utility # RUN nvidia-smi RUN echo 'install pip packages' RUN apt-get update RUN apt-get install python3.8 -y RUN apt-get install python3-pip -y RUN ln -s /usr/bin/python3 /usr/bin/python RUN pip3 --version RUN python --version WORKDIR / COPY deployment/video-blurring/requirements.txt /requirements.txt RUN pip3 install --upgrade pip RUN pip3 install --user -r /requirements.txt ## Set up the requisite environment variables that will be passed during the build stage ARG SERVER_ID ARG SERVERLESS_STAGE ARG SERVERLESS_REGION ENV SERVER_ID=$SERVER_ID ENV SERVERLESS_STAGE=$SERVERLESS_STAGE ENV SERVERLESS_REGION=$SERVERLESS_REGION COPY config/env-vars . ## Sets up the entry point for running the bashrc which contains environment variable and ## trigger the python task handler COPY script/*.sh / RUN ["chmod", "+x", "./initialise_task.sh"] ## Copy the code to /var/runtime - following the AWS lambda convention ## Use ADD to preserve the underlying directory structure ADD src /var/runtime/ ENTRYPOINT ./initialise_task.sh ```
0
answers
0
votes
2
views
asked 24 days ago

Scheduled Action triggering at time specified in another action

I have a CloudFormation setup with Scheduled Actions to autoscale services based on times. There is one action that scales up to start the service, and another to scale down to turn it off. I also occasionally add an additional action to scale up if a service is needed at a different time on a particular day. I'm having an issue where my service is being scaled down instead of up when I specify this additional action. Looking at the console logs I get an event that looks like: ``` 16:00:00 -0400 Message: Successfully set min capacity to 0 and max capacity to 0 Cause: scheduled action name ScheduleScaling_action_1 was triggered ``` However the relevant part of the CloudFormation Template for the Scheduled Action with the name in the log has a different time, e.g.: ``` { "ScalableTargetAction": { "MaxCapacity": 0, "MinCapacity": 0 }, "Schedule": "cron(0 5 ? * 2-5 *)", "ScheduledActionName": "ScheduleScaling_action_1" } ``` What is odd is that the time this action is triggering matches exactly with the Schedule time for another action. E.g. ``` { "ScalableTargetAction": { "MaxCapacity": 1, "MinCapacity": 1 }, "Schedule": "cron(00 20 ? * 2-5 *)", "ScheduledActionName": "ScheduleScaling_action_2" } ``` I am using CDK to generate the CloudFormation template, which doesn't appear to allow me to specify a timezone. So my understanding is that the times here should be UTC. What could cause the scheduled action to trigger at the incorrect time like this?
1
answers
0
votes
4
views
asked a month ago

High-Traffic, Load-Balanced Wordpress Site - Optimal DevOps setup for deployment?

TLDR: I inherited a Wordpress site that I now manage that had a DevOps deployment pipeline that worked when the site was low to medium traffic, but now the site consistently gets high-traffic and I'm trying to improve the deployment pipeline. The site I inherited uses Lightsail instances and a Lightsail load balancer in conjunction with one RDS database instance and an S3 bucket for hosted media. When I inherited the site, the deployment pipeline from the old developer was: *Scale site down to one instance, make changes to that one instance, once changes are complete, clone that updated instance as many times as you need* This worked fine when the site mostly ran with only one instance except during peak traffic times. However, now at all times we have 3-5 instances as even our "off-peak" traffic is really high requiring multiple instances. I'd like to improve the deployment pipeline to allow for deployment during peak-traffic times without issues. I'm worried about updating multiples instances behind the load balancer one by one sequentially because we have Session Persistence disabled to allow for more evenly distributed load balancing. And I'm worried a user hopping to different instances that have a different functions.php file will cause issues. Should I just enable session persistence when I want to make updates and sequentially updates instances behind the load balancer one by one? Or is there a better suited solution? Should I move to a containers setup? I'm admittedly a novice with AWS so any help is greatly appreciated. Really just looking for general advice and am confident I can figure out how to implement a suggested best-practice solution. Thanks!
1
answers
0
votes
7
views
asked a month ago

Amplify export infrastructures does not work with CDK V2

According to [Amplify documentation](https://docs.amplify.aws/cli/usage/export-to-cdk/) and [this official blog post](https://aws.amazon.com/blogs/mobile/export-amplify-backends-to-cdk-and-use-with-existing-deployment-pipelines/), it is possible to export infrastructures from Amplify then import into CDK. However, I try with CDK V2 and it does not work. I got error when installing **npm i @aws-amplify/cdk-exported-backend@latest**. CDK V2 **Construct** is not compatible with the **Construct** in aws-amplify/cdk-exported-backend, I think. So how to export Amplify infrastructure to CDK V2? Thank you! 1. Here is my package.json of CDK ``` { "name": "amplify-export-cdk", "version": "0.1.0", "bin": { "amplify-export-cdk": "bin/amplify-export-cdk.js" }, "scripts": { "build": "tsc", "watch": "tsc -w", "test": "jest", "cdk": "cdk" }, "devDependencies": { "@types/jest": "^26.0.10", "@types/node": "10.17.27", "jest": "^26.4.2", "ts-jest": "^26.2.0", "aws-cdk": "2.18.0", "ts-node": "^9.0.0", "typescript": "~3.9.7" }, "dependencies": { "aws-cdk-lib": "2.18.0", "constructs": "^10.0.0", "source-map-support": "^0.5.16" } } ``` 2. Here is errors when installing ``` npm ERR! code ERESOLVE npm ERR! ERESOLVE unable to resolve dependency tree npm ERR! npm ERR! While resolving: amplify-export-cdk@0.1.0 npm ERR! Found: constructs@10.0.108 npm ERR! node_modules/constructs npm ERR! constructs@"^10.0.0" from the root project npm ERR! npm ERR! Could not resolve dependency: npm ERR! peer constructs@"^3.2.27" from @aws-amplify/cdk-exported-backend@0.0.5 npm ERR! node_modules/@aws-amplify/cdk-exported-backend npm ERR! @aws-amplify/cdk-exported-backend@"0.0.5" from the root project npm ERR! npm ERR! Fix the upstream dependency conflict, or retry npm ERR! this command with --force, or --legacy-peer-deps npm ERR! to accept an incorrect (and potentially broken) dependency resolution. ```
0
answers
1
votes
3
views
asked a month ago

Slow lambda responses when bigger load

Hi, Currently, I'm doing load testing using Gatling and I have one issue with my lambdas. I have two lambdas one is written in Java 8 and one is written in Python. I'm using Gatling for my load testing and I have a test where I'm doing one request with 120 concurrent users then I'm ramping them from 120 to 400 users in 1 minute, and then Gatling is doing requests with 400 constants users per second for 2 minutes. There is a weird behavior in these lambdas because the responses are very high. In the lambdas there is no logic, they are just returning a String. Here are some screenshots of Gatling reports: [Java Report][1] [Python Report][2] I can add that I did some tests when Lambda is warm-up and there is the same behaviour as well. I'm using API Gateway to run my lambdas. Do you have any idea why there is such a big response time? Sometimes I'm receiving an HTTP error that says: i.n.h.s.SslHandshakeTimeoutException: handshake timed out after 10000ms Here is also my Gatling simulation code: public class OneEndpointSimulation extends Simulation { HttpProtocolBuilder httpProtocol = http .baseUrl("url") // Here is the root for all relative URLs .acceptHeader("text/html,application/xhtml+xml,application/json,application/xml;q=0.9,*/*;q=0.8") // Here are the common headers .acceptEncodingHeader("gzip, deflate") .acceptLanguageHeader("en-US,en;q=0.5") .userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0"); ScenarioBuilder scn = scenario("Scenario 1 Workload 2") .exec(http("Get all activities") .get("/dev")).pause(1); { setUp(scn.injectOpen( atOnceUsers(120), rampUsersPerSec(120).to(400).during(60), constantUsersPerSec(400).during(Duration.ofMinutes(1)) ).protocols(httpProtocol) ); } } I also checked logs and turned on the X-ray for API Gateway but there was nothing there. The average latency for these services was 14ms. What can be the reason for that slow Lambda responses? [1]: https://i.stack.imgur.com/sCx9M.png [2]: https://i.stack.imgur.com/SuHU0.png
0
answers
0
votes
6
views
asked a month ago

Lambda function updating cannot be made atomic with RevisionId

A number of Lambda API calls allow for a RevisionId argument to ensure that the operation only continues if the current revision of the Lambda function matches, very similar to an atomic Compare-And-Swap operation. However, this RevisionId appears to be useless for performing some atomic operations, for the following reason: Suppose I want to update a function's code and then publish it, in 2 separate steps (I know it can be done in 1 step, but this does not interest me, because I cannot set the description of a published version in 1 update/publish step...it must be done in 2 steps). The [update_function_code](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#Lambda.Client.update_function_code) call returns a RevisionId that corresponds to the "in progress" update of the function. This RevisionId cannot be used because it will change once the function becomes active/updated. This new RevisionId can only be obtained by [get_function](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#Lambda.Client.get_function). Update code -> RevisionId A (in progress) -> RevisionId B (updated/active) -> Get Function -> RevisionId B -> Publish Function There exists a race condition due to the fact that I must call `get_function` in order to get the current RevisionId before I continue with publishing my function. This race condition makes it impossible create an atomic sequence of operations that includes a `update_function_code` operation, because the RevisionId that it returns cannot be relied on, and has to be refreshed with a `get_function` call. Concurrently, another operation could change the RevisionId, and you wouldn't know, because you're depending on `get_function` to return an unknown RevisionId.
1
answers
0
votes
3
views
asked 2 months ago

Issue creating Lambda function layer versions in parallel process

Hi We are using Terraform (v0.13.5 with AWS provider "hashicorp/aws v3.38.0") to deploy AWS resources into our accounts. Some of these resources are Lambda functions with Lambda layers. We make use of certain automated processes (Gitlab pipelines) to run those deployments. We can change several Lambda functions at the same time. We use the same Lambda layer for ALL the Lambda functions, but create different versions of that layer with different code (ZIP files) and attach each version to a concrete Lambda function. Lately we realized that when modifying several Lambda functions at the same time, the code in the different versions of the same layer is mixed!!! So the code that should go for a concrete layer version is also appearing in other versions created at the same time. For example: * when we modify several Lambda functions at the same time (let's say L001 and L002), two new versions of layer MYLAY are created for each, and the corresponding version is linked to each of the modified Lambda functions. So we have L001 with MYLAY-001 and L002 with MYLAY-002. This is how we expect it, so fine so far * Each version of the layer should have its own code (different ZIP files) * We have detected that the code for MYLAY-001 is also appearing in MYLAY-002, even though the ZIP files used to create those versions are different!!! So from my point of view, it seems that the way in which **AWS is creating the layer versions for the same layer is not compatible with parallel creation**. Can anyone confirm of shed some light on how AWS is creating those versions? I guess the best approach considering the previous is to use different layer for each Lambda function, indeed. Thanks in advance and best regards Luis
1
answers
0
votes
3
views
asked 2 months ago

CloudFormation create-change-set Tags propagation

Hi everyone, I'm creating multiple resources using CloudFormation and now I have the need to tag them, the thing is that I'm using the CLI to invoke cloudformation and I'm passing a JSON file with the tags as parameter. I'm using gitlab CI/CD to create the CF stack also I want to review the changes before applying them, hence what I ended doing was creating a change set first (Update or Create, depending if the stack already exists), then if everything looks good I execute the change set. This is more or less what I'm doing in the pipeline: 1. I create the change set with: > aws cloudformation create-change-set --change-set-name MyChangeSet --change-set-type <<CREATE or UPDATE>> --stack-name MyCFStack --tags <<file_with_tags.json>> 2. If everything looks good then I'm executing the change set with the following command: > aws cloudformation execute-change-set --stack-name MyCFStack --change-set-name MyChangeSet CloudFormation creates the stack and tag it appropriately, the problem is that **does NOT propagate the tags to the resources**. In the documentation (https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudformation/create-change-set.html) says that CloudFormation should propagate the tags to the resources: > --tags (list) > Key-value pairs to associate with this stack. CloudFormation also propagates these tags to resources in the stack. You can specify a maximum of 50 tags. I know that using "create-stack" or "update-stack" you can pass the tag parameter too, but as you can see, my approach allows me to create the change set first and review the changes, even if it's the first time I'm creating the stack. Has anybody experienced something similar? Were you able to pass the tags using "create-change-set" and then execute? Am I missing something? I appreciate any help. Jorge.
1
answers
0
votes
4
views
asked 2 months ago

MSK Custom Configuration using Cloudformation

Hi AWS Users, I am trying to spin up a MSK cluster with a custom MSK configuration using my serverless app. I wrote the cloudformation template for the generation of the MSK Cluster and was able to successfully bring it up. I recently saw that AWS added cloudformation template of `AWS::MSK::Configuration`. [1] I was trying that out to create a custom configuration. The Configuration requires a `ServerProperties`key that is usually a PlainText in AWS console. An example of Server Properties: ``` auto.create.topics.enable=true default.replication.factor=2 min.insync.replicas=2 num.io.threads=8 num.network.threads=5 num.partitions=10 num.replica.fetchers=2 replica.lag.time.max.ms=30000 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 unclean.leader.election.enable=true zookeeper.session.timeout.ms=18000 ``` `AWS::MSK::Configuration` accepts base64 (api functionality) and I have been trying to implement this. I am using the cloudformation `Fn::Base64` functionality. e.g: ``` Resources: ServerlessMSKConfiguration: Type: AWS::MSK::Configuration Properties: ServerProperties: Fn::Base64: auto.create.topics.enable=true ``` This gives me back a 400 error during deploy. ``` Resource handler returned message: "[ClientRequestToken: xxxxx] Invalid request body (Service: Kafka, Status Code: 400, Request ID: 1139d840-c02d-4fdb-b68c-cee93673d89d, Extended Request ID: null)" (RequestToken: xxxx HandlerErrorCode: InvalidRequest) ``` Can someone please help me format this ServerProperties properly, not sure how to give the proper base64 string in the template. Any help is much appreciated. [1] - [https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html](MSK::Configuration)
0
answers
0
votes
5
views
asked 2 months ago

Run (custom) Keycloak 17 Docker Image on AWS Beanstalk

I've been trying to get a Keycloak Docker image to run on a Beanstalk environment for the last week without success. My Dockerfile looks like this: FROM quay.io/keycloak/keycloak:17.0.0 as builder ENV KC_DB=postgres RUN /opt/keycloak/bin/kc.sh build FROM quay.io/keycloak/keycloak:17.0.0 COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/ WORKDIR /opt/keycloak ENV KC_HTTP_ENABLED=true ENV KC_HOSTNAME_STRICT=false ENV KC_DB_POOL_INITIAL_SIZE=1 ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start-dev"] The Dockerimage runs fine on localhost (`docker run --rm -p 8080:8080 --env-file env.txt my/keycloak`). http://localhost:8080/ shows a start page. The chosen Plattform is a "Docker running on 64bit Amazon Linux 2/3.4.12". I upload the image to Amazon ECR and load it in a Beanstalk instance with the following Dockerrun.aws.json: { "AWSEBDockerrunVersion": "1", "Image": { "Name": "0815.eu-central-1.amazonaws.com/my/keycloak:latest" }, "Ports": [ {"ContainerPort": "8080"} ] } I have saved the necessary environment variables: KC_DB, KC_DB_PASSWORD, KC_DB_POOL_INITIAL_SIZE, KC_DB_SCHEMA, KC_DB_URL, KC_DB_USERNAME, KC_HOSTNAME_STRICT, KC_HTTP_ENABLED, KEYCLOAK_ADMIN, KEYCLOAK_ADMIN_PASSWORD As a load balancer, I set up the Classic Load Balancer with a listener from 8080/HTTP to 8080/HTTP. Now when I try to call the Beanstalk URL (http://Keycloak0815.eu-central-1.elasticbeanstalk.com:8080) I get a 503 error status. A look at the logs shows no abnormalities. Keycloak has started successfully within the docker. What am I doing wrong? What else do I need to configure to get access to the Docker image? I'm grateful for any further information.
0
answers
0
votes
1
views
asked 2 months ago
  • 1
  • 90 / page