Questions tagged with DevOps
Content language: English
Sort by most recent
Tips for managing several ML project with similar framework
Hi, I'm working on an end-to-end ml project which, for the moment, goes from training (it takes already processed train/val/test data from an S3 bucket) to deploy, passing through hyperparameter tunning. This project has been developed on SageMaker Studio and in the beginning, I decided to keep track on the project with a github's repository. So, my work has certain degree of maturity: I was able to successfully train, tune, deploy and infer over a dataset. But the troubles came when I try to replicate this work for another dataset (a new project with a similar framework). Let's say that I had a client "A" for which I developed this project and now I have a new client "B" with similar requirements than client A. I'm looking for the best way of "copy & paste" the project considering the following: a) I would like to keep working on the repository. The idea here is the repo been like a project's template ir order to clone the repository, make some few corrections (changing model's name, working bucket, etc) and then execute tuning, evaluation and deploy. b) There's a lot of changes and improvements that I should make in the future. So, I'd like those changes been reflected on both projects. If anyone could give me some tips, guidelines, share his experience with something like this I would be very grateful. Regards! :D
I can not invoke a Lambda function from AWS Lex
Hello, I would like to invoke a lambda function from AWS Lex v2, after the intent has been fulfilled. The needed resources (bot, lambda, rol, etc) must be defined using Cloudformation. This is apparently an easy task, just by defining a code hook on the Cloudformation template. But AFAIK it does not work, and the lambda is never invoked. I validate the lambda invocation using Cloudwatch logs. If you would like to reproduce the environment, here you can find the Cloudformation template: https://pastebin.com/i9mzZMZE Just deploy the template (lambda code is embedded). What am I missing? Thanks in advance. Roberto.
Using CodeArtifact as a dependency caching layer only
# Background Our team is currently using Codebuild/codepipeline for CI/CD. We are looking to speed-up CI/CD run-times by caching NPM downloads via CodeArtifact. We're not looking to make use of any CodeBuild caching solutions e.g. local cache as in the future we will be moving our CI/CD to different toolchain. However, we will still need a caching layer for our dependencies. # Use Case We want to use CodeArtifact as the caching layer for our NPM dependencies (another option could be JFrog). However, we won't be pushing any artifacts to CodeArtifact, just simply downloading publicly available dependencies from the NPM registry via CodeArtifact leveraging the caching capabilities of CodeArtifact. # Question Is our use-case of simply using CodeArtifact as a NPM caching layer appropriate and usable across different CI/CD solutions? For example, if it will work with CodeBuild today, will it work equally as well with BitBucket pipelines?
How to Push Step Functions to Repo
Hello, My team and I are working on implementing Repos for our ETL functions (Glue, Step Functions and Lambdas). I was wondering if there was a way to push changes to step Functions to a Repo? I know I can copy and paste code out the Edit window, but that feels pretty manual. I was hoping for a less manual way of having Version Control over Step Functions changes, as our step Functions are pretty extensive and would be better off editing in the console and then pushing the update with the click of a few buttons. Thanks!
Long running EC2s with Cloudformation and ServiceCatalog
My organisation runs with a number of fairly tight controls for how we can build our environments. One of these is that infrastructure must be deployed via the service catalog using products that have been validated before being published for use. This means we rely on Cloudformation to build the EC2 servers we use. My use case for EC2 is as long running compute nodes within a cluster that store data. This requirement comes from the software in use. This means they are not sacrificial like a web server or app server might be. As such using autoscaling groups for the EC2s is not ideal, as I don't want the EC2s being terminated unexpectedly as this can cause data loss, and also new nodes need to be configured into the cluster which would mean a loss of capacity. What I'm looking for is suggestions on how to manage the EC2s using Cfn and ServiceCatalog without having to push new versions of the Service Catalog products when we want to make changes to the EC2s, such as volume sizes, or instance types. One option could be custom macros and various lambdas but this gets complex quickly. Terraform works well for this but is not an option due to the restrictions I mention around service catalog. TIA....
Elastic Beanstalk upgrade major PHP version (e.g. 7.4 > 8.1) without creating new environment
Is there a way to easily upgrade an Elastic Beanstalk environment's major PHP version (e.g. 7.4 > 8.1) from the console without creating a new environment? Like this, but without doing it from the command line: (this was from a couple years ago) https://dev.to/paulmicheli/upgrade-aws-elastic-beanstalk-php-major-version-5do This should be such a simple option for AWS to provide - but I'm not seeing a way to do it: ![Sreenshot showing deprecated](/media/postImages/original/IM35Vjk4INRtGPPUu0lZiZDQ)
SES verified identity of type Domain via CDK
Hi team! is there a way to create/validate a domain identity in SES via CDK, on the CSK documentation, I only found `EmailIdentity` construct > https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ses-readme.html#email-identity should I use a custom construct? > https://github.com/mooyoul/aws-cdk-ses-domain-identity I want to create an SES verified identity of type Domain via CDK code. Thank you!
Generate AWS CloudFormation Template with Existing Resources
I found this article to build the cloudformation script for the existing infrastructure. But I could not find Cloudformer option in the drop down while creating the template. Is AWS still supporting this feature? Do we have any other options to build the script for the existing infrastructure? https://aws.amazon.com/blogs/devops/building-aws-cloudformation-templates-using-cloudformer/
Log Retention Lambda
Whenever we create a lambda using cdk, it also creates one log retention lambda. Is it by default ? I understand that logs in cloudwatch will retain for long period of time. To save cost, we might want to set the retention time to lower date. Can we not set this as single/global config ? Creating a log retention lambda for every Lambda function that we create just causing some noise. Can some expert in this forum help us - 1. Whether we can delete this lambda ? 2. Are there a way to set the log related config in single place ? 3. Are there any CDK config to change so that we can avoid creating this lambda in future function ? Please find the log retention lambda gist link https://gist.github.com/gvnavin/71ec6310480834f632370b882ab6a581