Questions tagged with DevOps
Content language: English
Sort by most recent
I'm setting up a gitlab project, such that I can start a processing job ( for some data processing ) in an ec2 instance, this is an initial set up , but i'm planning to run some training for my regression models in these instance . for this, i have to store my aws user credentials in gitlab. is there a recommendentaion from aws ? or any articles/blogs/examples for such set up?
This was working fine till yesterday and suddenly my component stopped working when I did a revise deployment
Getting error ModuleNotFoundError: No module named '_awscrt'.
On my component artifacts-unarchived location I can see the python packages are available
I see the packages folder awscrt, awsiot, boto3, botocore all packages that needed but when I try import
from awsiot.greengrasscoreipc.clientv2 import GreengrassCoreIPCClientV2, getting ModuleNotFoundError: No module named '_awscrt'
I am using the awsiotsdk==1.9.2, I also tried installing awscrt==0.13.3, awsiot==0.1.3 but no luck still getting the error ModuleNotFoundError: No module named '_awscrt'.
Need advise to fix this issue.
I realise CodeCatalyst is in preview, but it would be really useful if the API could be expanded so that we can manage issues via the CLI.
As an example, I'm thinking that it would be useful for a workflow to be able to raise an issue if there was a problem with a new run, and assign it to the author of the associated commit.
This could also provide a solution to my request for manual authorisation as part of a workflow - i.e. before deploying to prod, check if a ticket exists to approve the deployment, if not create one, and if one does exist, wait until it is closed by someone in a list. This would mean we need be able to
* query if a issue exists based on id, associated commit id, or other criteria. If the issue does exists, the details of the ticket, including the issue status should be returned
* create a query
asked 23 days ago
Hello All!
So I'm on my first foray in dockerized applications and would like to deploy a cron task on fargate to run the docker image.
**Some background:**
The docker image is published to ECR via CI/CD pipeline from my gitlab account. The image is a slim buster python, and I utilize docker in docker to build the image. The push to ECR is successful.
The basics of the image are:
1. Create a python env with the libraries specified in the requirements.txt
2. Import libraries into __main__.py along with all the user-defined funcs on a func.py file
__main__.py runs a series of DDL sqls and DML sqls passed to our snowflake via the snowflake python SDK which stages data. Then the staged data is looped through in batches with each batch sent to an external api (smarty streets) and the GET results sent back to snowflake (basically standardizing free text address fields is the point of the application).
**My Goal**
Have this run on a chron via a scheduled Fargate task once per day.
**My Problem**
I have a fargate cluster defined, I have a task definition, and a schedule rule (Event Bridge) created, but nothing happens. When I try to manually run the task definition nothing happens. The task shows that its in a pending status then disappears, and there are no logs given either (telling me something is failing) and nothing happens on the scheduled time from the event bridge rule I created. On the SQL DMLs I have a start|finish time for this process being logged to a snowflake table, which is showing that there is nothing being sent to snowflake. I'm sure there is something very stupid that I'm missing on getting a task to run the docker container but am not finding it. Looked through a lot of tutorials on youtube, but they all seem to be web app driven for a Fargate *service *(typically a flask), and not a simple burst ETL job on a schedule like I"m trying to do.
Any examples on how to tie this all together???
**Added Notes**
I'm pretty sure the subnets I chosen have their IPs whitelisted on our snowflake, but I am expecting the API to fail as its external. But at this point I just want to trigger the docker container/
Hi,
I changed my Elastic Beanstalk Configuration last time in December, but now I logged today and i can see new elastic beanstalk configuration panel and i can not update in it anything. If i change something and click "continue" or "apply" there is nothing changed and i can not see any information about error. May you help me what is wrong?
Hi,
We have an existing AWS Amplify application we would like to publish in AWS Marketplace as a service that customers can deploy on their infrastructure. How can I do this?
Thanks,
Hi, we use Azure Repos as our Version Control System. We would like to train ML models on our local env . Whenever ML code is pushed, we want to use Azure DevOps pipelines to integrate with SageMaker pipelines. We have couple of questions?
1. Can we track the experimentation done on local env using SageMaker API? If yes, can you share any relevant articles as I have come across mostly outdated code snippets online for that purpose.
2. What is the recommended method to integrate Azure Repos to SageMaker pipelines through Azure DevOps? As code gets merged, Azure DevOps pipeline gets triggered then the SageMaker pipeline for training and deployment gets executed. That's the intended approach. Appreciate any articles or references.
Thanks
I am Deploying EKS cluster using CDK pipeline in Typescript
This is May Cluster Stack,
import { PhysicalName, Stack, StackProps } from "aws-cdk-lib";
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import { Vpc } from "aws-cdk-lib/aws-ec2";
import * as eks from 'aws-cdk-lib/aws-eks';
import { Cluster } from "aws-cdk-lib/aws-eks/lib/cluster";
import { AccountRootPrincipal,Role } from "aws-cdk-lib/aws-iam";
import { Construct } from "constructs";
export interface DevOpsClusterStackProps extends StackProps {
cluster:Cluster,
vpc:Vpc,
}
export class DevOpsClusterStack extends Stack {
public readonly cluster: eks.Cluster;
accountId = this.account;
clusterName = "DevOpsCluster"
Role: Role;
/* Cluster Role Defined */
constructor(scope: Construct, id: string, props: DevOpsClusterStackProps) {
super(scope, id, props);
this.accountId = this.account;
this.clusterName = "DevOpsCluster";
const clusterAdmin = new Role(this, 'clusterAdmin', {
assumedBy: new AccountRootPrincipal(),
roleName: "clusterAdmin",
});
/* Cluster Configuration */
const cluster = new eks.Cluster(this, 'DevOpsCluster', {
clusterName: "DevOpsCluster",
version: eks.KubernetesVersion.V1_23,
defaultCapacity: 3,
mastersRole: clusterAdmin,
defaultCapacityInstance: ec2.InstanceType.of(ec2.InstanceClass.M5, ec2.InstanceSize.LARGE),
vpc:props.vpc,
vpcSubnets: [{
subnetType: ec2.SubnetType.PUBLIC
}],
});
cluster.addAutoScalingGroupCapacity('spot-group', {
instanceType: new ec2.InstanceType('m5.xlarge'),
});
if (Stack.of(this).region==this.region)
this.Role = createDeployRole(this, `for-1st-region`, cluster);
this.cluster = cluster;
}
}
function createDeployRole(scope: Construct, id: string, cluster: eks.Cluster): Role {
const role = new Role(scope, id, {
roleName: PhysicalName.GENERATE_IF_NEEDED,
assumedBy: new AccountRootPrincipal()
});
cluster.awsAuth.addMastersRole(role);
return role;
}
export interface PipelineStack extends StackProps {
Cluster: eks.Cluster,
Role: Role,
}
and
This is My Pipeline Stack to Deploy this cluster using Pipeline
import { Stack, StackProps, Stage } from 'aws-cdk-lib';
import * as codecommit from 'aws-cdk-lib/aws-codecommit';
import { CodePipeline, CodePipelineSource } from 'aws-cdk-lib/pipelines';
import * as pipelines from 'aws-cdk-lib/pipelines';
import { Construct } from 'constructs';
import { VpcStack } from './vpc-stack';
import { Cluster } from 'aws-cdk-lib/aws-eks/lib/cluster';
import { DevOpsClusterStack } from '../lib/devops-cluster-stack';
class DevelopmentStage extends Stage {
cluster: Cluster;
constructor(scope: Construct, id: string, props: StackProps) {
super(scope, id, props);
const vpcStack = new VpcStack(this, "VpcStack", {});
const ClusterStack = new DevOpsClusterStack (this, 'DevOpsCluster',{vpc:vpcStack.vpc , cluster:this.cluster});
}
}
/**
* Create a CI/CD pipelines for cluster deployment
*/
export class PipelineStack extends Stack {
cluster: Cluster;
static cluster: Cluster;
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
/**
* Here we provide pipeline start point as a Codecommit Soursecode to Create a CI/CD pipelines for cluster deployment
*/
const repository = codecommit.Repository.fromRepositoryName(this, 'Repository', 'CDK-Typescript-Project');
const source = CodePipelineSource.codeCommit(repository, "feature/create-eks-cluster")
const pipeline = new CodePipeline(this, 'Pipeline', {
pipelineName: 'EKS-CICD-Pipeline',
synth: new pipelines.ShellStep('Synth', {
input: source,
installCommands: ['npm i -g npm@latest',"npm install -g typescript"],
commands: [
'npm ci',
'npm run build',
'npx cdk synth',
]
})
});
// Developemnt stage This could include things like EC2 instances and more, depending on the needs of the application being developed.
const devStage = new DevelopmentStage(this, "Development", {
});
pipeline.addStage(devStage);
}
}
Also I have Created Separate VPC Stack
import { App, Stack, StackProps } from "aws-cdk-lib";
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import { IpAddresses } from "aws-cdk-lib/aws-ec2";
import { Construct } from "constructs";
/**
* Create a VPC with one Public and one Private Subnet
*/
export class VpcStack extends Stack {
public readonly vpc: ec2.Vpc;
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const vpc = new ec2.Vpc(this, 'vpc', {
natGateways: 1,
ipAddresses: IpAddresses.cidr("10.1.0.0/16"),
subnetConfiguration: [
{
name: 'Public',
subnetType: ec2.SubnetType.PUBLIC,
},
{
name: 'Private',
subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
}
],
maxAzs: 2
});
this.vpc = vpc;
}
}
/*I am receiving following error while deploying the Cluster Stack*/
ERROR is like: instance Fails to Join Kubernetes Cluster
DevOpsClusterNodegroupDefaultCapacity90B6204B
CREATE_FAILED
Resource handler returned message: "[Issue(Code=NodeCreationFailure, Message=Instances failed to join the kubernetes cluster, ResourceIds=[i-02c060ccb6d6e8c6f, i-048feaa20bfdca377, i-0a7a4184599e60cd2])] (Service: null, Status Code: 0, Request ID: null)" (RequestToken: e94890a6-5074-b4a3-a4e3-916cf510ef8a, HandlerErrorCode: GeneralServiceException)
Hi,
I need help to use a function that concatenates a parameter with a static value within a remediation rule.
I have the following custom Conformance Pack, where I want to define the *AutomationAssumeRole* as a parameter.
```
Parameters:
ParamAutomationAssumeRole:
Default: ComplianceRemediation
Type: String
Resources:
Ec2SecurityGroupAttachedToEni:
Type: AWS::Config::ConfigRule
Properties:
ConfigRuleName: ec2-security-group-attached-to-eni
Scope:
ComplianceResourceTypes:
- AWS::EC2::SecurityGroup
Source:
Owner: AWS
SourceIdentifier: EC2_SECURITY_GROUP_ATTACHED_TO_ENI
Ec2SecurityGroupAttachedToEniRemediation:
DependsOn: Ec2SecurityGroupAttachedToEni
Type: "AWS::Config::RemediationConfiguration"
Properties:
ConfigRuleName: ec2-security-group-attached-to-eni
ResourceType: "AWS::EC2::SecurityGroup"
TargetId: "AWSConfigRemediation-DeleteUnusedSecurityGroup"
TargetType: "SSM_DOCUMENT"
TargetVersion: "1"
Parameters:
GroupId:
ResourceValue:
Value: "RESOURCE_ID"
AutomationAssumeRole:
StaticValue:
Values:
Fn::Sub:
"arn:aws:iam::${AWS::AccountId}:role/${ParamAutomationAssumeRole}"
```
Based on this [doc](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-join.html), I could use the "Fn::Sub:" function, but the Conformance Pack deployment fails with the following error:
I'm not sure what I'm doing wrong here. Any help is much appreciated.
Thanks!
If I deploy component in my device with AWS Greengrass so, how can I know that my remote device has downloaded that code. Does it download in its system or anything else.
Thank you!
Nik
I have deleted the component from AWS cloud, from local device, and built another device in another region as well, but it shows device unhealthy, and showing error like this.... . Can anyone tell the reason behind this.
Thank you
Nik
Hi,
I have Python app that process data and I am running it in Docker. Locally, it is working fine. When I tried for a first time to run on AWS App Runner I needed health check, so I expose 80 port of container and install nginx that listen on that port. When I run container locally I can access the nginx homepage. I was trying to setup health check in AWS App Runner. I am using TCP check on port 80. For some reasons it is failing all the time. Could you please give me some suggestions what could be the issue?
Thank you in advance!