Deploy files to ec2 without adding ip to its security group. Possible?
Hello, The question is how to upload GitLab CI artifacts on EC2 servers (just simple EC2 with ssh daemon enabled) without running command aws ec2 authorize-security-group-ingress and next command to remove its ip back. Cause sometimes GitLab builds are failing and running next time add ip command cause an error that this rule (ip) already exists in security group! Because of this i was forced to add a sophisticated check on ip presence in security group, like if ip exists then; run build and deploy; else add ip and run cmd + deploy; done. Because of this, gitlab-ci.yml is long because i need to add such a logic to any env and deployment job. I would like to avoid this and remove all extra code. So is there any more elegant approach how to upload the artifacts to AWS EC2 without adding each time its ip to security group? Like service role, or something? I am using now public ip to connect, cause GitLab runner resides in another VPC than all the ec2 servers. So there is no any approach how to connect to them via internal IP except VPC peering, but, I do not want to add peering cause one of the VPC is production, so it is not secure because it should be isolated. And to run GitLab instance in each VPC is not an option as well because it will not be costs efficient. So if to recap the question it will be: how to deploy to several VPC from one GitLab server avoiding adding each time GitLab ip in each security group but keep it secure and cost-efficient? Thanks.
- Newest
- Most votes
- Most comments
Hi,
Yes, it is possible by using 3 properties:
- AWS Lambdas are by default connected to the Internet if you don't connect them to any of your VPC
- AWS service endpoints are reached outbound from the client as http request
- you can define Service endpoints in your VPC to privately reach any service.
So, proposal:
- You create a service endpont for Lambda and S3 in your VPC
- your EC2 instances calls Lambda endpoint with Invoke() to call a Lambda function that your develop
- This Lambda function interacts with Github via the Internet to collect artifacts and upload them in S3
- This Lambda notifies via SNS
- The EC2 instance receives the notification and collect the artifacts via the S3 Service Endpoint
This way, no need to add and remove ip address and rules from secgroup
Best,
Didier
I would recommend using AWS Codepipeline with CodeDeploy.
- You build your Artifact in Github which in turns uploads it to an S3 bucket.
- This upload will trigger AWS Codepipeline via EventBridge
- CodePipeline issues a command to CodeDeploy which then tells your Group of EC2s to download the code from an S3 bucket.
- Using the appsec file you can script what the EC2 does with these files it has downloaded like to stop/start services etc
This way there is no need for github to have any direct access to production EC2s. You can back this with endpoints and an S3 gateway to keep all traffic within the VPC if required also.
Another option is to use GitHub actions directly with CodeDeploy and not using CodePipeline following this article https://aws.amazon.com/blogs/devops/integrating-with-github-actions-ci-cd-pipeline-to-deploy-a-web-app-to-amazon-ec2/
Some articles.
https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-CodeDeploy.html https://docs.aws.amazon.com/codedeploy/latest/userguide/welcome.html
Relevant content
- asked 5 months ago
AWS OFFICIALUpdated 8 months ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 4 months ago
- AWS OFFICIALUpdated a year ago
I dont think the approach is clear enough. The EC2s will not know when to call the lambda function. The code/application is built when ever there is a code release and the aritifact would be stored/uploaded some where at that point. The Lambda will have to run on a schedule. The CICD pipeline can upload direct to S3 so no need to have an intermediatory. The lambda function will not be able to grab an artifact from a build and copy to S3 either as it would not know which to grab if its run on a schedule. How would the EC2 recieve the SNS notification?