Adding Prometheus metrics endpoint to AWS Blue/Green deployment with existing target groups

0

We have a microservice running in AWS ECS with OpenTelemetry instrumentation that exposes Prometheus metrics on a separate port. Currently, our deployment setup is:

Dev & Staging: Rolling update deployment strategy

Successfully configured with additional target group for Prometheus metrics port Load balancer routes metrics requests to the dedicated Prometheus port/target group

We just did something like

aws ecs update-service \
  --cluster app-staging \
  --service app-service-staging \
  --load-balancers \
    "targetGroupArn=arn:aws:elasticloadbalancing:ap-east-1:**********:targetgroup/staging-app-service/*********,containerName=app,containerPort=3000" \
    "targetGroupArn=arn:aws:elasticloadbalancing:ap-east-1:*********:targetgroup/app-staging-otel-tg/*********,containerName=app,containerPort=9465" \
  --force-new-deployment

Production: Blue/Green deployment strategy

Already using two target groups for the main application traffic Need to expose Prometheus metrics similarly to dev/staging

Question: What's the recommended approach to add Prometheus metrics endpoint support in production while maintaining our blue/green deployment strategy? Specifically:

How should we configure additional target groups for the metrics endpoint? How should we handle the routing rules in the load balancer? What considerations should we keep in mind for the blue/green deployment process with this additional endpoint?

Any insights or best practices would be appreciated.

1 Answer
0

To add Prometheus metrics endpoint support in your production environment while maintaining the blue/green deployment strategy, you'll need to make some adjustments to your current setup. Here's a recommended approach:

  1. Configure additional target groups:

    • Create two new target groups for the Prometheus metrics endpoint, one for the blue environment and one for the green environment.
    • These target groups should be configured to use the port where your Prometheus metrics are exposed (9465 in your case).
  2. Load balancer routing rules:

    • Add new listeners to your Application Load Balancer for the Prometheus metrics port.
    • Configure routing rules to direct traffic to the appropriate Prometheus metrics target group based on your blue/green deployment state.
    • You may need to use path-based or host-based routing to differentiate between application traffic and metrics traffic.
  3. ECS service configuration:

    • Update your ECS service definition to include the new Prometheus metrics target groups.
    • You'll need to specify four target groups in total: two for the main application (blue and green) and two for the Prometheus metrics (blue and green).
  4. Blue/Green deployment process considerations:

    • Ensure that your CodeDeploy application is updated to manage the additional target groups for Prometheus metrics.
    • During a deployment, CodeDeploy should update both the application target groups and the Prometheus metrics target groups simultaneously.
    • Verify that your health checks for the Prometheus metrics endpoint are properly configured to ensure accurate deployment results.
    • Consider implementing a traffic shifting strategy that includes both the application and metrics endpoints.
  5. Security considerations:

    • Ensure that the security groups for your ECS tasks allow inbound traffic on the Prometheus metrics port from the load balancer.
    • Consider implementing authentication or access controls for the metrics endpoint to prevent unauthorized access.
  6. Monitoring and testing:

    • Thoroughly test the blue/green deployment process with the new Prometheus metrics configuration in a staging environment before implementing in production.
    • Monitor the metrics collection during and after deployments to ensure continuity of data.

By following these recommendations, you should be able to successfully integrate Prometheus metrics endpoint support into your production environment while maintaining your blue/green deployment strategy. Remember to test thoroughly and monitor closely during the initial implementations to ensure everything works as expected.
Sources
Deploying an Amazon ECS service using a blue/green deployment - Amazon Elastic Container Service
Set up and configure Prometheus metrics collection on Amazon ECS clusters - Amazon CloudWatch

profile picture
answered 23 days ago
profile picture
EXPERT
reviewed 22 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions