Should we combine or split application and Infrastructure code for microservice projects deployed in EKS.

We are designing the architecture of a microservice solution where,

1. We will have tens of container-based microservices deployed in the EKS cluster.
2. Our microservices are java spring boot applications
3. There will be base infrastructure components such as VPC, Subnets, CloudFront, EKS, ALB, etc.
4. There will be infrastructure components shared across multiple microservices such as RDS, EFS, API Gateway, etc.
5. Each microservice could have its own dedicated cloud infrastructure (DynamoDB, SQS, etc.).
6. We will be using AWS CDK for deploying all the infrastructure components (3,4 and 5).
7. In addition to infrastructure components we will have Kubernetes configurations such as helm, configmap, policy as code (PoC) components related to OPA policies, Monitoring As code components etc. associated with each microservice.

We are trying to figure out an appropriate Git repository structure here. A few things we already decided to do are the following.

• Have CDK code for the base infra (3) and shared infra (4) in a single GitRepo and manage the deployment of it through some infra pipeline separately.
• Each springboot microservice will have its own application Git Repositories which will be tied up with its own codepipeline CI process that builds the container and registers in ECR upon commits at the main branch (trunk-based model).

The confusion we have here is where to store the infrastructure code that is associated with each microservice (items 5 and 7 in the above list). We thought of two options here and we find merits and demerits to both the options.

Option#1 : Keep the infra codes ( 5 & 7) related to the microservices in the application repository associated with the microservice.

Pros

• Easy to sync application and infra changes together as it's in a single repo. Infra and app changes can be deployed together and roll back together as well.

Cons

• Since app and infra codes are stored in the same repo, any infra change could trigger the app build pipeline. And similarly, an app code change could trigger automated infra deployment pipelines (if we have an automated CDK deployment pipeline). Considering app build is part of the CI process and infra deployment falls into the CD part, it would be tough to manage the CI/CD pipeline efficiently.
• Typically CI jobs would initiate code scans, code coverage checks, container scans etc. we wouldn't want to trigger those CI stages for simple changes in the infra segment of the code such as a change in configuration item or change in monitoring as a code component etc.
• In the standard CI/CD pipeline, after the docker container is built and registered in ECR repo, the container tag will be updated in the helm configuration (which is now stored in the app repository), this could trigger the CI flow again.
• In CD pipeline we typically execute automated E2E test, API test etc. One wouldn't want them to be triggered for simple infra changes alone.

Option#2 : Each microservice will have two repositories. One for keeping the application code, and another one for keeping all the infrastructure code (5 & 7) associated with it.

Pros

• Isolation between infra and application codes enables us to segregate CI and CD processes without conflict.

Cons

• It would be challenging to sync application and infra changes as they are now in two different repositories.

Our question is ,

• Looking at various articles, there are multiple ways to do it, but we would request advice from AWS community what is the best repo design that suits our unique case explained above.
• Do you have any specific advice on how infra associated with a microservice (CDK) and app container deployment in EKS can be integrated in a single deployment pipeline

Reference materials used https://devops.stackexchange.com/questions/12803/best-practices-for-app-and-infrastructure-code-repositories https://www.youtube.com/watch?v=MeU5_k9ssrs https://www.youtube.com/watch?v=f5EpcWp0THw