Self-service Developer Platform architecture on EKS with Crossplane
Hello aws re:Post
I am currently designing a solution that would allow my users to be able to provision workloads on demand through a click-to-deploy interface. In order to minimize costs, I thought of an architecture as follows :
2 K8S clusters (1 on-premise and 1 on AWS)
On-premise : Developer Platform management cluster
- nodePool mgmt
- Crossplane
- Customer cluster definition
- Catalog definition
- nodePool tools
- APIs
- Prometheus
- Grafana
- ...
- nodePool mgmt
AWS : Customer resources cluster
- nodePool customerName (one per customer)
- ns mgmt (created on customer register)
- ns tenant (one per tenant : production, development, ..)
- nodePool customerName (one per customer)
If you don't know Crossplane, you can look at the documentation here : https://crossplane.io/docs/v1.7/ but it is a bit like AWS Controllers for Kubernetes. It provides infrastructure abstraction as Kubernetes CRDs.
First, I want to get feedback on such an architecture and know if you have better ideas/suggestions. Secondly, I would like to get an opinion on how to manage clients on the AWS side please.
Thanks in advance for your answers.
Have you reviewed this project: https://github.com/aws-samples/crossplane-aws-blueprints ? While it may not be exactly what you need, I think it will help you, nonetheless.
- It seems your separation between tenant is based on namespaces. Depends on your data security requirements, it could be a weak separation. You do want to review your requirements, potentially implement additional security / policy controls / enforcement, and even consider cluster level separation for larger deployments, refer here;
- Not to push for AWS consumption but in many cases from a production deployment perspective you might want to consider putting the control plane of your offering together with your data plane (i.e. both on AWS). This simplifies networking connectivity and potentially help with high availability. Depends on how much data flows between them, you can reduce data transfer cost as well which offset the cluster cost.
- From client management perspective, there is not much special to begin with other than the multi-tenancy consideration for a regular Kubernetes cluster(s). To get started, have you looked into this workshop before?
- Yes, I'll add Network Policies, RBAC and PodPolicies, if the project is a success, I will certainly migrate to a multi-cluster infrastructure
- I am aware of this but for now the data flow is negligible. It is indeed planned to migrate the internal cluster to AWS
- I didn't know about this workshop, I'll definitely take a look at it, thanks !
Relevant questions
Dealing With LimitExceededExceptions For Rekogntion Video
asked 2 years agoHow to stay on the local network ?
asked 2 years agoQuestion lightsail + cloudfront + subdomains
asked a year agoDoes eks:AccessKubernetesApi allows to update any resources or it allows only readonly access?
asked a month agoSelf-service Developer Platform architecture on EKS with Crossplane
asked 2 months agoManage identities in AWS SSO - how to create Users via CLI or API ?
asked 6 months agoRunning FPGA machines on EKS
asked a month agoHow to clone a Bitbucket repo from a Springboot microservice running on AWS EKS?
asked 2 months agoSmall Scale VOD Streaming - Am I looking in the right place?
asked a month agoStructure of Repositories for Resource creation and lambdas
asked 25 days ago
Yes I know, I saw these blueprints, thanks for the link