AWS re:Invent 2024 - Kubernetes meets Serverless: Deploy EDAs with Kubernetes APIs

Modern organizations often find themselves operating both Kubernetes and serverless technologies, but struggle with separate tooling and processes between these worlds. At AWS re:Invent 2024, Emily Shea, Christina Andonov, and Giedrius Praspaliauskas demonstrated how to bridge this gap using AWS Controllers for Kubernetes (ACK) and Kube Resource Orchestrator (KRO).

These tools create a unified approach that lets developers deploy AWS serverless resources through familiar Kubernetes interfaces while maintaining security and governance - truly bringing Kubernetes and serverless together.

Modern Applications: Two Paths to Innovation

Organizations building modern applications face numerous choices when it comes to operational models. Emily explained in the opening that most companies prioritize building applications that are modular, operationally efficient, automated in deployment, and designed to scale to billions of users while responding in milliseconds.

To achieve these goals, customers typically follow one of two paths:

Path 1: Serverless/AWS Native Approach

• Using AWS-specific APIs and services
• Focusing on autonomous development teams
• Minimizing infrastructure management
• Gravitating toward services like Amazon ECS with AWS Fargate, AWS Lambda, and Amazon EventBridge

Path 2: Kubernetes-Based Approach

• Building around Kubernetes APIs and tools
• Emphasizing centralized platform teams and standards
• Making use of community and open-source ecosystem
• Typically using Amazon EKS or self-managed Kubernetes on Amazon EC2

"Both of these are totally valid models", Emily noted. "Customers are very successful with either approach".

The Reality: Heterogeneous Environments

What's particularly interesting is that despite efforts to standardize, most large organizations end up with heterogeneous environments. This happens for several reasons: acquisitions (companies might standardize on Amazon ECS, then acquire a company using Kubernetes), technology preferences (teams have different expertise and comfort levels), and workload optimization (some use cases simply work better with specific compute options).

Emily highlighted the UK's Driver and Vehicle Licensing Agency (DVLA) as a prime example. This government agency, which collects about £6 billion in revenue annually, recently modernized their driver services applications. Their solution? A hybrid architecture using Amazon EKS for UI components and standardized services, while choosing serverless and event-driven approaches for asynchronous processing workflows.

The Platform Team Dilemma

Christina shared her personal experience as a platform engineer before joining AWS, illustrating the frustration many platform teams face.

"We had a ticketing system where developers would request resources like Amazon S3 buckets or AWS Lambda functions. We called it a 'hero rotation', but when I was on that rotation, I did not feel much like a hero", Christina explained. "Every once in a while I'd get a ticket saying 'Can I get a Lambda?' and I knew that ticket was going to take more than a week of back-and-forth conversations".

To solve this problem, Christina's team built a Terraform platform to standardize infrastructure provisioning across the organization. Initially, it seemed like a great solution - improving security posture and supporting collaboration with other teams. However, they soon discovered a critical flaw: developer velocity actually decreased.

"We didn't realize how often Terraform pipelines fail, sometimes for no apparent reason", Christina explained. "For experienced Terraform users, this was just part of the job. But for developers without that experience, they'd simply go back to the ticketing system and say 'Hey hero, can you come fix my pipeline?'".

The Missing Piece: Kubernetes-Native AWS Resource Management

Christina's experience mirrors what many AWS customers encounter: maintaining separate toolsets for Kubernetes workloads and AWS serverless resources creates friction. Developers comfortable with Kubernetes APIs face a challenging learning curve or ticketing delays when they need to integrate with serverless services.

The solution emerged in the form of two complementary projects:

1. AWS Controllers for Kubernetes (ACK) - Provides Kubernetes controllers for AWS services, with 50 controllers now in General Availability and covered by AWS Support.
2. Kube Resource Orchestrator (KRO) - An open-source project for grouping resources and passing values between them, just released two weeks before re:Invent

"Earlier this year, the Amazon EKS service team decided to take ownership of all ACK controllers", Christina explained. KRO provides the "missing piece of the puzzle" that gives platform teams the capability to create abstracted resource groups that developers can instantiate without understanding all the underlying AWS resources.

Real-World Example: Data Processing with EMR on Amazon EKS and Amazon Step Functions

To demonstrate these concepts, Giedrius showcased an event-driven data processing architecture that combines Kubernetes and serverless components. The workflow includes:

1. Data objects arriving in Amazon S3, triggering an event to Amazon EventBridge
2. EventBridge initiating an Amazon Step Functions workflow execution
3. The workflow starting an Apache Spark job on Amazon EMR on Amazon EKS within a specified namespace
4. Amazon Step Functions copying incoming data to a data lake in parallel with processing
5. When Spark completes, results being written to Amazon DynamoDB
6. Finally, a notification sent via Amazon SNS to subscribers

"Everything is in that same single namespace", Giedrius emphasized. "And it uses the script specified in the configuration to process the data".

What's particularly powerful about this approach is that data engineers can deploy the entire stack through a simple Kubernetes resource definition, without requiring additional AWS permissions or knowledge of the underlying AWS services.

The Code: Simplicity for Developers, Control for Platform Teams

Giedrius demonstrated how data engineers can deploy this complex architecture using a simple approach. The solution requires about 20 lines of YAML from a developer perspective, specifying basic parameters like input bucket, Spark script name, email address for notifications, and default settings.

Meanwhile, platform teams maintain a more detailed resource group definition that includes:

• Namespace creation
• Amazon S3 bucket configurations
• Amazon DynamoDB table schema
• Amazon Step Functions workflow definition (about 170 lines of ASL)
• IAM policies with least-privilege permissions

"We specify particular bucket, particular prefix for input, for temporary processing output. And even for the script used for processing, we lock it down to a particular script name", Giedrius explained, highlighting how this approach enforces security best practices.

Demonstration in Action

During the live demonstration, Giedrius showed the entire workflow. He uploaded three new months of NYC taxi data to the input Amazon S3 bucket, which triggered three Amazon Step Functions executions to automatically start in parallel. After processing (which took about 3 minutes), the data appeared in Amazon DynamoDB and a QuickSight dashboard. The system also sent email notifications when processing completed. The entire process ran without any manual intervention or resource provisioning outside of Kubernetes.

Conclusion: Best of Both Worlds

Emily wrapped up the session with key takeaways:

1. Heterogeneous technology choices are both common and valid in modern application development
2. New tools like ACK and KRO bridge the gap between Kubernetes and serverless technologies
3. The most effective way to empower developers is to give them APIs they can use directly

"While it used to be a challenging experience to bring these technologies together in terms of the experience of a ticketing process or Terraform pipeline", Emily summarized, "now there are exciting new tools available in terms of what you can use".

For organizations looking to modernize application platforms, combining Kubernetes and serverless doesn't have to be an either/or decision. With the right tools and approach, you can deploy event-driven architectures through Kubernetes APIs, giving platform teams the control they need while empowering developers with self-service capabilities.

For those interested in exploring further, the KRO repository includes examples of both the serverless microservice and data processing architectures demonstrated in the session. The full session recording is available on the AWS YouTube channel, featuring detailed explanations and demonstrations from Emily, Christina, and Giedrius.