Common internal data access layer pattern for DynamoDB?

Question

Hi,

My project currently has one publicly facing REST API implemented using API Gateway w/ Lambda handlers written in Python. The data this API interrogates is stored in a single DynamoDB table that utilises Single Table Design.

I now have a requirement to develop a separate GraphQL API (I intend to use AppSync) that will also need to access the data in the same DynamoDB table.

For the sake of fully exploring possible solutions in this question, the Lambda handlers backing the new GraphQL/AppSync API of the  will be written using Go, instead of Python, as we want to maximise performance.

Currently, we don't have any "common" layer to mediate access to the DDB table and crucially, we'd like to centralise validation of incoming data before this gets written to the DDB table.

I'm trying to understand and evaluate the possible options to achieve this goal. In terms of ideas, I have:

1. Implement validation logic at the API handler level (i.e. write the validation logic into the respective Lambda funcs themselves). This would essentially mean duplicating the logic for each API, as I'd need to write the code twice, once in Python and the other in Go.

2. Build some kind of ORM-layer in code and distribute as a Lambda layer (however, I'm assuming I'd still need to implement this twice, once in Go, and the other in Python).

3. Build some kind of common internal data access layer, possibly using a microservices architecture. In this design, the Lambda handlers powering each API would call this common DAL, so the validation logic would be applied equally to both. I think I could possibly use service discovery (e.g. AWS Cloud Map) to help here.

4. Build a common internal data access layer using a monolithic architecture of some description.

Have I missed any other possible solutions here, and does anyone have thoughts/suggestions on which (if any) may be most appropriate (or definitely not appropriate!) for this scenario. I understand there's a lot of 'it depends' here, so I'm just really interested in understanding what others have done when faced with a similar situation and your main reasons for picking this solution over others. There's a lot of "academic" info online, but I'm struggling to find many real-world examples.

Many thanks in advance.

Answer

Based on the details provided, here are a few thoughts on approaches you could take:

* Implementing validation logic directly in each Lambda function would work but has the downside of duplicating code as you mentioned. This could become difficult to maintain over time.
* Creating a common data access layer (DAL) using a microservices architecture with service discovery (Cloud Map) is a good option. This centralizes the validation logic while keeping the APIs decoupled. The Lambda functions would make requests to the DAL microservice.
* Another approach could be creating reusable Lambda layers with the shared validation code. The layers could be implemented in a language-agnostic way using something like protocol buffers. The API Lambda functions would import the common layer.
* For simplicity, you could also consider a monolithic DAL deployed as an API Gateway private API. The validation functions would live here and be called by the public APIs.

In general, centralizing the shared logic in a reusable way (layers, microservice) while keeping the APIs decoupled is preferable to duplication. It improves maintainability. I'd evaluate the tradeoffs of the microservices vs monolithic approach based on your specific needs and growth expectations. Let me know if any part needs more explanation or if you have additional questions!

Common internal data access layer pattern for DynamoDB?

Relevant content