By using AWS re:Post, you agree to the AWS re:Post Terms of Use

Learning AWS - How to use AWS in DEV

0

Hello,

I am currently working with Spark and HDFS (on-premise) and have recently started learning AWS (covering various services such as S3, DynamoDB, RDS, Redshift, EMR, etc.). I have two main questions:

I learned that AWS CloudFormation is primarily used to request the creation of resources as follows:

Create or modify a CloudFormation template. Submit a Pull Request (PR). Once approved, proceed with creating the resources in different environments (DEV, TEST, PRE-PROD, PROD). Is this the correct process for resource creation using CloudFormation?

How can we develop locally for DEV environments while working with services like DynamoDB, Redshift, EMR, S3, etc.? Would this require a dedicated development environment? If so, wouldn't it be costly? I have come across LocalStack as a tool to simulate AWS services locally. Could you provide more details about this tool (e.g., costs, limitations, alternatives)? Are there any other tools or approaches for such use cases?

Thank you very much!

4 Answers
3
Accepted Answer

Hello,

If you're looking to manage your infrastructure as code, I’d strongly recommend checking out AWS CDK. It allows you to define AWS resources using your favorite programming language (e.g., Python, TypeScript, etc.), making it much more developer-friendly than writing raw Cloud Formation templates. It then generates Cloud Formation templates behind the scenes anyway. (CDK doc). You can also use Terraform if you are more familiar with it.

Regarding costs, you’re right - some services like databases can add up quickly. However, depending on your project’s needs, you don’t have to replicate everything (like VPCs or subnets) for every environment. You could share some resources across environments while maintaining isolation. It’s a bit tricky to give a one-size-fits-all answer without knowing more about your specific use case. But for example, developers could share a database to save costs.

if you have more questions or need help with specific details, feel free to ask - I’d be happy to help 😊

profile pictureAWS
answered 14 days ago
profile picture
EXPERT
reviewed 14 days ago
3

Your understanding of AWS CloudFormation is generally correct, but let me provide some clarification and additional information.

AWS CloudFormation is indeed used to create and manage AWS resources, but the process you described is more aligned with a typical development workflow rather than CloudFormation's specific functionality. Here's a more accurate description of how CloudFormation is typically used:

  1. Create or modify a CloudFormation template (in JSON or YAML format).
  2. Validate the template using CloudFormation's built-in validation tools.
  3. Create a stack using the template, which provisions the specified resources.
  4. Update the stack as needed when changes are required.

The process you mentioned involving Pull Requests and different environments is more related to infrastructure-as-code practices and continuous integration/continuous deployment (CI/CD) pipelines, which can certainly incorporate CloudFormation but are not inherent to CloudFormation itself.

Regarding local development for AWS services, you're right that it can be challenging and potentially costly to maintain separate development environments for each developer. There are several approaches to address this:

  1. LocalStack: This is indeed a popular tool for emulating AWS services locally. It provides a local testing environment that mimics various AWS services, including DynamoDB, S3, and others. LocalStack is free for personal use, with a paid version (LocalStack Pro) offering additional features. While it's a useful tool for basic testing and development, it may have limitations in fully replicating the behavior of actual AWS services, especially for more complex scenarios.

  2. AWS SAM (Serverless Application Model): This is an open-source framework that extends CloudFormation to simplify serverless application development. It includes a local development and testing feature.

  3. AWS CDK (Cloud Development Kit): This tool allows you to define cloud infrastructure using familiar programming languages. It can generate CloudFormation templates and includes features for local testing.

  4. Docker containers: For some services, you can use Docker containers to run local versions of AWS services. For example, there's an official AWS-provided Docker image for running DynamoDB locally.

  5. AWS Free Tier: For actual AWS services, you can make use of the AWS Free Tier, which provides a limited amount of free usage for many AWS services. This can be useful for small-scale development and testing.

  6. Dedicated development AWS accounts: Some organizations create separate AWS accounts for development purposes, which can help manage costs and provide isolation from production environments.

The best approach often depends on your specific needs, the scale of your development team, and the complexity of your applications. A combination of local emulation tools and carefully managed AWS environments is common in many development workflows.
Sources
Working with CloudFormation templates - AWS CloudFormation
Getting started with CloudFormation - AWS CloudFormation
Creating Amazon RDS resources with AWS CloudFormation - Amazon Relational Database Service

profile picture
answered 14 days ago
profile picture
EXPERT
reviewed 14 days ago
2

I think these informations will be helpful with your:

  • About CloudFormation: It is used to provision Cloud Infrastructure as Code (you do not need to go to AWS console, and configure steps by steps, it will reduce human mistake). If your company have separate team for Cloud operation and Developer: Cloud Operation Team will define CloudFormation template, Developer only focus to input some necessary params to complete the yaml file.

  • Pipeline to provision resources: It will have separate environments for each stage (dev, test, stag, prod,... the way you define environment depend on you business). Normally, all environment using the same CloudFormation template, but will have some different in input params (ex: you have different role ARN, more restrictions, Security Group, NACL,...)

  • Can we develop locally (without provision real service on AWS): In my case, the answer will be no. Don't worry about cost of developing phase, traffic of user very low when you compare to prod environment. Almost case, you won't go over free tier.

profile picture
answered 14 days ago
0

Thank you very much for your response. Regarding development costs, how are they typically managed in companies (not for learning purposes)? Do most companies rely on LocalStack, or do they use actual development environments with minimal resources (e.g., limited CPU, RAM, WCU, RCU, etc.)?

answered 14 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions