SNS as message broker - Where and when to create the SNS topics?

Question

Hi,

In our application we're considering SNS (fifo) to communicate between our services, each domain should have it's own topic for other domains to subscribe to using SQS (fifo).

Due to the fact that SNS topics must be deployed in order for others to be able to subscribe to it - We ran into some deployment considerations regarding where and when to create these topics.

I hope you guys might help us get a clear answer / guidance towards the preferred approach.

Here are the options we thought could work:
1. **Each domain creates it's own topic for other domains to subscribe to**
Downside - If Domain B subscribes to Domain A's topic - then Domain B is dependent on Domain A's deployment, meaning that Domain B can't be deployed until Domain A is deployed (which can also yield circular dependency).
2. **Create the topic at the moment of subscription**
At the moment when Domain B tries to subscribe to Domain A's topic - if it doesn't exists - Domain B creates Domain A's topic.
Downside - This eliminates the dependency issue from section 1, but causes a mess who's responsible for which topic
3. **Create the topics in an infra domain that should be deployed before the rest of the application is deployed**
This also eliminates the dependency issue
Downside - now the infra domain has to know all the other domains.
4. **Create a single topic in an infra domain for the entire application and filter the messages on every subscription**
Downside - A single subscription has its limits (1000 subscribers ,200 filter policies).

What we're really after is the combination of section 1 (each domain creates it's own subscription) with the ability for "lazy subscription", meaning that subscribing domains won't be dependent of the topic being actually created before they can finish their deployment, something like a "pending subscription" mode.

Have we missed anything?
Is it recommended to look for any AWS CDK ways (other than custom resources) to check if topics actually exists before subscribing to them?

P.S - The same applies to EventBridge and an event bus per domain.

Thanks a lot in advance :)

Answer

Personally I would go with option 1 but have 2 deployment steps for each domain.  So Domain A Step 1 for example does all the setup that makes Domain A a service used by other domains.  All the "Step 1" stuff for all domains is done before moving on to Step 2.  Domain A Step 2 for example does all the setup linking Domain A as a client of other domain services.

Answer

I would go with the 4th option: A centralized bus that everyone publishes to and subscribes on using filters. The limits you mentioned can be increased, so you can look into that.

Your Lazy doesn't seems like a good option. Decoupling the services means that the producers don't know about the consumers, or their subscriptions.

Do you really need FIFO capabilities?  If you do, EventBridge is not an option for you.

SNS as message broker - Where and when to create the SNS topics?

Relevant content