Error: Exceeding Concurrency Limit (2500) on /protos.Deliver with Amazon Managed Blockchain Hyperledger Fabric

Question

I'm using Amazon Managed Blockchain with Hyperledger Fabric for our private blockchain setup, and we're experiencing a recurring issue recently. Some times, we receive the following error and it is getting down:
[BlockEventSource]: Failed to start event service message=2 UNKNOWN: too many requests for /protos.Deliver, exceeding concurrency limit (2500)
Stack Trace (partial):
Error: 2 UNKNOWN: too many requests for /protos.Deliver, exceeding concurrency limit (2500)
    at ServiceClientImpl.makeBidiStreamRequest (node_modules/@grpc/grpc-js/build/src/client.js:403:32)

It seems that our requests are surpassing the concurrency limit set by Amazon Managed Blockchain, resulting in these failures. We have developed Smart contract in Go and it is running EC2 and we develpoed API in nodejs and it is also running in EC2 sometimes this api is getting down and when i check decker logs this error showing.

Questions:
1. How can we manage or increase the concurrency limit (2500) for /protos.Deliver on Amazon Managed Blockchain, if possible?
2. Are there best practices or optimizations to reduce the load on the with Hyperledger Fabric?
3. Is there any configuration available on Amazon Managed Blockchain to handle a higher volume of event service requests without hitting the limit?
4. Is it something else with API ?

Any advice on how to avoid this would be greatly appreciated. Thanks in advance!

AWS-User-6621777 · Answer

1. **Increasing the Concurrency Limit**: Unfortunately, the concurrency limit of 2500 for the /protos.Deliver endpoint is a hard limit set by Amazon Managed Blockchain and cannot be increased. This limit is in place to ensure the stability and performance of the Managed Blockchain service.
2. **Best Practices and Optimizations for Hyperledger Fabric**:
   - **Batch Transactions**: Instead of sending individual transactions, try to batch multiple transactions together and submit them in a single request. This can help reduce the overall number of requests to the Managed Blockchain service.
   - **Optimize Smart Contract Logic**: Review your smart contract code and optimize it to reduce the number of calls to the /protos.Deliver endpoint. This may involve refactoring your contract logic or introducing caching mechanisms.
   - **Implement Retry Mechanisms**: Implement a retry mechanism in your application to handle the "too many requests" errors gracefully. This will allow your application to continue operating even when the limit is temporarily exceeded.
   - **Leverage Asynchronous Patterns**: Consider using asynchronous patterns, such as event-driven architectures or message queues, to decouple your application logic from the direct interaction with the Managed Blockchain service.
3. **Configuration Options in Amazon Managed Blockchain**:
   - Unfortunately, there are no configuration options available in Amazon Managed Blockchain to directly handle a higher volume of event service requests. The concurrency limit is a fixed setting to ensure the stability of the service.
4. **API Optimization**:
   - Review your API implementation and identify any potential bottlenecks or inefficiencies. Ensure that your API is optimized for performance and is not making unnecessary calls to the Managed Blockchain service.
   - Consider implementing caching mechanisms in your API to reduce the number of requests to the Managed Blockchain service.
   - Analyze your API usage patterns and identify any potential areas for optimization, such as batching requests or introducing asynchronous processing.
In summary, while you cannot directly increase the concurrency limit in Amazon Managed Blockchain, you can implement various optimization strategies and best practices to reduce the load on the event service and mitigate the impact of the limit. This may involve optimizing your smart contract logic, implementing retry mechanisms, leveraging asynchronous patterns, and optimizing your API implementation.
If you continue to experience issues after implementing these strategies, you may want to consider reaching out to AWS Support for further guidance and assistance

Error: Exceeding Concurrency Limit (2500) on /protos.Deliver with Amazon Managed Blockchain Hyperledger Fabric

Relevant content