When to invoke a lambda directly or via API Gateway

I believe a bit more context would help.

There might be scenarios where adding an API GATEWAY would help if you want to expose your lambda functions. ie: you have a lambda function to process a request from a WEBSOCKET API GATEWAY call. Also, there could be a lambda function to process some data and you want it to be called from a website, etc. Essentially, if you need to expose the function, you'll need an API Gateway in order to do so.

If you have a bunch of lambda functions which doesn't need to be exposed, then using the SDK would suffice.

ie: I have a lambda function to process some files on S3. This is an internal function and I only need to run the function under certain circumstances. So I just call it from time to time using the SDK because the service that calls it is inside of the same VPC so no need to expose it.

If you are somehow orchestrating the execution of lambdas from within lambdas (functions calling other functions) or somewhere else (a process/script running under some trigger). I would recommend you use Step Functions.

Step Functions is a service to orchestrate the integration of different steps that compose a bigger workflow. You write a State Machine using json/yml. Nodes in that state machine can be a lambda function execution, integration with other AWS services, bigger tasks running in a server... virtually whatever you want.

With Step Functions you get some some benefits out-of-the-box like concurrency control, sync/async executions, failure controls, retries, etc. While keeping your lambda code small.

You can then trigger the Step Function under certain conditions or you can set up an API Gateway in front of it if need to expose an API.

You can call them using SDK and it's a valid solution, anyway if you want to make your service more homogeneous, using the REST rather than call directly via SDK is preferred (if the rest of your API of your application are REST, obviously) Using the API gateway, with a custom authorizer, would allow you to use the same authentication mechanism that it's used for the other APIs in your stack, rather than having to assign a role or the IAM keys to make the call to lambda.

About the private access, you could deploy a private API gateway so that the traffic stays inside the VPC.

To summarize:

• Check your authentication mechanism, to determinate if it's worth to add roles to the applications that would call lambda or makes more sense to reuse the authentication mechanism that it's already available in your stack.
• Check your current environment and what's the type of interface you already have and if it's worth or not the extra overhead to add the API gateway for REST