Workflow information needed for AWS Lambda image processing

If your team's current experience level with AWS is such that "AWS is too confusing" ( and it definitely is confusing until you gain more experience with it), then I would stick with what you have, more scalable solutions are inherently 'more confusing' -- until its not :) definitely many more moving pieces and things to learn.
However - its not difficult -- it just takes time to learn to think about architecture differently and learn about how to build complex things from many simple parts.
How I would do this (and how I have done similar jobs in the past) is different then the other suggestions. I would start with StepFunctions as the core 'workflow engine'. StepFunctions are managed workflows which you can author entirely online, graphicly , and via simple JSON or YAML configuration documents. I would take each of your 'steps' and make them a seperate 'activity' in StepFunctions -- starting with a 'dummy' step for each ( use a PASS state as a placeholder). Once you get the basic structure in place then you replace each PASS state with either a Lambda Task step or a direct AWS API Integration step. Most AWS APIs can be directly invoked from StepFunctions. Lambda may be easier for some things, depends on how complex the API is -- i.e. if its too complicated to figure out the raw REST/JSON payload you may find it easier to use a Lambda and language specific AWS SDK which supports typed API calls and more traditional integrations. Everything you describe can be implemented with a simple StepFunctions workflow and a few lambda calls. Except possibly the 'update the Database' part - depending on what Database that is -- you may find it easier to integrate using the StepFunctions Activity API which allows any program (running on EC2, Docker, or anywhere in your datacenter including a traveling laptop) to take part in the workflow.

What Stepfunctions brings to the table is visibility and reliability. Visibility - every step is fully exposed as an input and output JSON document. Full logging and persistence of every execution step comes built in. You can literally 'see' every thing that went into and out of each step and function call, I find this makes understanding and debugging much easier then in a program running on EC2 or lambda or docker where it may be difficult to directly debug and time consuming to instrument with enough logs -- let alone a way to get at the logs and match them up against the execution's. StepFunctions provides a GUI and API that tracks everything cleanly. Reliability - Advice I have learned 'the hard way' --- Try to keep your 'Orchestration' physically seperate from your 'Execution' -- When you put everything in one basket -- say a program in EC2 -- then any errors or problems in a single bit of the code -- one line gone bad -- can break everything. That can mean it breaks the debug logs -- It also means bugs in one part of your code -- seemingly unrelated to any other -- can break other parts mysteriously. Run out of temp space in the Resize step ? you might break the Compress step, Bug in the S3 copy might crash crash the logs. Figuring out where something breaks is much harder in 'the cloud' then 'on your desktop' due to lack of visibility -- Run it in step functions -- and step functions itself will not break no matter what your code does -- you can build reliably retries and logging that is impervious to your code's bugs. Run a 'monolithic' app that does 10 steps -- and 'something goes wrong' -- where ? what part of the app is responsible for managing errors ? Same thing if you split it out to pure Lambda calls -- which lambda call is responsible for error recovery or figuring out "the next step" ? HOw to debug THAT one ? Lambda can be 'fragile' -- and when it breaks it simply doesn't return -- will there be debug logs ? maybe, maybe not, depends on how bad it broke. What if the lambda code worked but the 'calling the next one' breaks ? what manages that ? Now you start to need Queues and dead letter queues and lambdas watching lambdas and more lambdas cleaning up the problems of the other ones ... Really Messy and hard to code and debug --- think spaghetti code is bad ? try 20 lambdas all trying to coordinate a workflow blindly and independently.

Step functions solves that problem by reversing the control paradigm. Make your code 'do one thing' and let the workflow 'decide what to do next'

Step Functions comes built in with a controlled parallel Map task. Takes your 'array of 1000 file names' and it will divide-and-conquer -- splitting the work into batches and iterating over each in an independant thread of execution -- scaling to whatever concurrency you want without getting 'blown out of the waters' if you throw 100x more at it then expected.

It takes some practice to get used to how to design the logic -- its definitely different then procedural programming, and it is very VERY limited wrt expressions and data manipulation -- you may have to call a lambda to do simple things like multiply ... But as you get used to it you will find ways to work within the limits -- the design/edit/test iteration cycle is fast -- as short as a few seconds -- and able to be entirely developed in the console GUI or offline in a text editor or both.

There are many solutions in your usecase and lambda is a best part of solution

1- how your images are detected to be treated, you fetch them from a database or something like that?

2- is your images uploaded somewhere, if so can you set upload location to s3, if possible simply trigger an event to sqs and trigger the lambda behind that sqs to avoid concurrency risks.

3- simply put un event to sqs from ec2 and trigger the lambda from sqs

4- use stepfunctions which has an internal queue and will help you to avoid reaching concurency limits

As a compliment, the lambda has a default concurency soft limit which can be increased, also you have a burst capacity which is added to your concurency limit if you don't use your limit for sometimes, but based on region will not passe 3000.