You might consider creating a custom image using image builder with your software stack.
If each video segment is processed independently, AWS Batch might be easier to use. It also allows you to use GPU instances if your image processing application can use GPU libraries to speed up image processing. You should be able to create an image that supports MPI on a single node.
AWS parallel cluster is more helpful when there is communication between different nodes and allows running of MPI jobs across multiple nodes.
For both of the above check on spot pricing options if video processing does not need to occur on demand. Do also try out the tutorials to see what will suite your use case best.
Hi! this blog post can help you to choose between AWS ParallelCluster and AWS Batch: https://aws.amazon.com/blogs/hpc/choosing-between-batch-or-parallelcluster-for-hpc/
ParallelCluster 3.x has the
pcluster build-image command that permits you to build your custom AMI and then use it for your clusters, detailed instructions in the link above.
Typical HPC MPI use shared file system, so that you can put your MPI application in a shared file system, for example EFS(NFS), or FsX for lustre, MPI application is multi-processes model which could run across multiple hosts, When you submit MPI application job, HPC scheduler will allocate hosts resource so you MUST define resource requirement when you submit job, it can not be dynamically scaling when job running. When HPC scheduler allocate host, it will run your first MPI application process in first allocated host, and then MPI will be responsible for starting other processes in this or other hosts allocated, you do not need handle it by your-selves. If the data was saved in S3, you can create a FsX file system to bind this S3's bucket, so the compute node can access the data as POSIX file system and handle it.
Distributed processing in AWSasked 7 months ago
How to save a .html file to S3 that is created in a Sagemaker processing containerAccepted Answerasked 2 months ago
WorkDocs Drive app stuck "processing", can't mount WorkDocs Drive on WorkSpaceasked an hour ago
Looking for any process locking or latest file processing with msk and lambdaasked 7 days ago
How Do I Change the Game Mode Camera / Mouse Speed?asked 6 years ago
how to choose ml.g4dn.* instances in sagemaker processing jobsasked 8 months ago
AWS ElasticSearch Stuck Upgrade processing after 3.5 hoursasked 2 years ago
I am trying to write an ETL job to the Data Catalog but its writing the Headers as DataAccepted Answerasked 7 months ago
How to load large amount of data from S3 onto Sagemaker?asked 9 months ago
Does the pre and post-processing need to be incorporate in SageMaker?asked 8 months ago