- Newest
- Most votes
- Most comments
You might consider creating a custom image using image builder with your software stack.
If each video segment is processed independently, AWS Batch might be easier to use. It also allows you to use GPU instances if your image processing application can use GPU libraries to speed up image processing. You should be able to create an image that supports MPI on a single node.
AWS parallel cluster is more helpful when there is communication between different nodes and allows running of MPI jobs across multiple nodes.
For both of the above check on spot pricing options if video processing does not need to occur on demand. Do also try out the tutorials to see what will suite your use case best.
Hi! this blog post can help you to choose between AWS ParallelCluster and AWS Batch: https://aws.amazon.com/blogs/hpc/choosing-between-batch-or-parallelcluster-for-hpc/
In ParallelCluster you can use Slurm as a scheduler and you can customize your cluster by using a Custom Boostrap Action or building a Custom AWS ParallelCluster AMI.
ParallelCluster 3.x has the pcluster build-image command that permits you to build your custom AMI and then use it for your clusters, detailed instructions in the link above.
Enrico
Typical HPC MPI use shared file system, so that you can put your MPI application in a shared file system, for example EFS(NFS), or FsX for lustre, MPI application is multi-processes model which could run across multiple hosts, When you submit MPI application job, HPC scheduler will allocate hosts resource so you MUST define resource requirement when you submit job, it can not be dynamically scaling when job running. When HPC scheduler allocate host, it will run your first MPI application process in first allocated host, and then MPI will be responsible for starting other processes in this or other hosts allocated, you do not need handle it by your-selves. If the data was saved in S3, you can create a FsX file system to bind this S3's bucket, so the compute node can access the data as POSIX file system and handle it.
Relevant content
- client vpn(mac) said "The VPN process failed to start the port is already in use by another process"Accepted Answerasked a year ago
- asked a year ago
AWS OFFICIALUpdated 9 months ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
Thank you bakwms.
Spot availability on GPU instances can be a bit harder to find than other instance types. If you go this route make sure the application can be size flexible (aka, work on different size GPU instances), and that you use every AZ in the region for maximum possible spot capacity pools. If there's not a ton of data to move around (watch the data egress costs if there is), then you might want to look into a couple regions for better availability.