slurm cgroup.conf in PCS

Thank you for the feedback. AWS PCS does not currently support setting cgroups.conf settings.

The proposed aws pcs update-cluster command generated an error

Unknown parameter in slurmConfiguration: "SlurmCustomSettings", must be one of: scaleDownIdleTimeInSeconds, slurmCustomSettings, accounting, slurmRest

A modified command generated the error

An error occurred (ValidationException) when calling the UpdateCluster operation: Parameter validation failed:
Slurm custom settings only accepts the following keys: [AccountingStorageEnforce, AccountingStorageTRES, AccountingStoreFlags, AuthAltParameters, CliFilterPlugins, DefMemPerCPU, EnforcePartLimits, Epilog, FairShareDampeningFactor, HealthCheckInterval, HealthCheckNodeState, HealthCheckProgram, JobRequeue, LaunchParameters, Licenses, OverTimeLimit, PreemptExemptTime, PreemptMode, PreemptParameters, PreemptType, PriorityCalcPeriod, PriorityDecayHalfLife, PriorityFavorSmall, PriorityFlags, PriorityMaxAge, PriorityUsageResetPeriod, PriorityWeightAge, PriorityWeightAssoc, PriorityWeightFairshare, PriorityWeightJobSize, PriorityWeightPartition, PriorityWeightQOS, PriorityWeightTRES, Prolog, PrologFlags, RequeueExit, RequeueExitHold, SchedulerParameters, SelectTypeParameters, SrunPortRange, TaskEpilog, TaskPluginParam, TaskProlog, UnkillableStepProgram, UnkillableStepTimeout]

In AWS Parallel Computing Service (PCS), you can enable the ConstrainDevices option in Slurm's cgroup configuration using custom Slurm settings. This setting helps constrain jobs to their allocated GPUs when running multiple jobs on the same multi-GPU compute node.

You can apply this setting by updating your cluster using the AWS CLI or console. Here's how to do it using the AWS CLI:

aws pcs update-cluster \
--cluster-identifier my-cluster \
--slurm-configuration \
'SlurmCustomSettings=[{parameterName=ConstrainDevices,parameterValue="yes"}]'

This configuration will set ConstrainDevices=yes in the Slurm cgroup.conf file, which ensures that jobs are constrained to only use the devices (GPUs) that are allocated to them.

For a more comprehensive cgroup configuration, you might want to include additional related settings. For example, SageMaker HyperPod uses a configuration that includes multiple cgroup constraints:

CgroupAutomount=yes
ConstrainCores=yes
CgroupPlugin=autodetect
ConstrainDevices=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
SignalChildrenProcesses=yes

These settings work together to provide proper resource isolation between jobs, ensuring that each job can only access its allocated CPUs, memory, and GPUs.

After applying these settings, you'll be able to run multiple GPU jobs on the same node with each job constrained to its allocated GPUs, preventing jobs from interfering with each other's GPU resources.
Sources
Announcing expanded support for Custom Slurm Settings in AWS Parallel Computing Service | AWS HPC Blog
SageMaker HyperPod FAQs - Amazon SageMaker AI
Configuring custom Slurm settings in AWS PCS - AWS PCS