Jobs submitted to worker nodes stay in pending after rebooting head node

Question

Hello,

I have set up a small cluster using Parallel Cluster where I have head and worker nodes in the same public network.

I used it yesterday, and it worked well. I submitted jobs using Slurm to on-demand and spot instances without any issues. When these jobs finished, I stopped the head node instance. After restarting it, every newly submitted jobs remain in a pending state. When checking their status using squeue, I see errors like "ReqNodeNotAvail" and "UnavailableNode." I have attempted to submit jobs to different types of on-demand instances, but the issue persists.

Has anything changed in the configuration after the reboot? If so, what is the best way to stop an instance if I am planning to use it in the future?

Answer

Hi @cortezero

please refer to official documentation, in particular be sure to stop the compute fleet before stopping the head node.

> [...] you can manually stop and start an instance that doesn't have instance stores. For this case and for other cases of instances without ephemeral volumes, continue to [Stop and start a cluster's head node](https://docs.aws.amazon.com/parallelcluster/latest/ug/instance-updates-ami-patch-v3.html#instance-headnode-stop-start-v3).
> If your instance has ephemeral drives and its been stopped, the data in the instance store is lost. You can determine if the instance type used for the head node has instance stores from the table found in [Instance store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes).

Source: https://docs.aws.amazon.com/parallelcluster/latest/ug/instance-updates-ami-patch-v3.html#instance-updates-headnode-v3

Answer

https://stackoverflow.com/questions/50661730/slurm-how-to-restart-failed-worker-job

You can use --requeue

```
#SBATCH --requeue                   ### On failure, requeue for another try
```

**--requeue**
Specifies that the batch job should eligible to being requeue. The job may be requeued explicitly by a system administrator, after node failure, or upon preemption by a higher priority job. When a job is requeued, the batch script is initiated from its beginning. Also see the --no-requeue option. The JobRequeue configuration parameter controls the default behavior on the cluster.

Jobs submitted to worker nodes stay in pending after rebooting head node

Relevant content