"Pod Predicate NodeAffinity failed" causing the pod unable to started in a managed nodegroup. Can anyone suggests us what to investigate more?

Hi,

We received an alert due to the pod status is not Running in our system. This behavior seems randomly occur and seems not specific to anything. The context of this issue is that we have multiple managed node groups, each containing specific labels based on workload type. There are many Deployment per node group but each node will have single pod of every deployments. We implemented this using a combination of NodeAffinity to select node and podAntiAffinity to repel pod of the same kind in the same node.

The Pod Predicate Node Affinity failed is the error we received when there are two Deployments that scaled out and one of them stuck in this state, another one can be scheduled on the same node and running fine. This issue occurred once a month but we want to investigate the root cause of this issue. Can anyone suggest what should we investigate further?