Setting the Step Concurrency Level is doing nothing

https://aws.amazon.com/about-aws/whats-new/2019/11/amazon-emr-now-allows-you-to-run-multiple-steps-in-parallel-cancel-running-steps-and-integrate-with-aws-step-functions/
EMR's latest release allows you to run steps in parallel. This is a great feature. Problem is, when I set this value from it's default of 1 to 10, there is no noticeable change. The Steps still execute in sequential order where only one runs and the others are in pending. I'm setting this value after the cluster is made with the modify-cluster command in the cli

Environment:
Version: emr-5.28.0
Application: Spark
Instance types: Master(1) ->m4.2xlarge, Cores(2)->m4.xlarge

I've tried experimenting with the yarn schedulers, but it seems to make no difference. For the sake of simplicity lets just say it's the default Capacity Scheduler that EMR boots with.

Even with the default Yarn capacity scheduler I can successfully do the old hack of tricking EMR into adding more applications to Yarn which will cause them to execute in parallel(desired) as outlined here:https://stackoverflow.com/a/54069971 This works because of the spark.yarn.submit.waitAppCompletion=false property which makes an EMR step complete right away and move on to the next one, even before the Spark application finishes. This is making me think the problem is ultimately EMR not obeying the StepConcurrencyLevel for whatever reason unknown to me. This hacky solution is not ideal, because it basically creates a disconnect from EMR and the Spark job, which means if you wanted to use another aws service like the Serverless Step functions communicating with EMR, certain features may not work correctly (https://aws.amazon.com/blogs/aws/new-using-step-functions-to-orchestrate-amazon-emr-workloads/)

Does it matter if the spark app is submitted in cluster-mode or client-mode? Any tips, tricks, or pointers would be much appreciated.

I would imagine if it was working you would see two or more "RUNNING" steps on the aws EMR console in the Step tab with that also reflected in yarn (yarn application -list)