The AWS batch job interface seems to have changed over the weekend and now I find that I can't make any jobs with an array size of over 1000. This is despite the fact that the Array Size help text says "When you create an AWS Batch job, you can turn on Array job. In the Array size box, enter a number between 2 and 10,000 to define how many child jobs are created." However, if I try to enter an array size > 1000 and then go to the next page then the array size box turns red and it gives me an error saying "This value must be an integer between 2 and 1000" and it doesn't let me proceed.
On the previous interface we'd set the array size up to 10000 and just set the Number of Files to Process per Container to something small in the command line with no problem, so I'm not sure why it's changed now (did someone just miss a zero during the interface update or something, since the help text actually does say "between 2 and 10,000"?). Or are we missing something else here?
We need to run our batch processes on many thousands of files so being limited to 1000 each time would be a bit more inconvenient. For example, I'd previously set the array size to 5996 and the Number of Files to Process per Container to 4 to process 23984 files - but if this sticks then now I'd have to set the array size to 1000 and Number of Files to Process per Container to 24? Would each job even be able to handle that many files at once?
Either way, if the help text says one thing for the limit and the interface says another then that's probably something that needs to be fixed.
AWS OFFICIALUpdated 2 years ago
