1 Answer
- Newest
- Most votes
- Most comments
0
Hi,
If you are required to run files in sequential order only, what is the reason to start tasks in parallel? For me, it makes no sense even though you can implement that by adding some semaphore/lock.
Relevant content
- Accepted Answerasked a month ago
AWS OFFICIALUpdated 16 days ago- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated a year ago
Files need to run in parallel as this needs for quick processing. Let's say we have 100 files and each file took 1 minute then overall process will be completed in 100minutesm whereas if these files running in parallel then the process will be completed in 1 minutes to 2 minites
But from your explanation, you have correct results ONLY if they are run consequently. Maybe you can PROCESS them in parallel and then concatenate them in a certain order later?
Its not possible, I think u didn't get what exactly the problem is. If running in parallel then one file results is merging into other files. Each text file has thousand of words, so processing by docker image and lets say 4 files are running in parallel then all results mixed into these 4 files
Yeah, it's still not clear what you expect to have.
Let's assume we have 3 files: 1.txt (contains two lines, each only one symbol "1"), 2.txt (contains two lines, each only one symbol "2"), 3.txt (contains two lines, each only one symbol "3"). We want to process all 3 files in parallel and calculate the sum of all numbers (one per line), for 1.txt it's 2, for 2.txt it's 4, for 3.txt it's 6.
Where are you expecting these results to appear? If example is not correct, please provide your.