For processing genomics datasets in containers launched from AWS Batch, is it possible to stream the data directly from S3, using an approach similar to SageMaker pipe mode (or fast file mode)?
Also, are there any hardware limitations that I should be aware of? For example, in this article, the maximum transfer speed between EC2 and S3 is 100 Gbps (VPC endpoints and public IPs in the same region). In a 2018 article, the bandwidth between EC2 and S3 was stated as 25 Gbps