AWS announces AWS Interconnect – multicloud (preview), providing simple, resilient, high-speed private connections to other cloud service providers. AWS Interconnect - multicloud is easy to configure and provides high-speed, resilient connectivity with dedicated bandwidth, enabling customers to interconnect AWS networking services such as AWS Transit Gateway, AWS Cloud WAN, and Amazon VPC to other cloud service providers with ease.
Can I use Amazon S3 for Hadoop storage instead of HDFS?
I want to configure Amazon EMR to use Amazon Simple Storage Service (Amazon S3) as the Apache Hadoop storage system instead of the Hadoop Distributed File System (HDFS).
Resolution
Although the EMR File System (EMRFS) uses Amazon S3 as storage, you can't configure Amazon EMR to use Amazon S3 as the Hadoop storage layer. HDFS is an implementation of the Hadoop FileSystem API that models POSIX file system behavior. EMRFS is an object store, not a file system. For more information, see Object Stores vs. FileSystems on the Apache Hadoop website.
For more information about the storage layer in Amazon EMR, see Storage.
For information about when to use each file system, see Work with storage and file systems.
- Topics
- Analytics
- Tags
- Amazon EMR
- Language
- English
EMRFS has "file system" in the name but you are saying it is not a file system. This is very confusing and counter intuitive for users.
Thank you for your comment. We'll review and update the Knowledge Center article as needed.
Relevant content
- asked 3 years ago
- asked a year ago
- asked 3 years ago
- asked 4 years ago
- AWS OFFICIALUpdated a year ago
