- Le plus récent
- Le plus de votes
- La plupart des commentaires
So I understand that the nodes don't have external connectivity except the master, so you cannot run DistCP even inside the cluster.
I think the easiest would be to create a script that runs on the master and takes files onto the local dist and uses the standard aws s3 command line client to uploading (tweaking a bit the bandwidth and parallelism).
The other option if you don't want to do the temporary local copy would be to run DistCp in local mode, so it runs only on the master but can access hdfs and s3 directly.
AFAIK, the web solutions you propose to access the cluster externally, would require the DataNodes to be reachable (the master doesn't actually have the data).
The workaround would be to use some proxy service like Knox but it's too much hassle to handle all the security compared with the option of running a script on the cluster master.
Contenus pertinents
- demandé il y a 7 mois
- demandé il y a un an
- AWS OFFICIELA mis à jour il y a un an
- AWS OFFICIELA mis à jour il y a 2 ans
- AWS OFFICIELA mis à jour il y a 6 mois