EMR autoscaling: 'org.apache.hadoop.util.DiskChecker$DiskErrorException(No space available in any of the local directories.)'

Question

I get following error when running tez query.
This is in EMR cluster with auto scaling enabled.

Root device EBS volume size: 100 GiB

Additional EBS volume: 200 GiB

```
bash-4.2$ls -lh /tmp
lrwxrwxrwx 1 root root 8 Jun  2 13:20 /tmp -> /mnt/tmp
```

/mnt has enough space:

```
/dev/dev1 195G 3.7G 192G 2% /mnt
```

```
INFO  : Cleaning up the staging area file:/tmp/hadoop/mapred/staging/hdfs1254373830/.staging/job_local1254373830_0002
ERROR : Job Submission failed with exception 'org.apache.hadoop.util.DiskChecker$DiskErrorException(No space available in any of the local directories.)'
org.apache.hadoop.util.DiskChecker$DiskErrorException: No space available in any of the local directories.
    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:416)
    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:165)
    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:130)
    at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:123)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:172)
    at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:794)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:251)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:423)
    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:149)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
    at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:330)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. No space available in any of the local directories.
```

Answer

Hi,

Generally in hive "No space available in any of the local directories" refers to a permissions issue rather than a disk space issue.

you check the hive server 2 logs to confirm this, you should ideally see something like "org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not writable: /mnt/mapred" exception

this is because of the default permissions given to /mnt/mappred

Changing the permissions on the Primary Node's local /mnt/mapred/ should allow queries to run

[hadoop@ip- ~]$ sudo chmod 777 /mnt/mapred/

Hope this helps, please feel free to share the errors in hive server logs if this does not help

EMR autoscaling: 'org.apache.hadoop.util.DiskChecker$DiskErrorException(No space available in any of the local directories.)'

Relevanter Inhalt