ImageIo exceptions inside spark executors

Question

I have a spark job which loads images from hive tables and then i run a cnn on them to tag them in memory and write back to hadoop. With average load of 40-50k images and around 50 executors with 2 cores i dont really have any problems with the ImageIO read library. But when the number of images increases this library starts to break.

Solutions i have tried is setting -: ImageIO.setUseCache(false),but to no avail. Can someone tell me how to fix this issue.

Exception while converting image
        javax.imageio.IIOException: Can't create cache file!
            at javax.imageio.ImageIO.createImageInputStream(ImageIO.java:361)
            at javax.imageio.ImageIO.read(ImageIO.java:1351)
            at com.agoda.pixmen.tagging.business.ImageTagging$.com$agoda$pixmen$tagging$business$ImageTagging$$filterCorruptImages(ImageTagging.scala:43)
            at com.agoda.pixmen.tagging.business.ImageTagging$$anonfun$1$$anonfun$2.apply(ImageTagging.scala:84)
            at com.agoda.pixmen.tagging.business.ImageTagging$$anonfun$1$$anonfun$2.apply(ImageTagging.scala:84)
            at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:463)
            at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
            at scala.collection.Iterator$Partner$1.hasNext(Iterator.scala:1209)
            at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
            at scala.collection.Iterator$class.foreach(Iterator.scala:893)
            at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
            at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
            at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
            at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
            at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
            at scala.collection.AbstractIterator.to(Iterator.scala:1336)
            at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
            at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
            at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
            at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
            at com.agoda.pixmen.tagging.business.ImageTagging$$anonfun$1.apply(ImageTagging.scala:88)
            at com.agoda.pixmen.tagging.business.ImageTagging$$anonfun$1.apply(ImageTagging.scala:81)
            at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:196)
            at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:193)
            at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
            at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
            at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
            at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
            at org.apache.spark.scheduler.Task.run(Task.scala:108)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
        Caused by: java.nio.file.NoSuchFileException: /ssd/yarn/nm/usercache/pix-men/appcache/application_1536091434343_135862/container_e22_1536091434343_135862_01_000053/tmp/imageio1438929695028204573.tmp
            at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
            at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
            at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
            at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
            at java.nio.file.Files.newByteChannel(Files.java:361)
            at java.nio.file.Files.createFile(Files.java:632)
            at java.nio.file.TempFileHelper.create(TempFileHelper.java:138)
            at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161)
            at java.nio.file.Files.createTempFile(Files.java:897)
            at javax.imageio.stream.FileCacheImageInputStream.<init>(FileCacheImageInputStream.java:102)
            at com.sun.imageio.spi.InputStreamImageInputStreamSpi.createInputStreamInstance(InputStreamImageInputStreamSpi.java:69)
            at javax.imageio.ImageIO.createImageInputStream(ImageIO.java:357)
            ... 44 more