Efficient Multi dimenstion Mutable Scala Array

Question

Is there memory efficient Scala multi dimension array such as in Java?

I'm trying to solve Hackerrank problem with strict memory constraints: 256mb. My solution breaks with out of memory error while creating 2D array with (39384,39384) elements:

Array.ofDim[Long](39384,39384)

The same happens in scala console.

java.lang.OutOfMemoryError: Java heap space
        at scala.reflect.ManifestFactory$$anon$9.newArray(Manifest.scala:115)
        at scala.reflect.ManifestFactory$$anon$9.newArray(Manifest.scala:113)
        at scala.Array$.ofDim(Array.scala:222)
        at Solution$.solve(Solution.scala:4)
        at Solution$$anonfun$main$1.apply$mcVI$sp(Solution.scala:41)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
        at Solution$.main(Solution.scala:37)
        at Solution.main(Solution.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at scala.reflect.internal.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:68)
        at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
        at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:99)
        at scala.reflect.internal.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:68)
        at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:99)
        at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:22)
        at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:39)
        at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:29)
        at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:39)
        at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:72)
        at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:94)
        at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:103)
        at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)

Answer 1

Array.ofDim[Long](39384,39384) creates array with size 39384 * 39384 * Long = 1551099456 * 8 = 11 Gb which is obviously more than 256 Mb. Just try less dimensions to see how it works:

scala> Array.ofDim[Long](3,3)
res10: Array[Array[Long]] = Array(Array(0, 0, 0), Array(0, 0, 0), Array(0, 0, 0))

If you need some coordinates processing for large geometric space - you may just create Map[Point, Long] of ponts, like Map(Point(39382, 9000) -> 5L, Point(1,0) -> 9L) .

If you actually just need two arrays (each 39384 sized) - then just create two arrays Array.ofDim[Long](39384,2)

P.S. If your algorithm is scalable you may also use multiple nodes of Apache Spark for calculations.