error: A fatal error has been detected by the Java Runtime Environment

Question

I have a dataframe va with two column vertex1 and vertex2. va has

6731710209 rows

I want to compute similarity between vertex1 and vertex2. To do that I have the following UDF function:

weight_udf = udf(edgeWeight, FloatType())
va = va.withColumn('similarity', weight_udf(va.vertex1, va.vertex2))

When running the code I get the following strange error:

#
# A fatal error has been detected by the Java Runtime Environment:

    #
    #  SIGSEGV (0xb) at pc=0x00002b5fe21d0ba0, pid=67441, tid=0x00002b605e421700
    #
    # JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
    # Problematic frame:
    # J 12180 C2 org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap$1.next()Z (60 bytes) @ 0x00002b5fe21d0ba0 [0x00002b5fe21d0ac0+0xe0]
    #
    # Core dump written. Default location: /project/6008168/moudi/job/core or core.67441
    #
    # An error report file with more information is saved as:
    # /project/6008168/moudi/job/hs_err_pid67441.log
    [thread 47692633986816 also had an error]
    #
    # If you would like to submit a bug report, please visit:
    #   http://bugreport.java.com/bugreport/crash.jsp
    #
Aborted (core dumped)

I cannot figure out what is the problem. Can someone please help?