What causes SunCC crash in bfd_asm.c?

Question

I'm trying to track down the cause of a SunCC compiler crash. It has been around since the early SunCC 12.x days, and it is present in the latest SunCC 12.6. The crash happens when attempting to multiply polynomials on x86 systems. The code below is part of the GCM algorithm:

$ cat test.cxx
# include <emmintrin.h>
# include <tmmintrin.h>
# include <wmmintrin.h>

__m128i GCM_Reduce_CLMUL(__m128i c0, __m128i c1, __m128i c2, const __m128i &r)
{
  __m128i t = r;
  c1 = _mm_xor_si128(c1, _mm_slli_si128(c0, 8));
  t = _mm_clmulepi64_si128(c0, r, 0x10);
  c1 = _mm_xor_si128(c1, t);
  c0 = _mm_srli_si128(c0, 8);
  c0 = _mm_xor_si128(c0, c1);
  c0 = _mm_slli_epi64(c0, 1);
  c0 = _mm_clmulepi64_si128(c0, r, 0x0);
  c2 = _mm_xor_si128(c2, c0);
  t = _mm_srli_si128(c1, 8);
  c2 = _mm_xor_si128(c2, t);
  c1 = _mm_unpacklo_epi64(c1, c2);
  c1 = _mm_srli_epi64(c1, 63);
  c2 = _mm_slli_epi64(c2, 1);
  return _mm_xor_si128(c2, c1);
}

And:

$ /opt/developerstudio12.6/bin/CC -DNDEBUG -g3 -xO3 -m64 -KPIC -template=no%extdef -xarch=aes -c test.cxx
lf    25 PCLMULP_Xx      REG %x4    UND 0      REG %x4    REG %x0    UND 0      UND 0      UND 0      UND 0      UND 0      UND 0      off:0 uc:2 nxt: 29 bb:   3 FDI:F Ln:10 Ex:22

"test.cxx", [__1cQGCM_Reduce_CLMUL6FXXXrkX_X_]: assertion failed in function dump_asm_instruction() @ bfd_asm.c:2602
assert(nd_not_null_( disp.disp[0]) || (hf_dump_node(bfd_lf_node), 0))

CC: ube failed for test.cxx

I have not been able to locate information on the crash that happens due to bfd_asm.c:2602. I believe it is the same crash at What causes SunCC crash in g3mangler.cc when using -std=XXX?, but it just moved around.

Our strategy to date has been to disable the code paths while waiting for a fix. It does not look like it is going to be fixed anytime soon so we want to find a workaround and re-enable the code.

What is causing the crash and how do I fix it?

---

I think this is the MCVE though it is not very useful:

$ cat test.cxx
# include <emmintrin.h>
# include <tmmintrin.h>
# include <wmmintrin.h>

__m128i GCM_Reduce_CLMUL(__m128i c0, __m128i c1, __m128i c2, const __m128i &r)
{
  __m128i t = r;
  t = _mm_clmulepi64_si128(c0, t, 0x10);
  c0 = _mm_clmulepi64_si128(c0, t, 0x0);
  return _mm_xor_si128(c1, c0);
}

jwalton@solaris2:~/cryptopp$ /opt/developerstudio12.6/bin/CC -DNDEBUG -g3 -xO3 -m64 -KPIC -template=no%extdef -xarch=aes -c test.cxx
lf    17 PCLMULP_Xx      REG %x2    UND 0      REG %x2    REG %x0    UND 0      UND 0      UND 0      UND 0      UND 0      UND 0      off:0 uc:2 nxt: 21 bb:   3 FDI:F Ln:9 Ex:15

"test.cxx", [__1cQGCM_Reduce_CLMUL6FXXXrkX_X_]: assertion failed in function dump_asm_instruction() @ bfd_asm.c:2602
assert(nd_not_null_( disp.disp[0]) || (hf_dump_node(bfd_lf_node), 0))

CC: ube failed for test.cxx

Answer 1

This is a bug in the compiler. From the internal representation dump I see the wrong type for the second input operand of the PCLMUL instruction. It has to be immediate but in the dump it is XMM.

The sad thing is that this is very unlikely to be fixed. The only workaround I can think of is using -xarch=avx or higher as this will be using different instruction representation and the bug will not appear.