Understand and troubleshoot SIGILL

Question

I have built a copy of rocksdb for erlang (gitlab.com/Vagabond1/erlang-rocksdb@8708fe8) for a armv7 based target. It's a iMX6 SoloX processor, here's the cpuinfo:

processor       : 0
model name      : ARMv7 Processor rev 10 (v7l)
BogoMIPS        : 7.54
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x2
CPU part        : 0xc09
CPU revision    : 10

Hardware        : Freescale i.MX6 SoloX (Device Tree)
Revision        : 0500

I have a program that loads the compiled liberocksdb.so but it crashes with a SIGILL, here's the back trace:

#0  0xb1b2a7b0 in std::__detail::_Mod_range_hashing::operator()(unsigned int, unsigned int) const () from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#1  0xb1b79b48 in std::__detail::_Hash_code_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel>, std::__detail::_Select1st, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, true>::_M_bucket_index(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) const () from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#2  0xb1b751d4 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_bucket_index(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) const ()
   from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#3  0xb1b6e01c in std::pair<std::__detail::_Node_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel>, false, true>, bool> std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_insert<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel>, true> > > >(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> const&, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel>, true> > > const&, std::integral_constant<bool, true>) () from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#4  0xb1b63edc in std::__detail::_Insert_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::insert(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> const&) ()
   from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#5  0xb1b568b8 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_Hashtable<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> const*>(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> const*, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> const*, unsigned int, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::__detail::_Mod_range_hashing const&, std::__detail::_Default_ranged_hash const&, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::__detail::_Select1st const&, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> > const&) () from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#6  0xb1b489fc in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_Hashtable(std::initializer_list<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> >, unsigned int, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> > const&) () from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#7  0xb1b3c10c in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, rocksdb::OptionsSanityCheckLevel, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> > >::unordered_map(std::initializer_list<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> >, unsigned int, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, rocksdb::OptionsSanityCheckLevel> > const&) () from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#8  0xb1b28b04 in __static_initialization_and_destruction_0 () from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#9  0xb1b29d30 in _GLOBAL__sub_I_db_impl.cc () from /opt/miner/lib/rocksdb-1.5.0/priv/liberocksdb.so
#10 0xb6fddff8 in ?? () from /lib/ld-linux-armhf.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Here's the disassembly:

(gdb) disas /m
Dump of assembler code for function _ZNKSt8__detail18_Mod_range_hashingclEjj:
   0xb1b2a790 <+0>:     push    {r11}           ; (str r11, [sp, #-4]!)
   0xb1b2a794 <+4>:     add     r11, sp, #0
   0xb1b2a798 <+8>:     sub     sp, sp, #20
   0xb1b2a79c <+12>:    str     r0, [r11, #-8]
   0xb1b2a7a0 <+16>:    str     r1, [r11, #-12]
   0xb1b2a7a4 <+20>:    str     r2, [r11, #-16]
   0xb1b2a7a8 <+24>:    ldr     r3, [r11, #-12]
   0xb1b2a7ac <+28>:    ldr     r2, [r11, #-16]
=> 0xb1b2a7b0 <+32>:    udiv    r2, r3, r2
   0xb1b2a7b4 <+36>:    ldr     r1, [r11, #-16]
   0xb1b2a7b8 <+40>:    mul     r2, r1, r2
   0xb1b2a7bc <+44>:    sub     r3, r3, r2
   0xb1b2a7c0 <+48>:    mov     r0, r3
   0xb1b2a7c4 <+52>:    add     sp, r11, #0
   0xb1b2a7c8 <+56>:    pop     {r11}           ; (ldr r11, [sp], #4)
   0xb1b2a7cc <+60>:    bx      lr
End of assembler dump.

I am not an expert in this domain, but as far as I understand, the program crashed because the instruction udiv is illegal for my CPU. I've checked ARM developer website, this is a Cortex M3 instruction and I believe that my CPU is Cortex M4, so I guess it should have this instruction?

Perhaps my analysis is wrong and this is something else?

My cross compilation was made with the cross toolchain for that CPU and rootfs, and contained arguments such as -march=armv7-a -marm -mfpu=neon -mfloat-abi=hard which should provide compatible binaries.

Thanks

Answer 1

The cmake script forced the use of -march=armv8-a+crc which is not compatible with my CPU.

https://gitlab.com/Vagabond1/erlang-rocksdb/-/commit/6bf4a440b8132ef643682ba87dee6de87dc47d58