I am using the crash utility to investigate a core file dump. From this core dump, I can see that one processes has two deadlocked threads. The cause of the deadlock seems to be task->mm->mmap_sem being held for way too long while trying to serve a page fault. I am trying to figure out the faulting address that caused this issue.
While serving a page fault, the Linux kernel function do_page_fault reads the fault address from cr2 register and then proceeds to server the page fault. See code below.
dotraplinkage void __kprobes
do_page_fault(struct pt_regs *regs, unsigned long error_code)
{
struct vm_area_struct *vma;
struct task_struct *tsk;
unsigned long address;
struct mm_struct *mm;
int fault;
int write = error_code & PF_WRITE;
unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE |
(write ? FAULT_FLAG_WRITE : 0);
tsk = current;
mm = tsk->mm;
/* Get the faulting address: */
address = read_cr2();
/*
* Detect and handle instructions that would cause a page fault for
* both a tracked kernel page and a userspace page.
*/
if (kmemcheck_active(regs))
kmemcheck_hide(regs);
prefetchw(&mm->mmap_sem);
if (unlikely(kmmio_fault(regs, address)))
return;
.....
And this is a disassembly of the do_page_fault function from the core dump:
Dump of assembler code for function do_page_fault:
0xffffffff81441c77 <+0>: push %rbp
0xffffffff81441c78 <+1>: mov %rsp,%rbp
0xffffffff81441c7b <+4>: push %r15
0xffffffff81441c7d <+6>: push %r14
0xffffffff81441c7f <+8>: push %r13
0xffffffff81441c81 <+10>: push %r12
0xffffffff81441c83 <+12>: push %rbx
0xffffffff81441c84 <+13>: sub $0xd8,%rsp
0xffffffff81441c8b <+20>: data32 data32 data32 xchg %ax,%ax
0xffffffff81441c90 <+25>: mov %esi,%eax
0xffffffff81441c92 <+27>: mov %rdi,%rbx
0xffffffff81441c95 <+30>: mov %rsi,%r13
0xffffffff81441c98 <+33>: and $0x2,%eax
0xffffffff81441c9b <+36>: cmp $0x1,%eax
0xffffffff81441c9e <+39>: sbb %eax,%eax
0xffffffff81441ca0 <+41>: add $0x29,%eax
0xffffffff81441ca3 <+44>: mov %eax,-0xe4(%rbp)
0xffffffff81441ca9 <+50>: mov %gs:0xc400,%r15
0xffffffff81441cb2 <+59>: mov 0x270(%r15),%rax
0xffffffff81441cb9 <+66>: mov %rax,-0xf0(%rbp)
0xffffffff81441cc0 <+73>: mov %cr2,%rax
0xffffffff81441cc3 <+76>: data32 data32 xchg %ax,%ax
0xffffffff81441cc7 <+80>: mov %rax,%r12
0xffffffff81441cca <+83>: mov -0xf0(%rbp),%rax
0xffffffff81441cd1 <+90>: add $0x60,%rax
0xffffffff81441cd5 <+94>: mov %rax,-0xf8(%rbp)
0xffffffff81441cdc <+101>: prefetcht0 (%rax)
0xffffffff81441cdf <+104>: movabs $0x7fffffffefff,%rax
0xffffffff81441ce9 <+114>: cmp %rax,%r12
0xffffffff81441cec <+117>: jbe 0xffffffff81441d50 <do_page_fault+217>
0xffffffff81441cee <+119>: test $0xd,%r13b
0xffffffff81441cf2 <+123>: jne 0xffffffff81441d04 <do_page_fault+141>
0xffffffff81441cf4 <+125>: mov %r12,%rdi
0xffffffff81441cf7 <+128>: callq 0xffffffff81441884 <vmalloc_fault>
0xffffffff81441cfc <+133>: test %eax,%eax
0xffffffff81441cfe <+135>: jns 0xffffffff81441ff9 <do_page_fault+898>
0xffffffff81441d04 <+141>: mov %r12,%rsi
0xffffffff81441d07 <+144>: mov %r13,%rdi
0xffffffff81441d0a <+147>: callq 0xffffffff81441af0 <spurious_fault>
0xffffffff81441d0f <+152>: test %eax,%eax
0xffffffff81441d11 <+154>: jne 0xffffffff81441ff9 <do_page_fault+898>
0xffffffff81441d17 <+160>: testb $0x3,0x88(%rbx)
0xffffffff81441d1e <+167>: jne 0xffffffff81441e3e <do_page_fault+455>
0xffffffff81441d24 <+173>: mov %gs:0xd4e0,%rax
0xffffffff81441d2d <+182>: test %rax,%rax
0xffffffff81441d30 <+185>: je 0xffffffff81441e3e <do_page_fault+455>
0xffffffff81441d36 <+191>: mov $0xe,%esi
0xffffffff81441d3b <+196>: mov %rbx,%rdi
0xffffffff81441d3e <+199>: callq 0xffffffff81441253 <kprobe_fault_handler>
0xffffffff81441d43 <+204>: test %eax,%eax
0xffffffff81441d45 <+206>: jne 0xffffffff81441ff9 <do_page_fault+898>
0xffffffff81441d4b <+212>: jmpq 0xffffffff81441e3e <do_page_fault+455>
0xffffffff81441d50 <+217>: testb $0x3,0x88(%rbx)
0xffffffff81441d57 <+224>: jne 0xffffffff81441d7c <do_page_fault+261>
0xffffffff81441d59 <+226>: mov %gs:0xd4e0,%rax
0xffffffff81441d62 <+235>: test %rax,%rax
0xffffffff81441d65 <+238>: je 0xffffffff81441d7c <do_page_fault+261>
0xffffffff81441d67 <+240>: mov $0xe,%esi
0xffffffff81441d6c <+245>: mov %rbx,%rdi
0xffffffff81441d6f <+248>: callq 0xffffffff81441253 <kprobe_fault_handler>
0xffffffff81441d74 <+253>: test %eax,%eax
0xffffffff81441d76 <+255>: jne 0xffffffff81441ff9 <do_page_fault+898>
0xffffffff81441d7c <+261>: testb $0x3,0x88(%rbx)
0xffffffff81441d83 <+268>: je 0xffffffff81441d97 <do_page_fault+288>
0xffffffff81441d85 <+270>: callq 0xffffffff810be11d <trace_hardirqs_on>
0xffffffff81441d8a <+275>: sti
0xffffffff81441d8b <+276>: data32 xchg %ax,%ax
0xffffffff81441d8e <+279>: data32 xchg %ax,%ax
0xffffffff81441d91 <+282>: or $0x4,%r13
0xffffffff81441d95 <+286>: jmp 0xffffffff81441dac <do_page_fault+309>
0xffffffff81441d97 <+288>: testb $0x2,0x91(%rbx)
0xffffffff81441d9e <+295>: je 0xffffffff81441dac <do_page_fault+309>
0xffffffff81441da0 <+297>: callq 0xffffffff810be11d <trace_hardirqs_on>
0xffffffff81441da5 <+302>: sti
0xffffffff81441da6 <+303>: data32 xchg %ax,%ax
0xffffffff81441da9 <+306>: data32 xchg %ax,%ax
0xffffffff81441dac <+309>: test $0x8,%r13b
0xffffffff81441db0 <+313>: je 0xffffffff81441dc0 <do_page_fault+329>
0xffffffff81441db2 <+315>: mov %r12,%rdx
0xffffffff81441db5 <+318>: mov %r13,%rsi
0xffffffff81441db8 <+321>: mov %rbx,%rdi
0xffffffff81441dbb <+324>: callq 0xffffffff810369ea <pgtable_bad>
0xffffffff81441dc0 <+329>: mov 0x8ea4f2(%rip),%eax # 0xffffffff81d2c2b8 <perf_swevent_enabled+8>
0xffffffff81441dc6 <+335>: test %eax,%eax
0xffffffff81441dc8 <+337>: je 0xffffffff81441df8 <do_page_fault+385>
0xffffffff81441dca <+339>: test %rbx,%rbx
0xffffffff81441dcd <+342>: mov %rbx,%rcx
0xffffffff81441dd0 <+345>: jne 0xffffffff81441de4 <do_page_fault+365>
0xffffffff81441dd2 <+347>: lea -0xe0(%rbp),%r14
0xffffffff81441dd9 <+354>: mov %r14,%rdi
0xffffffff81441ddc <+357>: callq 0xffffffff81037284 <perf_fetch_caller_regs>
0xffffffff81441de1 <+362>: mov %r14,%rcx
0xffffffff81441de4 <+365>: mov %r12,%r8
0xffffffff81441de7 <+368>: xor %edx,%edx
0xffffffff81441de9 <+370>: mov $0x1,%esi
0xffffffff81441dee <+375>: mov $0x2,%edi
0xffffffff81441df3 <+380>: callq 0xffffffff810d24b2 <__perf_sw_event>
0xffffffff81441df8 <+385>: mov %gs:0xc408,%rax
0xffffffff81441e01 <+394>: testl $0xefffffff,-0x1fbc(%rax)
0xffffffff81441e0b <+404>: jne 0xffffffff81441e3e <do_page_fault+455>
0xffffffff81441e0d <+406>: cmpq $0x0,-0xf0(%rbp)
0xffffffff81441e15 <+414>: je 0xffffffff81441e3e <do_page_fault+455>
0xffffffff81441e17 <+416>: mov -0xf8(%rbp),%rdi
0xffffffff81441e1e <+423>: callq 0xffffffff810721e4 <down_read_trylock>
0xffffffff81441e23 <+428>: test %eax,%eax
0xffffffff81441e25 <+430>: jne 0xffffffff81441e5d <do_page_fault+486>
0xffffffff81441e27 <+432>: test $0x4,%r13b
0xffffffff81441e2b <+436>: jne 0xffffffff81441e51 <do_page_fault+474>
0xffffffff81441e2d <+438>: mov 0x80(%rbx),%rdi
0xffffffff81441e34 <+445>: callq 0xffffffff8106bb2c <search_exception_tables>
0xffffffff81441e39 <+450>: test %rax,%rax
0xffffffff81441e3c <+453>: jne 0xffffffff81441e51 <do_page_fault+474>
0xffffffff81441e3e <+455>: mov %r12,%rdx
0xffffffff81441e41 <+458>: mov %r13,%rsi
0xffffffff81441e44 <+461>: mov %rbx,%rdi
0xffffffff81441e47 <+464>: callq 0xffffffff8103707e <bad_area_nosemaphore>
0xffffffff81441e4c <+469>: jmpq 0xffffffff81441ff9 <do_page_fault+898>
0xffffffff81441e51 <+474>: mov -0xf8(%rbp),%rdi
0xffffffff81441e58 <+481>: callq 0xffffffff8143def4 <down_read>
0xffffffff81441e5d <+486>: mov -0xf0(%rbp),%rdi
0xffffffff81441e64 <+493>: mov %r12,%rsi
0xffffffff81441e67 <+496>: callq 0xffffffff810f62eb <find_vma>
0xffffffff81441e6c <+501>: test %rax,%rax
0xffffffff81441e6f <+504>: mov %rax,%r14
0xffffffff81441e72 <+507>: je 0xffffffff81441ea6 <do_page_fault+559>
0xffffffff81441e74 <+509>: cmp %r12,0x8(%rax)
0xffffffff81441e78 <+513>: jbe 0xffffffff81441eb9 <do_page_fault+578>
0xffffffff81441e7a <+515>: testb $0x1,0x31(%rax)
0xffffffff81441e7e <+519>: je 0xffffffff81441ea6 <do_page_fault+559>
0xffffffff81441e80 <+521>: test $0x4,%r13b
0xffffffff81441e84 <+525>: je 0xffffffff81441e97 <do_page_fault+544>
0xffffffff81441e86 <+527>: lea 0x10100(%r12),%rax
0xffffffff81441e8e <+535>: cmp 0x98(%rbx),%rax
0xffffffff81441e95 <+542>: jb 0xffffffff81441ea6 <do_page_fault+559>
0xffffffff81441e97 <+544>: mov %r12,%rsi
0xffffffff81441e9a <+547>: mov %r14,%rdi
0xffffffff81441e9d <+550>: callq 0xffffffff810f6ce9 <expand_stack>
0xffffffff81441ea2 <+555>: test %eax,%eax
0xffffffff81441ea4 <+557>: je 0xffffffff81441eb9 <do_page_fault+578>
0xffffffff81441ea6 <+559>: mov %r12,%rdx
0xffffffff81441ea9 <+562>: mov %r13,%rsi
0xffffffff81441eac <+565>: mov %rbx,%rdi
0xffffffff81441eaf <+568>: callq 0xffffffff81037093 <bad_area>
0xffffffff81441eb4 <+573>: jmpq 0xffffffff81441ff9 <do_page_fault+898>
0xffffffff81441eb9 <+578>: test $0x2,%r13b
0xffffffff81441ebd <+582>: je 0xffffffff81441ec6 <do_page_fault+591>
0xffffffff81441ebf <+584>: testb $0x2,0x30(%r14)
0xffffffff81441ec4 <+589>: jmp 0xffffffff81441ed1 <do_page_fault+602>
0xffffffff81441ec6 <+591>: test $0x1,%r13b
0xffffffff81441eca <+595>: jne 0xffffffff81441ed7 <do_page_fault+608>
0xffffffff81441ecc <+597>: testb $0x7,0x30(%r14)
0xffffffff81441ed1 <+602>: jne 0xffffffff81441fce <do_page_fault+855>
0xffffffff81441ed7 <+608>: mov %r12,%rdx
0xffffffff81441eda <+611>: mov %r13,%rsi
0xffffffff81441edd <+614>: mov %rbx,%rdi
0xffffffff81441ee0 <+617>: callq 0xffffffff810370e1 <bad_area_access_error>
0xffffffff81441ee5 <+622>: jmpq 0xffffffff81441ff9 <do_page_fault+898>
0xffffffff81441eea <+627>: mov %r14d,%ecx
0xffffffff81441eed <+630>: mov %r12,%rdx
0xffffffff81441ef0 <+633>: mov %r13,%rsi
0xffffffff81441ef3 <+636>: mov %rbx,%rdi
0xffffffff81441ef6 <+639>: callq 0xffffffff8103712f <mm_fault_error>
0xffffffff81441efb <+644>: test %eax,%eax
0xffffffff81441efd <+646>: jne 0xffffffff81441ff9 <do_page_fault+898>
0xffffffff81441f03 <+652>: testb $0x8,-0xe4(%rbp)
0xffffffff81441f0a <+659>: je 0xffffffff81441fc0 <do_page_fault+841>
0xffffffff81441f10 <+665>: test $0x4,%r14b
0xffffffff81441f14 <+669>: je 0xffffffff81441f61 <do_page_fault+746>
0xffffffff81441f16 <+671>: incq 0x3f8(%r15)
0xffffffff81441f1d <+678>: mov 0x8ea3a5(%rip),%eax # 0xffffffff81d2c2c8 <perf_swevent_enabled+24>
0xffffffff81441f23 <+684>: test %eax,%eax
0xffffffff81441f25 <+686>: je 0xffffffff81441fab <do_page_fault+820>
0xffffffff81441f2b <+692>: test %rbx,%rbx
0xffffffff81441f2e <+695>: mov %rbx,%rcx
0xffffffff81441f31 <+698>: jne 0xffffffff81441f50 <do_page_fault+729>
0xffffffff81441f33 <+700>: lea -0xe0(%rbp),%rcx
0xffffffff81441f3a <+707>: mov %rcx,%rdi
0xffffffff81441f3d <+710>: mov %rcx,-0x100(%rbp)
0xffffffff81441f44 <+717>: callq 0xffffffff81037284 <perf_fetch_caller_regs>
0xffffffff81441f49 <+722>: mov -0x100(%rbp),%rcx
0xffffffff81441f50 <+729>: mov %r12,%r8
0xffffffff81441f53 <+732>: xor %edx,%edx
0xffffffff81441f55 <+734>: mov $0x1,%esi
0xffffffff81441f5a <+739>: mov $0x6,%edi
0xffffffff81441f5f <+744>: jmp 0xffffffff81441fa6 <do_page_fault+815>
0xffffffff81441f61 <+746>: incq 0x3f0(%r15)
0xffffffff81441f68 <+753>: mov 0x8ea356(%rip),%eax # 0xffffffff81d2c2c4 <perf_swevent_enabled+20>
0xffffffff81441f6e <+759>: test %eax,%eax
0xffffffff81441f70 <+761>: je 0xffffffff81441fab <do_page_fault+820>
0xffffffff81441f72 <+763>: test %rbx,%rbx
0xffffffff81441f75 <+766>: mov %rbx,%rcx
0xffffffff81441f78 <+769>: jne 0xffffffff81441f97 <do_page_fault+800>
0xffffffff81441f7a <+771>: lea -0xe0(%rbp),%rcx
0xffffffff81441f81 <+778>: mov %rcx,%rdi
0xffffffff81441f84 <+781>: mov %rcx,-0x100(%rbp)
0xffffffff81441f8b <+788>: callq 0xffffffff81037284 <perf_fetch_caller_regs>
0xffffffff81441f90 <+793>: mov -0x100(%rbp),%rcx
0xffffffff81441f97 <+800>: mov %r12,%r8
0xffffffff81441f9a <+803>: xor %edx,%edx
0xffffffff81441f9c <+805>: mov $0x1,%esi
0xffffffff81441fa1 <+810>: mov $0x5,%edi
0xffffffff81441fa6 <+815>: callq 0xffffffff810d24b2 <__perf_sw_event>
0xffffffff81441fab <+820>: and $0x400,%r14d
0xffffffff81441fb2 <+827>: je 0xffffffff81441fc0 <do_page_fault+841>
0xffffffff81441fb4 <+829>: andl $0xfffffff7,-0xe4(%rbp)
0xffffffff81441fbb <+836>: jmpq 0xffffffff81441e51 <do_page_fault+474>
0xffffffff81441fc0 <+841>: mov -0xf8(%rbp),%rdi
0xffffffff81441fc7 <+848>: callq 0xffffffff8107222e <up_read>
0xffffffff81441fcc <+853>: jmp 0xffffffff81441ff9 <do_page_fault+898>
0xffffffff81441fce <+855>: mov -0xe4(%rbp),%ecx
0xffffffff81441fd4 <+861>: mov -0xf0(%rbp),%rdi
0xffffffff81441fdb <+868>: mov %r14,%rsi
0xffffffff81441fde <+871>: mov %r12,%rdx
0xffffffff81441fe1 <+874>: callq 0xffffffff810f45bf <handle_mm_fault>
0xffffffff81441fe6 <+879>: test $0x433,%eax
0xffffffff81441feb <+884>: mov %eax,%r14d
0xffffffff81441fee <+887>: je 0xffffffff81441f03 <do_page_fault+652>
0xffffffff81441ff4 <+893>: jmpq 0xffffffff81441eea <do_page_fault+627>
0xffffffff81441ff9 <+898>: add $0xd8,%rsp
0xffffffff81442000 <+905>: pop %rbx
0xffffffff81442001 <+906>: pop %r12
0xffffffff81442003 <+908>: pop %r13
0xffffffff81442005 <+910>: pop %r14
0xffffffff81442007 <+912>: pop %r15
0xffffffff81442009 <+914>: leaveq
0xffffffff8144200a <+915>: retq
Now, is it possible to find out what is the address of the page fault? where is it stored on the function's stack?
UPDATE:
here is the output if bt -f
#0 [ffff8801f01159f0] __schedule at ffffffff8143d229
ffff8801f01159f8: 0000000000000082 ffff8801f1201818
ffff8801f0115a08: ffff880100000000 ffff8801f0114010
ffff8801f0115a18: ffff8801b9880780 0000000000011b80
ffff8801f0115a28: ffff8801f0115fd8 ffff8801f0115fd8
ffff8801f0115a38: 0000000000011b80 ffff8801f19264c0
ffff8801f0115a48: ffff8801b9880780 ffffffff810f3f7b
ffff8801f0115a58: 00000001b74d4828 ffffea00b74d4860
ffff8801f0115a68: ffff8801f15fa5a0 ffff8801b9880780
ffff8801f0115a78: 0000000000000001 fffffffeffffffff
ffff8801f0115a88: ffff8801b9880780 ffff8801f0115aa0
ffff8801f0115a98: ffffffff8143d3b5
#1 [ffff8801f0115a98] schedule at ffffffff8143d3b5
ffff8801f0115aa0: ffff8801f0115b00 ffffffff8143e7ed
#2 [ffff8801f0115aa8] rwsem_down_failed_common at ffffffff8143e7ed
ffff8801f0115ab0: ffff8801f15fa5b0 ffff8801f15fa5b0
ffff8801f0115ac0: 0000000000000000 00007fea00000001
ffff8801f0115ad0: 80000001ed0c0067 0000000000000000
ffff8801f0115ae0: ffff8801f0115c88 00007fea45ccbfe7
ffff8801f0115af0: 0000000000000002 0000000000000000
ffff8801f0115b00: ffff8801f0115b10 ffffffff8143e846
#3 [ffff8801f0115b08] rwsem_down_read_failed at ffffffff8143e846
ffff8801f0115b10: ffff8801f0115b68 ffffffff812166c4
#4 [ffff8801f0115b18] call_rwsem_down_read_failed at ffffffff812166c4
ffff8801f0115b20: ffffffff81120c26 0000000000000ff8
ffff8801f0115b30: 0000000000000000 0000000000000004
ffff8801f0115b40: 00007fea45ccbfe7 ffff8801f1201818
ffff8801f0115b50: ffffffff8144afe0 ffff8801f15fa5a0
ffff8801f0115b60: ffffffff8143df0b ffff8801f0115c78
ffff8801f0115b70: ffffffff81441e5d
#5 [ffff8801f0115b70] do_page_fault at ffffffff81441e5d
ffff8801f0115b78: ffff8801f0115ba8 ffff8801f15fa5a0
ffff8801f0115b88: ffff8801f15fa540 00000029811333a0
ffff8801f0115b98: ffff8801f0115bb8 ffff8801eff11940
ffff8801f0115ba8: 0000000000000068 ffff8802d3001080
ffff8801f0115bb8: 00000000000000d0 00000000000000d0
ffff8801f0115bc8: ffff8801f0115c18 ffffffff8110ecc5
ffff8801f0115bd8: 0000000000000020 0000000200000202
ffff8801f0115be8: 00000000000000d0 0000000000000002
ffff8801f0115bf8: ffff8802d3ad4aa0 0000000000000002
ffff8801f0115c08: ffffea0009e3b150 ffffea0009e3b128
ffff8801f0115c18: ffff8801f0115c98 ffff8801f0115de8
ffff8801f0115c28: ffffffff812167ca 0000000000000ff8
ffff8801f0115c38: 0000000000000000 0000000000000004
ffff8801f0115c48: 00007fea45ccbfe7 0000000000000001
ffff8801f0115c58: ffff8801a41b8078 0000000000000ff8
ffff8801f0115c68: 0000000000000000 0000000000002ff0
ffff8801f0115c78: ffff8801f0115de8 ffffffff8143f105
#6 [ffff8801f0115c80] page_fault at ffffffff8143f105
[exception RIP: pipe_read+324]
RIP: ffffffff81120c26 RSP: ffff8801f0115d38 RFLAGS: 00010206
RAX: ffff8801f0115ec8 RBX: ffff8801ba6bcd40 RCX: 0000000000000000
RDX: 0000000000000ff8 RSI: 0000000000001017 RDI: 0000000000000ff8
RBP: ffff8801f0115de8 R8: 00007fea45ccbfe7 R9: 0000000000000004
R10: 0000000000000000 R11: 0000000000000ff8 R12: ffff8801a41b8078
R13: 0000000000000ff8 R14: 0000000000000000 R15: 0000000000002ff0
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
ffff8801f0115c88: 0000000000002ff0 0000000000000000
ffff8801f0115c98: 0000000000000ff8 ffff8801a41b8078
ffff8801f0115ca8: ffff8801f0115de8 ffff8801ba6bcd40
ffff8801f0115cb8: 0000000000000ff8 0000000000000000
ffff8801f0115cc8: 0000000000000004 00007fea45ccbfe7
ffff8801f0115cd8: ffff8801f0115ec8 0000000000000000
ffff8801f0115ce8: 0000000000000ff8 0000000000001017
ffff8801f0115cf8: 0000000000000ff8 ffffffffffffffff
ffff8801f0115d08: ffffffff81120c26 0000000000000010
ffff8801f0115d18: 0000000000010206 ffff8801f0115d38
ffff8801f0115d28: 0000000000000018 ffffffff81120bb8
ffff8801f0115d38: ffffffff81211ef8 ffff8801b9880780
ffff8801f0115d48: 0000000000001ff8 ffff8801ef41e390
ffff8801f0115d58: ffff8801ba6bcd88 00000003f12012d0
ffff8801f0115d68: ffff8801ba582000 ffff8801f0115ec8
ffff8801f0115d78: 00000001f0115dc8 ffffffff81617180
ffff8801f0115d88: 00000001f0115dc8 ffff8801ba582ff8
ffff8801f0115d98: 0000000df0115da8 0000000000000ff8
ffff8801f0115da8: ffff8801f1508500 000000000003d010
ffff8801f0115db8: 0000000000100073 ffff8801f0115df8
ffff8801f0115dc8: ffff8801f0115f58 ffff8801f1508500
ffff8801f0115dd8: ffff8801f0115ec8 0000000000000003
ffff8801f0115de8: ffff8801f0115ef8 ffffffff81118dfe
#7 [ffff8801f0115df0] do_sync_read at ffffffff81118dfe
ffff8801f0115df8: 0000000000011b80 0000000000000000
ffff8801f0115e08: 0000000000000000 ffffffff00000001
ffff8801f0115e18: ffff8801f1508500 0000000000000000
ffff8801f0115e28: 0000000000000000 0000000000000000
ffff8801f0115e38: 0000000000000000 ffff8801b9880780
ffff8801f0115e48: 0000000000000000 0000000000000000
ffff8801f0115e58: 0000000000000000 ffff8801ef41e358
ffff8801f0115e68: 0000000000040000 0000000000000003
ffff8801f0115e78: 0000000000040000 ffffffff811e4d73
ffff8801f0115e88: ffff8801f0115ef8 ffff8801f1508500
ffff8801f0115e98: 0000000000000004 0000000000000000
ffff8801f0115ea8: ffff8801f0115ec8 ffffffff811e4de0
ffff8801f0115eb8: 0000000000040000 ffff8801f1508500
ffff8801f0115ec8: 00007fea45ccaff0 000000000003d010
ffff8801f0115ed8: ffff8801f1508500 00007fea45cc8000
ffff8801f0115ee8: ffff8801f0115f58 0000000000040000
ffff8801f0115ef8: ffff8801f0115f38 ffffffff8111988f
#8 [ffff8801f0115f00] vfs_read at ffffffff8111988f
ffff8801f0115f08: 0000000000000001 00007fea43ceb000
ffff8801f0115f18: 0000000000000003 ffff8801f1508500
ffff8801f0115f28: 00007fea45cc8000 00007fea45cc8000
ffff8801f0115f38: ffff8801f0115f78 ffffffff811199ae
#9 [ffff8801f0115f40] sys_read at ffffffff811199ae
ffff8801f0115f48: 0000000000000000 0000000000040000
ffff8801f0115f58: 0000000000000000 00000001f0114000
ffff8801f0115f68: 0000003dcdd8e6c0 0000000000040000
ffff8801f0115f78: 0000000000000000 ffffffff81445742
#10 [ffff8801f0115f80] system_call_fastpath at ffffffff81445742
RIP: 0000003dcdadb51d RSP: 00007fea454ed0d0 RFLAGS: 00003246
RAX: 0000000000000000 RBX: ffffffff81445742 RCX: 00007fea4907b088
RDX: 0000000000040000 RSI: 00007fea45cc8000 RDI: 0000000000000000
RBP: 0000000000000000 R8: 00000000ffffffff R9: 0000000000000000
R10: 0000000000000022 R11: 0000000000003293 R12: 0000000000040000
R13: 0000003dcdd8e6c0 R14: 00000001f0114000 R15: 0000000000000000
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
The faulting address is very likely of little to no significance. All necessary data should be visible "around" the stack frame.
What is the context here? Did you get a panic from hung task detector with a thread waiting to get the semaphore? As in, are you sure you are looking at the right thread here?
While I can't verify now, afair the address is obtainable from the register dump you can see when you 'bt'. Alternatively, as the commenter noted the address lands in r12. Chances are further assembly moves it around, but otherwise it should be in that register OR pushed to the stack if a function is called. Computing its location is left as an exercise for the reader, it is only mildly cumbersome. In fact, 'bt -f' will likely make the address easily stand out without much analysis. If not, you can 'dis -r' on the return address to disassembly upwards from that part.
Chances are what you are looking at is a classic: a nfs-based mmapped file, where the server is not responding. Notes of a not responding server will be seen in dmesg, but mere bt should tell you it is waiting for something.
Now the update.
The bt as posted clearly shows this very thread got 'stuck' for some time waiting for the lock owner. So what you should investigate is the lock owner, as opposed to this thread. The pointer to the owner should be stored somewhere in the semaphore in relatively recent kernels. For super old kernels (and it seems you are running one), you may need to resort to investigating all traces.
As a side note, it is not hard to spot a userspace-y address in the dump: 00007fea45ccbfe7
Looking at arguments passed to the read system call, we see rsi 00007fea45cc8000 (the passed buffer) and rdx 0000000000040000. That is, the address definitely belongs to the buffer, but the offset for a page fault is somewhat odd. You would have to disassemble to confirm. However, as noted earlier, this is the wrong thread to look at in the first place.