ThreadSanitizer (TSan) instrumentation using LLVM opt and TSan passes

Question

My goal is to instrument my initial IR with proper calls to TSan runtime library functions using LLVM opt tool and TSan passes. In other words, I want to end up with similar TSan instrumentation as when using clang -fsanitize=thread -S but by directly using opt and TSan passes instead.

As far as I know, LLVM has two passes for TSan instrumentation: tsan-module (a module pass) and tsan (a function pass). Both passes are available by default in opt, i.e. are included in opt -print-passes report.

I choose tiny_race.c as my sample programe, where the main thread and the thread it spawns (Thread1) form a data race while accessing a global variable Global.

Here are the two steps I take to instrument the code my way:

1. Generating the initial LLVM IR for tiny_race.c:

   clang -S -emit-llvm tiny_race.c -o tiny_race.ll

2. Using LLVM opt to instrument tiny_race.ll with the two TSan passes:

   opt -passes='tsan-module,tsan' tiny_race.ll -S -o myInstrumented.ll

The above pass pipeline executes fine but the resulting myInstrumented.ll lacks some TSan instrumentations. More specifically:

• Thread1 (child thread) is left completely un-instrumented.
• main thread only has @__tsan_func_entry and @__tsan_func_exit instrumentations and its accesses to Global are not instrumented.

Could anyone please explain why my approach produces a partially-instrumented output? Any suggestion is greatly appreciated.

---

To better display the difference between the IR resulting from my approach and the expected one, bellow you can find definitions of main and Thread1 in each of them.

Here is myInstrumented.ll:

; Function Attrs: noinline nounwind optnone uwtable
define dso_local ptr @Thread1(ptr noundef %x) #0 {
entry:
  %x.addr = alloca ptr, align 8
  store ptr %x, ptr %x.addr, align 8
  store i32 42, ptr @Global, align 4
  %0 = load ptr, ptr %x.addr, align 8
  ret ptr %0
}


; Function Attrs: noinline nounwind optnone uwtable
define dso_local i32 @main() #0 {
entry:
  %0 = call ptr @llvm.returnaddress(i32 0)
  call void @__tsan_func_entry(ptr %0)      *****TSAN INSTRUMENTATION*****
  %retval = alloca i32, align 4
  %t = alloca i64, align 8
  store i32 0, ptr %retval, align 4
  %call = call i32 @pthread_create(ptr noundef %t, ptr noundef null, ptr noundef @Thread1, ptr noundef null) #4
  store i32 43, ptr @Global, align 4
  %1 = load i64, ptr %t, align 8
  %call1 = call i32 @pthread_join(i64 noundef %1, ptr noundef null)
  %2 = load i32, ptr @Global, align 4
  call void @__tsan_func_exit()            *****TSAN INSTRUMENTATION*****
  ret i32 %2
}

And here is the resulting IR when using clang -fsanitize=thread -S -emit-llvm tiny_race.c which is my expected result:

; Function Attrs: noinline nounwind optnone sanitize_thread uwtable
define dso_local ptr @Thread1(ptr noundef %x) #0 {
entry:
  %0 = call ptr @llvm.returnaddress(i32 0)
  call void @__tsan_func_entry(ptr %0)     *****TSAN INSTRUMENTATION*****
  %x.addr = alloca ptr, align 8
  store ptr %x, ptr %x.addr, align 8
  call void @__tsan_write4(ptr @Global)    *****TSAN INSTRUMENTATION*****
  store i32 42, ptr @Global, align 4
  %1 = load ptr, ptr %x.addr, align 8
  call void @__tsan_func_exit()            *****TSAN INSTRUMENTATION*****
  ret ptr %1
}


; Function Attrs: noinline nounwind optnone sanitize_thread uwtable
define dso_local i32 @main() #0 {
entry:
  %0 = call ptr @llvm.returnaddress(i32 0)
  call void @__tsan_func_entry(ptr %0)      *****TSAN INSTRUMENTATION*****
  %retval = alloca i32, align 4
  %t = alloca i64, align 8
  store i32 0, ptr %retval, align 4
  %call = call i32 @pthread_create(ptr noundef %t, ptr noundef null, ptr noundef @Thread1, ptr noundef null) #4
  call void @__tsan_write4(ptr @Global)     *****TSAN INSTRUMENTATION*****
  store i32 43, ptr @Global, align 4
  call void @__tsan_read8(ptr %t)           *****TSAN INSTRUMENTATION*****
  %1 = load i64, ptr %t, align 8
  %call1 = call i32 @pthread_join(i64 noundef %1, ptr noundef null)
  call void @__tsan_read4(ptr @Global)      *****TSAN INSTRUMENTATION*****
  %2 = load i32, ptr @Global, align 4
  call void @__tsan_func_exit()             *****TSAN INSTRUMENTATION*****
  ret i32 %2
}

Answer 1

The function-level ThreadSanitizerPass checks (ThreadSanitizer.cpp:548) if the function you are operating on has the SanitizeThread attribute (before instrumenting loads and stores)

    // Instrument memory accesses only if we want to report bugs in the function.
    if (ClInstrumentMemoryAccesses && SanitizeFunction)                          
      for (const auto &II : AllLoadsAndStores) {                                 
        Res |= instrumentLoadOrStore(II, DL);                                    
      }
    }

I would suggest writing a custom pass that iterates over all functions and assigns the Attribute::SanitizeThread attribute to them before applying the ThreadSanitizer pass.