How to experience cache miss and hits in Linux system?

Question

Hello I've been trying to experience cache miss and hits in Linux. To do so, I've done a program in C, where I mesure the time in CPU cycle to do the instruction printf(). The first part mesure the time needed for a miss and the second one for a hit. Here is the given program :

#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sched.h>
#include <sys/types.h> 
#include <unistd.h> 
#include <signal.h>

uint64_t rdtsc() {
  uint64_t a, d;
  asm volatile ("mfence");
  asm volatile ("rdtsc" : "=a" (a), "=d" (d));
  a = (d<<32) | a;
  asm volatile ("mfence");
  return a;
}

int main(int argc, char** argv)
{
    size_t time = rdtsc();
    printf("Hey ");
    size_t delta1 = rdtsc() - time;
    printf("delta: %zu\n", delta1);

    size_t time2 = rdtsc();
    printf("Hey ");
    size_t delta2 = rdtsc() - time2;
    printf("delta: %zu\n", delta2);
    sleep(100);
} 

Now I would like to show that two processes (two terminals) have cache in commun. So I thought that running this program in two terminals would result in :

Terminal 1:
miss
hit
Terminal 2:
hit 
hit

But now I have something like:

Terminal 1:
miss
hit
Terminal 2:
miss 
hit

Is my understanding incorrect? Or my program wrong?

Answer 1

Your assumption is somewhat correct.
printf is part of the libc library. If you use dynamic linking, the operating system may optimize memory usage by only loading the library once for all processes using it.
However, there are multiple reasons why I don't expect you to measure any sizable difference:

1. compared to the difference between a cache hit and cache miss, printf takes an enormous amount of time to complete and there is a lot going on that introduces noise. With just a single measurement, it is very unlikely that you're able to measure that tiny difference.
2. the actual reason for the first measurement to take longer is likely the lazy binding of the library function printf being resolved by the loader (https://maskray.me/blog/2021-09-19-all-about-procedure-linkage-table) or some other magic happening (buffers being setup, etc.) for the first output.
3. a lot of libc functions are used by many different processes. If the library is shared, it is likely, that printf may be cached even though you did not use it.

I would suggest to mount a Flush+Reload attack (https://eprint.iacr.org/2013/448.pdf) on printf in one of the terminals and use it in the other terminal. Then, you may see a timing difference.
Note: to find the actual address of printf for the attack, you need to be familiar with dynamic linking and the plt. Just using something like void* addr = printf will probably not work!