how fio loads various io engines when it starts?

Question

fio supports a whole bunch of io engines - all supported engines are present here : https://github.com/axboe/fio/tree/master/engines

I have been trying to understand the internals of how fio works and got stuck on how fio loads all the io engines.

For example I see every engine has a method to register and unregister itself, for example sync.c registers and unregisters using the following methods

fio_syncio_register : https://github.com/axboe/fio/blob/master/engines/sync.c#L448

and fio_syncio_unregister : https://github.com/axboe/fio/blob/master/engines/sync.c#L461

My question is who calls these methods ?

To find answer I tried running fio under gdb - placed a break point in fio_syncio_register and in the main function, fio_syncio_register gets called even before main which tells me it has something to do with __libc_csu_init and backtrace confirmed that

(gdb) bt
#0  fio_syncio_register () at engines/sync.c:450
#1  0x000000000047fb9d in __libc_csu_init ()
#2  0x00007ffff6ee27bf in __libc_start_main (main=0x40cd90 <main>, argc=2, argv=0x7fffffffe608, init=0x47fb50 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe5f8)
    at ../csu/libc-start.c:247
#3  0x000000000040ce79 in _start ()

I spent sometime reading about __libc_csu_init and __libc_csu_fini and every single description talks about methods being decorated with __attribute__((constructor)) will be called before main, but in the case of fio sync.c I dont see fio_syncio_register decorated with __attribute__

Can someone please help me out in understanding how this flow works ? Are there other materials I should be reading to understand this ?

Thanks

Answer 1

Interesting question. I couldn't figure out the answer looking at the source, so here are the steps I took:

$ make
$ find . -name 'sync.o'
./engines/sync.o

$ readelf -WS engines/sync.o | grep '\.init'
  [12] .init_array       INIT_ARRAY      0000000000000000 0021f0 000008 00  WA  0   0  8
  [13] .rela.init_array  RELA            0000000000000000 0132a0 000018 18     36  12  8

This tells us that global initializers are present in this object. These are called at program startup. What are they?

$ objdump -Dr engines/sync.o | grep -A4 '\.init'
Disassembly of section .init_array:

0000000000000000 <.init_array>:
    ...
            0: R_X86_64_64  .text.startup

Interesting. There is apparently a special .text.startup section. What's in it?

$ objdump -dr engines/sync.o | less
...
Disassembly of section .text.startup:

0000000000000000 <fio_syncio_register>:
   0:   48 83 ec 08             sub    $0x8,%rsp
   4:   bf 00 00 00 00          mov    $0x0,%edi
                        5: R_X86_64_32  .data+0x380
   9:   e8 00 00 00 00          callq  e <fio_syncio_register+0xe>
                        a: R_X86_64_PC32        register_ioengine-0x4
...

Why, it's exactly the function we are looking for. But how did it end up in this special section? To answer that, we can look at preprocessed source (in retrospect, I should have started with that).

How could we get it? The command line to compile sync.o is hidden. Looking in Makefile, we can unhide the command line with QUIET_CC=''.

$ rm engines/sync.o && make QUIET_CC=''
gcc -o engines/sync.o -std=gnu99 -Wwrite-strings -Wall -Wdeclaration-after-statement -g -ffast-math  -D_GNU_SOURCE -include config-host.h -I. -I. -O3 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -DBITS_PER_LONG=64 -DFIO_VERSION='"fio-2.16-5-g915ca"' -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DFIO_INTERNAL -DFIO_INC_DEBUG -c engines/sync.c
  LINK fio

Now we know the command line, and can produce preprocessed file:

$ gcc -E -dD -std=gnu99 -ffast-math  -D_GNU_SOURCE -include config-host.h -I. -I. -O3 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -DBITS_PER_LONG=64 -DFIO_VERSION='"fio-2.16-5-g915ca"' -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DFIO_INTERNAL -DFIO_INC_DEBUG engines/sync.c -o /tmp/sync.i

Looking in /tmp/sync.i, we see:

static void __attribute__((constructor)) fio_syncio_register(void)
{
 register_ioengine(&ioengine_rw);
 register_ioengine(&ioengine_prw);
...

Hmm, it is __attribute__((constructor)) after all. But how did it get there? Aha! I missed the fio_init on this line:

static void fio_init fio_syncio_register(void)

What does fio_init stand for? Again in /tmp/sync.i:

#define fio_init __attribute__((constructor))

So that is how it works.