LUA_HOOKRET not working when function is called inside return

Question

I am trying to setup a small profiler and in consequence I need to record every time the program enters and leaves a function. The problem is that it doesn't work when the function called is inside a 'return' (eg. return f() ).

For instance in the code below, when executing the code 'missing_call_and_ret' we can only see the call to g (but not f) and we can't see the exit of f and g.

Does anyone know why it doesn't work ? I'm using lua 5.4.6

This code reproduce the problem:

#include <lauxlib.h>
#include <lua.h>
#include <lualib.h>
#include <stdlib.h>

void profile_hook(lua_State *L, lua_Debug *ar) {
  if (!lua_getinfo(L, "Sn", ar)) {
    exit(EXIT_FAILURE);
  }
  switch (ar->event) {
  case LUA_HOOKCALL:
    printf("Enter %s %s:%d\n", ar->name, ar->short_src, ar->linedefined);
    break;
  case LUA_HOOKRET:
    printf("Leave %s %s:%d\n", ar->name, ar->short_src, ar->linedefined);
    break;
  }
}

void main() {
  lua_State *L = luaL_newstate();
  luaL_openlibs(L);
  lua_sethook(L, profile_hook, LUA_MASKCALL | LUA_MASKRET, 0);
  const char *okay_ret = "function f() return 7 end\n"
                         "function g() local v=f() return v end\n"
                         "print(g())\n";

  const char *missing_call_and_ret = "function f() return 7 end\n"
                                     "function g() return f() end\n"
                                     "print(g())\n";

  printf("Okay ret:\n");
  if (luaL_dostring(L, okay_ret) != LUA_OK) {
    exit(EXIT_FAILURE);
  }

  printf("\nMissing call and ret:\n");
  if (luaL_dostring(L, missing_call_and_ret) != LUA_OK) {
    exit(EXIT_FAILURE);
  }
}

Answer 1

return f() is a tail call, it is explained in section 3.4.10:

A call of the form return functioncall not in the scope of a to-be-closed variable is called a tail call. Lua implements proper tail calls (or proper tail recursion): In a tail call, the called function reuses the stack entry of the calling function. Therefore, there is no limit on the number of nested tail calls that a program can execute. However, a tail call erases any debug information about the calling function.

This mechanism of Lua can avoid some stack overflow problems, but the disadvantage is that Lua will remove debugging informations.

You need to use LUA_HOOKTAILCALL to catch the event. There is also a istailcall property in lua_Debug that can help you find the corresponding return event (flag t needs to be specified).