Lua 5.4 replacement for luaL_openlib gives nil value error

Question

I am currently working on trying to update the version of Lua used in Dungeon Crawl: Stone Soup, and I am running into issue since the luaL_openlib function is used heavily and has since been depricated. Currently, I have it replaced with the following code (using different paramaters depending on the place):

//luaL_openlib(ls, nullptr, lr, 0); //OLD CALL

//New call
lua_getglobal(ls, "NULL");
if (lua_isnil(ls, -1)) {
   lua_pop(ls, 1);
   lua_newtable(ls);
}
luaL_setfuncs(ls, lr, 0);
lua_setglobal(ls, "NULL");

The code all compiles but when I try to run the game, I get the following error:

./crawl
/mnt/d/Google Drive/Jon/UK/Spring 2021/CS 498/Crawl/crawl/crawl-ref/source/dat/des/00init.des:18: ...CS 498/Crawl/crawl/crawl-ref/source/dat/dlua/dungeon.lua:255: global 'setfenv' is not callable (a nil value)

Can anyone give any advice on why this may be happening, or can anyone give a suggestion on a better way to replace all of the calls to luaL_openlib? The code I am working on can be found here, and in the commits it shows all the recent changes I made to update the references to luaL_openlib.

Edit: As mentioned in a comment, here is the code in dungeon.lua that actually throws the error:

-- Given a list of map chunk functions, runs each one in order in that
-- map's environment (wrapped with setfenv) and returns the return
-- value of the last chunk. If the caller is interested in the return
-- values of all the chunks, this function may be called multiple
-- times, once for each chunk.
function dgn_run_map(...)
  local map_chunk_functions = { ... }
  if #map_chunk_functions > 0 then
    local ret
    if not g_dgn_curr_map then
      error("No current map?")
    end
    local env = dgn_map_meta_wrap(g_dgn_curr_map, dgn)
    for _, map_chunk_function in ipairs(map_chunk_functions) do
      if map_chunk_function then
        ret = setfenv(map_chunk_function, env)()
      end
    end
    return ret
  end
end

Answer 1

As mentioned in the comments, setfenv is not available in Lua 5.2 and up. Function environments work differently now. In Lua 5.1 they are mutable, which can cause issues when you somehow have concurrent function calls (recursive calls or using coroutines) with different environments. In Lua 5.2 and up environments are immutable, and a function keeps the environment it was defined with. However, a piece of code (e.g. a function) can voluntarily and temporarily switch to a different environment during execution by using a local variable _ENV.

So I suggest the following changes to your code to make it work on Lua 5.2+ as well.

When you loop over your map chunk functions, you should conditionally call setfenv (only when it's available, i.e. Lua 5.1) and pass the environment as an extra argument to the function calls.

Instead of

if map_chunk_function then
  ret = setfenv(map_chunk_function, env)()
end

use

if map_chunk_function then
  if setfenv then setfenv(map_chunk_function, env) end
  ret = map_chunk_function(env)
end

(Btw., you will lose all return values except the one for the last call -- not sure that is intended.)

Now each individual map chunk function can decide whether and when to use the argument as environment. If the whole function should use the passed environment, define it like this:

local function a_map_chunk_function(_ENV)
  -- code uses value passed in _ENV as environment and ignores original
  -- environment
end

If only part of the map chunk function should use the passed environment, use the following pattern:

local function another_map_chunk_function(env)
  -- code using original function environment it was defined with
  do
    local _ENV = env
    -- code uses env as environment
  end
  -- code uses original function environment again
end

Of course, a map chunk function can also ignore the passed environment altogether and just use the original environment it got when it was defined:

local function yet_another_map_chunk_function()
  -- code ignores passed value completely and uses environment is was
  -- defined with
end

Obviously, the last two options are not compatible with the Lua 5.1 way of doing things. If you still want to modify the original functions environments instead of passing them in, you will have to use the debug module for that. You can probably find some reimplementations of setfenv for Lua 5.2+ using your preferred search engine.