How to embed python 3.8+ in a C++ application while using a virtual environment?

Question

This is a self-answered question. I've found that there aren't any good examples of this specific situation, and the seemingly related questions don't address this use case.

My C++ application previously embedded Python 2.7 using a virtual environment. This was done using Py_SetPythonHome(venv_path) followed by Py_Initialize(). Currently, I am migrating the app to Python 3. Python 3.8 introduces the new Python Initialization Configuration API, which is now the preferred method of initialization. Additionally, it introduces the concept of an "isolated" initialization, which I would like to use here. However, when I try to use this API in a similar way to the Python 2.7 initialization by setting config.home, I get an initialization error that suggest that the base Python libraries could not be found.

My virtual environment is initialized as follows:

py -3.10 -m venv C:\path\to\venv

When I execute the following code:

PyConfig config;
PyConfig_InitIsolatedConfig(&config);

auto venv_path = L"C:\\path\\to\\venv";
PyConfig_SetString(&config, &config.home, venv_path);

status = Py_InitializeFromConfig(&config);
PyConfig_Clear(&config);

if (PyStatus_Exception(status)) {
    std::cout << "status.func: " << (status.func ? status.func : "N/A") << '\n';
    std::cout << "status.err_msg: " << (status.err_msg ? status.err_msg : "N/A") << '\n';
}

I get the following output and error message:

Python path configuration:
  PYTHONHOME = 'C:\path\to\venv'
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 1
  environment = 0
  user site = 0
  import site = 1
  sys._base_executable = 'C:\\demo\\build\\PythonDemo.exe'
  sys.base_prefix = 'C:\\path\\to\\venv'
  sys.base_exec_prefix = 'C:\\path\\to\\venv'
  sys.platlibdir = 'lib'
  sys.executable = 'C:\\demo\\build\\PythonDemo.exe'
  sys.prefix = 'C:\\path\\to\\venv'
  sys.exec_prefix = 'C:\\path\\to\\venv'
  sys.path = [
    'C:\\Python310\\python310.zip',
    'C:\\path\\to\\venv\\DLLs',
    'C:\\path\\to\\venv\\lib',
    'C:\\demo\\build',
  ]
status.func: init_fs_encoding
status.err_msg: failed to get the Python codec of the filesystem encoding

The initialization configuration documentation specifically mentions "The following configuration files are used by the path configuration: pyvenv.cfg [...]", suggesting that virtual environments should be properly handled during initialization. However, it doesn't seem like the initialization is finding that file based on only setting config.home, and assumes that C:\path\to\venv is a complete Python installation rather than a virtual environment.

I've found that manually setting base_prefix and base_exec_prefix to C:\Python310 (partially) resolves the issue. However, I do not want to hardcode the path to Python in my application, as the app's users may have installed Python somewhere else. Besides, the home path provided in pyvenv.cfg should be automatically used for these.

How do I get Py_InitializeFromConfig to properly handle my virtual environment?

Answer 1

Python has a built-in way of handling virtual environments using the site module.

This module is automatically imported during initialization. [...]

If a file named “pyvenv.cfg” exists one directory above sys.executable, sys.prefix and sys.exec_prefix are set to that directory and it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the “real” prefixes of the Python installation).

In short, instead of setting config.home to <venv>, set config.executable to <venv>\Scripts\python.exe. Setting config.home manually interferes with Python's automatic virtual environment detection.

PyConfig config;
PyConfig_InitIsolatedConfig(&config);

auto venv_executable = L"C:\\path\\to\\venv\\Scripts\\python.exe";
PyConfig_SetString(&config, &config.executable, venv_executable);

status = Py_InitializeFromConfig(&config);
PyConfig_Clear(&config);

if (PyStatus_Exception(status)) {
    Py_ExitStatusException(status);
}

PyRun_SimpleString("import sys; print(f'{sys.executable=}\\n{sys.prefix=}\\n{sys.exec_prefix=}\\n{sys.base_prefix=}\\n{sys.base_exec_prefix=}\\n{sys.path=}')");

The above will produce the following output, showing that the virtual environment has been successfully identified and that the proper values are used for the various sys attributes.

sys.executable='C:\\path\\to\\venv\\Scripts\\python.exe'
sys.prefix='C:\\path\\to\\venv'
sys.exec_prefix='C:\\path\\to\\venv'
sys.base_prefix='C:\\Python310'
sys.base_exec_prefix='C:\\Python310'
sys.path=['C:\\Python310\\python310.zip', 'C:\\Python310\\DLLs', 'C:\\Python310\\lib', 'C:\\Python310', 'C:\\path\\to\\venv', 'C:\\path\\to\\venv\\lib\\site-packages']

Answer 2

It seems like you are encountering issues with Py_InitializeFromConfig when trying to initialize Python 3 within a virtual environment. To properly handle your virtual environment during initialization, you might need to consider a different approach.

Instead of setting config.home directly, you can leverage the PyConfig_SetBytes function to set the home directory in bytes, which is expected for virtual environments. Additionally, you can use PyConfig_SetString to set the home directory as a string for compatibility.

Here's an adjusted version of your code:

    PyConfig config;
    PyConfig_InitIsolatedConfig(&config);
    
    const wchar_t* venv_path = L"C:\\path\\to\\venv";
    PyConfig_SetBytes(&config, &config.home, reinterpret_cast<const uint8_t*>(venv_path), wcslen(venv_path) * sizeof(wchar_t));
    
    status = Py_InitializeFromConfig(&config);
    PyConfig_Clear(&config);
    
    if (PyStatus_Exception(status)) {
        std::cout << "status.func: " << (status.func ? status.func : "N/A") << '\n';
        std::cout << "status.err_msg: " << (status.err_msg ? status.err_msg : "N/A") << '\n';
    }

By using PyConfig_SetBytes and passing the bytes representation of the home directory, you ensure proper handling of the virtual environment during initialization. This should help resolve the issue you're facing with Py_InitializeFromConfig in the context of Python 3 and virtual environments.