[FIXED] How to embed python 3.8+ in a C++ application while using a virtual environment?

Issue

This is a self-answered question. I've found that there aren't any good examples of this specific situation, and the seemingly related questions don't address this use case.

My C++ application previously embedded Python 2.7 using a virtual environment. This was done using Py_SetPythonHome(venv_path) followed by Py_Initialize(). Currently, I am migrating the app to Python 3. Python 3.8 introduces the new Python Initialization Configuration API, which is now the preferred method of initialization. Additionally, it introduces the concept of an "isolated" initialization, which I would like to use here. However, when I try to use this API in a similar way to the Python 2.7 initialization by setting config.home, I get an initialization error that suggest that the base Python libraries could not be found.

My virtual environment is initialized as follows:

py -3.10 -m venv C:\path\to\venv

When I execute the following code:

PyConfig config;
PyConfig_InitIsolatedConfig(&config);

auto venv_path = L"C:\\path\\to\\venv";
PyConfig_SetString(&config, &config.home, venv_path);

status = Py_InitializeFromConfig(&config);
PyConfig_Clear(&config);

if (PyStatus_Exception(status)) {
    std::cout << "status.func: " << (status.func ? status.func : "N/A") << '\n';
    std::cout << "status.err_msg: " << (status.err_msg ? status.err_msg : "N/A") << '\n';
}

I get the following output and error message:

Python path configuration:
  PYTHONHOME = 'C:\path\to\venv'
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 1
  environment = 0
  user site = 0
  import site = 1
  sys._base_executable = 'C:\\demo\\build\\PythonDemo.exe'
  sys.base_prefix = 'C:\\path\\to\\venv'
  sys.base_exec_prefix = 'C:\\path\\to\\venv'
  sys.platlibdir = 'lib'
  sys.executable = 'C:\\demo\\build\\PythonDemo.exe'
  sys.prefix = 'C:\\path\\to\\venv'
  sys.exec_prefix = 'C:\\path\\to\\venv'
  sys.path = [
    'C:\\Python310\\python310.zip',
    'C:\\path\\to\\venv\\DLLs',
    'C:\\path\\to\\venv\\lib',
    'C:\\demo\\build',
  ]
status.func: init_fs_encoding
status.err_msg: failed to get the Python codec of the filesystem encoding

The initialization configuration documentation specifically mentions "The following configuration files are used by the path configuration: pyvenv.cfg [...]", suggesting that virtual environments should be properly handled during initialization. However, it doesn't seem like the initialization is finding that file based on only setting config.home, and assumes that C:\path\to\venv is a complete Python installation rather than a virtual environment.

I've found that manually setting base_prefix and base_exec_prefix to C:\Python310 (partially) resolves the issue. However, I do not want to hardcode the path to Python in my application, as the app's users may have installed Python somewhere else. Besides, the home path provided in pyvenv.cfg should be automatically used for these.

How do I get Py_InitializeFromConfig to properly handle my virtual environment?

Solution

Python has a built-in way of handling virtual environments using the site module.

This module is automatically imported during initialization. [...]

If a file named “pyvenv.cfg” exists one directory above sys.executable, sys.prefix and sys.exec_prefix are set to that directory and it is also checked for site-packages (sys.base_prefix and sys.base_exec_prefix will always be the “real” prefixes of the Python installation).

In short, instead of setting config.home to <venv>, set config.executable to <venv>\Scripts\python.exe. Setting config.home manually interferes with Python's automatic virtual environment detection.

PyConfig config;
PyConfig_InitIsolatedConfig(&config);

auto venv_executable = L"C:\\path\\to\\venv\\Scripts\\python.exe";
PyConfig_SetString(&config, &config.executable, venv_executable);

status = Py_InitializeFromConfig(&config);
PyConfig_Clear(&config);

if (PyStatus_Exception(status)) {
    Py_ExitStatusException(status);
}

PyRun_SimpleString("import sys; print(f'{sys.executable=}\\n{sys.prefix=}\\n{sys.exec_prefix=}\\n{sys.base_prefix=}\\n{sys.base_exec_prefix=}\\n{sys.path=}')");

The above will produce the following output, showing that the virtual environment has been successfully identified and that the proper values are used for the various sys attributes.

sys.executable='C:\\path\\to\\venv\\Scripts\\python.exe'
sys.prefix='C:\\path\\to\\venv'
sys.exec_prefix='C:\\path\\to\\venv'
sys.base_prefix='C:\\Python310'
sys.base_exec_prefix='C:\\Python310'
sys.path=['C:\\Python310\\python310.zip', 'C:\\Python310\\DLLs', 'C:\\Python310\\lib', 'C:\\Python310', 'C:\\path\\to\\venv', 'C:\\path\\to\\venv\\lib\\site-packages']

Answered By - Martin.