Delay-Load in Windows

Question

I am trying to get my head around some code (adapted straight from PyCXX). It is a multiplatform C++ Python wrapper.

EDIT: Original code here.

It appears to be catering for some particular phenomenon that only exists in Windows:

#ifdef PY_WIN32_DELAYLOAD_PYTHON_DLL
:
#else
:
#endif

I will give the complete file listing below, it is quite long.

This PY_WIN32_DELAYLOAD_PYTHON_DLL token doesn't exist within CPython, it is also not defined in PyCXX. Hence I can only imagine PyCXX intends that it is supplied as an optional compiler flag.

What I would like to know is: What is its purpose? What problem is it solving? Why does this mechanism even exist?

Maybe someone who is familiar with Windows programming can figure it out from the code?

I would like to know whether the problem it is solving is still present in modern Windows, as the code is >15 years old.

The key question is: Can I remove it, or replace it with something cleaner?

I would very much like to cut it out; but does it still serve some useful purpose on a modern Windows environment?

Code:

#include "Base.hxx" //"IndirectPythonInterface.hxx"

namespace Py
{

// = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

#ifdef PY_WIN32_DELAYLOAD_PYTHON_DLL

#   ifndef MS_WINDOWS
#       error "Can only delay load under Win32"
#   endif
#   include <windows.h> // !!! Is it possible we ever want Windows.h but no delay load? If so, this is wrong...

static HMODULE python_dll;

#   define DECLARE_ERROR( THE_ERROR ) \
        static PyObject* ptr__Exc_##THE_ERROR = nullptr;
ALL_ERRORS( DECLARE_ERROR )


static PyObject* ptr__PyNone        = nullptr;
static PyObject* ptr__PyFalse       = nullptr;
static PyObject* ptr__PyTrue        = nullptr;

#   define DECLARE_TYPE( T ) \
        static PyTypeObject* ptr__##T##_Type = nullptr;
ALL_TYPES( DECLARE_TYPE )


static int* ptr_Py_DebugFlag        = nullptr;
static int* ptr_Py_InteractiveFlag  = nullptr;
static int* ptr_Py_OptimizeFlag     = nullptr;
static int* ptr_Py_NoSiteFlag       = nullptr;
static int* ptr_Py_VerboseFlag      = nullptr;

static char** ptr__Py_PackageContext = nullptr;

#   ifdef Py_REF_DEBUG
int* ptr_Py_RefTotal; // !!! Why not static?
#   endif


//--------------------------------------------------------------------------------
class GetAddressException
{
public:
    GetAddressException( const char* _name )
        : name( _name )
    { }
    virtual ~GetAddressException() { }
    const char* name;
};


//--------------------------------------------------------------------------------

#   define GET_PTR( FUNC, RETURN_TYPE ) \
        static FUNC( const char *name ) \
            { \
                FARPROC addr = GetProcAddress( python_dll, name ); \
                if( addr == nullptr ) \
                    throw GetAddressException( name ); \
                \
                return RETURN_TYPE addr; \
            }

GET_PTR( PyObject *             GetPyObjectPointer_As_PyObjectPointer       , *(PyObject **)        )
GET_PTR( PyObject *                    GetPyObject_As_PyObjectPointer       ,  (PyObject *)         )
GET_PTR( PyTypeObject *     GetPyTypeObjectPointer_As_PyTypeObjectPointer   , *(PyTypeObject**)     )
GET_PTR( PyTypeObject *            GetPyTypeObject_As_PyTypeObjectPointer   ,  (PyTypeObject*)      )
GET_PTR( int *                              GetInt_as_IntPointer            ,  (int*)               )
GET_PTR( char **                    GetCharPointer_as_CharPointerPointer    ,  (char**)             )


#   ifdef _DEBUG
static const char python_dll_name_format[] = "PYTHON%1.1d%1.1d_D.DLL";
#   else
static const char python_dll_name_format[] = "PYTHON%1.1d%1.1d.DLL";
#   endif

//--------------------------------------------------------------------------------
bool InitialisePythonIndirectInterface()
{
    char python_dll_name[sizeof(python_dll_name_format)];

    _snprintf( python_dll_name, sizeof(python_dll_name_format) / sizeof(char) - 1, python_dll_name_format, PY_MAJOR_VERSION, PY_MINOR_VERSION );

    python_dll = LoadLibraryA( python_dll_name );
    if( python_dll == nullptr )
        return false;

    try
    {
#   ifdef Py_REF_DEBUG
        ptr_Py_RefTotal             = GetInt_as_IntPointer( "_Py_RefTotal" );
#   endif
        ptr_Py_DebugFlag            = GetInt_as_IntPointer( "Py_DebugFlag" );
        ptr_Py_InteractiveFlag      = GetInt_as_IntPointer( "Py_InteractiveFlag" );
        ptr_Py_OptimizeFlag         = GetInt_as_IntPointer( "Py_OptimizeFlag" );
        ptr_Py_NoSiteFlag           = GetInt_as_IntPointer( "Py_NoSiteFlag" );
        ptr_Py_VerboseFlag          = GetInt_as_IntPointer( "Py_VerboseFlag" );
        ptr__Py_PackageContext      = GetCharPointer_as_CharPointerPointer( "_Py_PackageContext" );

#   define ASSIGN_PTR( E ) \
        ptr__Exc_##E    = GetPyObjectPointer_As_PyObjectPointer( "PyExc_" #E );
        ALL_ERRORS( ASSIGN_PTR )

        ptr__PyNone                 = GetPyObject_As_PyObjectPointer( "_Py_NoneStruct" );
        ptr__PyFalse                = GetPyObject_As_PyObjectPointer( "_Py_ZeroStruct" );
        ptr__PyTrue                 = GetPyObject_As_PyObjectPointer( "_Py_TrueStruct" );

#   define MAKE_PTR( TYPENAME ) \
        ptr__##TYPENAME##_Type      = GetPyTypeObject_As_PyTypeObjectPointer( "Py" #TYPENAME "_Type" );
        ALL_TYPES( MAKE_PTR )
    }
    catch( GetAddressException &e )
    {
        OutputDebugStringA( python_dll_name );
        OutputDebugStringA( " does not contain symbol " );
        OutputDebugStringA( e.name );
        OutputDebugStringA( "\n" );

        return false;
    }

    return true;
}


//#if 0
//#define Py_INCREF(op) (                         \
//    _Py_INC_REFTOTAL  _Py_REF_DEBUG_COMMA       \
//    ((PyObject*)(op))->ob_refcnt++)
//
//#define Py_DECREF(op)                           \
//    if (_Py_DEC_REFTOTAL  _Py_REF_DEBUG_COMMA   \
//        --((PyObject*)(op))->ob_refcnt != 0)    \
//        _Py_CHECK_REFCNT(op)                    \
//    else                                        \
//        _Py_Dealloc((PyObject *)(op))
//#endif

void _XINCREF( PyObject* op )
{
    // This function must match the contents of Py_XINCREF(op)
    if( op == nullptr )
        return;

#   ifdef Py_REF_DEBUG
    (*ptr_Py_RefTotal)++;
#   endif
    (op)->ob_refcnt++;

}

void _XDECREF( PyObject* op )
{
    // This function must match the contents of Py_XDECREF(op);
    if( op == nullptr )
        return;

#   ifdef Py_REF_DEBUG
    (*ptr_Py_RefTotal)--;
#   endif

    if (--(op)->ob_refcnt == 0)
        _Py_Dealloc((PyObject *)(op));
}


#else // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


//
//    Needed to keep the abstactions for delayload interface
//
// !!! π Py_XDECREF has been deprecated in favour of Py_CLEAR

void _XINCREF( PyObject* op )
{
    Py_XINCREF( op );
}

void _XDECREF( PyObject* op )
{
    Py_XDECREF( op );
}

#endif // PY_WIN32_DELAYLOAD_PYTHON_DLL

// = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =


//
//    Wrap Python-types, type checks, errors, flags, etc as function calls
//

#ifdef PY_WIN32_DELAYLOAD_PYTHON_DLL
#   define IF_DELAYLOAD_ELSE( A, B ) A
#else
#   define IF_DELAYLOAD_ELSE( A, B ) B
#endif


#define _Foo_Check( TYPENAME ) \
    bool _##TYPENAME##_Check( PyObject *pyob )  \
    { \
        return pyob->ob_type == _##TYPENAME##_Type(); \
    }
ALL_TYPES( _Foo_Check )

#define _Foo_Type( TYPENAME ) \
    PyTypeObject* _##TYPENAME##_Type() \
    { \
        return IF_DELAYLOAD_ELSE( ptr__##TYPENAME##_Type, & Py##TYPENAME##_Type ); \
    }
ALL_TYPES( _Foo_Type )



#define _Exc_Foo( E ) \
    PyObject* _Exc_##E() \
    { \
        return IF_DELAYLOAD_ELSE( ptr__Exc_##E, ::PyExc_##E ); \
    }
ALL_ERRORS( _Exc_Foo )


int& _Py_DebugFlag()                    { return IF_DELAYLOAD_ELSE( *ptr_Py_DebugFlag       , Py_DebugFlag ); }
int& _Py_InteractiveFlag()              { return IF_DELAYLOAD_ELSE( *ptr_Py_InteractiveFlag , Py_InteractiveFlag ); }
int& _Py_OptimizeFlag()                 { return IF_DELAYLOAD_ELSE( *ptr_Py_OptimizeFlag    , Py_OptimizeFlag ); }
int& _Py_NoSiteFlag()                   { return IF_DELAYLOAD_ELSE( *ptr_Py_NoSiteFlag      , Py_NoSiteFlag ); }
int& _Py_VerboseFlag()                  { return IF_DELAYLOAD_ELSE( *ptr_Py_VerboseFlag     , Py_VerboseFlag ); }

char* __Py_PackageContext()             { return IF_DELAYLOAD_ELSE( *ptr__Py_PackageContext , _Py_PackageContext ); }


PyObject* _None()                       { return IF_DELAYLOAD_ELSE( ptr__PyNone             , &::_Py_NoneStruct ); }
PyObject* _False()                      { return IF_DELAYLOAD_ELSE( ptr__PyFalse            , Py_False ); }
PyObject* _True()                       { return IF_DELAYLOAD_ELSE( ptr__PyTrue             , Py_True ); }

} // namespace Py

Answer 1

On Win32, delay loading is a mechanism to allow a PE file to reference another PE file which is not where the loader expects it at the time that the file is launched, or to gracefully fall back if it's not there at all. It looks to me as if this is catering to a Windows program that is embedding python itself, but doesn't want to have the DLL containing python sitting in PATH.

Some googling further suggests that this is related to avoiding a circularity between python and a module made to be loaded by python.

Answer 2

Macros can be defined with pre-processor arguments, that is why you don't see them anywhere. With Microsoft compiler is with the /D argument.

/D has the same effect as the #define directive at the beginning of a source code file, except that /D strips quotation marks on the command line and #define retains them.

http://msdn.microsoft.com/en-us/library/hhzbb5c8.aspx

With gcc is with -D check this reference link:

https://www.daniweb.com/software-development/c/threads/348802/passing-string-as-d-compiler-option

Delayed load is a mechanism to load the library only when it is used instead of the OS doing it before the application launches. It can save precious memory and load time as (depending on code flow) the dll might not be loaded at all.

This code is trying to implement the delayed load by itself if the macro is defined else just go the normal way. Microsoft linker can do the work for you automatically (i.e. You don't need to program anything). Keep in mind this is not a feature of the platform but a linker feature.

Check this reference: http://en.wikipedia.org/wiki/Dynamic-link_library#Delayed_loading

You can do away with the code and instruct Microsoft linker to add the code for you if you want.

You can do it with the /DELAYLOAD argument, as explained in this article:

http://msdn.microsoft.com/en-us/library/yx9zd12s.aspx

Just make sure you catch the proper exceptions and hooks if the dll is not found.