I have an x64 MFC application (VS2019, fresh project from the wizard and filled with lots of old project code). Debug build is working fine but the project has a problem to start up in the Release build. Should be noted that it is a big project with several different subprojects and libraries included. (EDIT: pugi xml library seems to be causing the trouble, see below)
In AfxMainWin() function the pThread and pApp pointers are created correctly, and I can for example look at the command line arguments pThread->m_lpCmdLine. Then AfxMainWin calls pThread->InitInstance(), but inside the InitInstace function the this-pointer is corrupt, which leads to a crash when the code tries to access m_lpCmdLine or any other member-variable.
EDIT1: I found out that the this-pointer was okay on the first line in InitInstance(). Then a small init function was run that destroyed the this-pointer in InitInstance. This function reads settings from an xml file using Pugi Xml library, and specifically this line is the one that destroys my this pointer:
status = ((pugi::xml_document*)get_params())->load_file(file).status;
Probable reason: The pugixml.dll is incomaptible with my release build and causes memory corruption. I have used dependency walker on the pugixml.dll, and the result was not good. This DLL seems to be built in debug mode since it adds dependencies to the debug versions of MFC libraries, such as VCRUNTIME140D.DLL - and that is not good for a release build.
I do not have access to any release version of pugixml.dll, so my solution will be to throw out the pugixml.dll and include the .cpp in my project instead, or maybe use another xml reader instead. That should do the trick!
But I have a final question for you folks, so that I learn as much as possible from this: what is the most likely cause of my memory corruption? Is it the debug DLL that is incompatible with my release build? Or is it more likely that the DLL is old (v140 toolset) and my project is toolset v142? Or is it, as suggested by @Iinspectable , the actual function call (with c-style casting of the returned pointer) that is messed up and trashing my stack?
I note that the get_params() function returns a void* pointer (void* the_params;), but that void* pointer has earlier been created using the_params = new pugi::xml_document; - so at least it is being casted to the correct type.
EDIT2: after replacing the faulty pugixml.dll file with pugixml.cpp I could debug the issue, turns out to be some pugixml-related problem. I had to post that as another issue, since it is now a very different problem.