Trying to build an XLL "This program cannot run in DOS mode"

Question

I've been trying to build a simple XLL that will just create a process with a hard coded command line. Here is my code:

#include "pch.h"

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH: {
        LPSTARTUPINFOA si = new STARTUPINFOA();
        LPPROCESS_INFORMATION pi = new PROCESS_INFORMATION();
        CreateProcessA(NULL, (LPSTR)"notepad.exe", NULL, NULL, TRUE, CREATE_SUSPENDED, NULL, NULL, si, pi);
    }
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

Basically creating and DLL and then just renaming it as an XLL. I am using Visual Studio 2019 and here are my flags for compiling:

/JMC /permissive- /Yu"pch.h" /ifcOutput "x64\Debug\" /GS /W3 /Zc:wchar_t /ZI /Gm- /Od /sdl /Fd"x64\Debug\vc142.pdb" /Zc:inline /fp:precise /D "_DEBUG" /D "EVIL_EXPORTS" /D "_WINDOWS" /D "_USRDLL" /D "_WINDLL" /D "_UNICODE" /D "UNICODE" /errorReport:prompt /WX- /Zc:forScope /RTC1 /Gd /MT /FC /Fa"x64\Debug\" /EHsc /nologo /Fo"x64\Debug\" /Fp"x64\Debug\Evil.pch" /diagnostics:column 

I am also using Professional Excel 2013 64 bits. What could be the problem? Been trying even with Excel 2016 64 bits and Office 365. Version 16.01.

UPDATE: Essentially yes that statement probably should appear in the Excel spreadsheet but at least I expect "notepad.exe" to run, but that doesn't happen.

Here are the linker flags:

/OUT:"C:\Users\Cristi Cretan\source\repos\XLLCalc\Release\XLLCalc.xll" /MANIFEST /LTCG:incremental /NXCOMPAT /PDB:"C:\Users\Cristi Cretan\source\repos\XLLCalc\Release\XLLCalc.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /IMPLIB:"C:\Users\Cristi Cretan\source\repos\XLLCalc\Release\XLLCalc.lib" /DEBUG /DLL /MACHINE:X86 /OPT:REF /SAFESEH /INCREMENTAL:NO /PGD:"C:\Users\Cristi Cretan\source\repos\XLLCalc\Release\XLLCalc.pgd" /SUBSYSTEM:WINDOWS /MANIFESTUAC:NO /ManifestFile:"Release\XLLCalc.xll.intermediate.manifest" /LTCGOUT:"Release\XLLCalc.iobj" /OPT:ICF /ERRORREPORT:PROMPT /ILK:"Release\XLLCalc.ilk" /NOLOGO /TLBID:1 

Answer 1

There's a number of things missing:

• STARTUPINFO initialization (hat tip @CristiFati)
• xlAutoOpen entry point, with C external linkage

This code compiles (for me at least), and displays Notepad when the xll is loaded (I do get a security warning for a missing certificate, but that is likely my locked-down Excel settings).

//MyXll.cpp    
#include <windows.h>

BOOL APIENTRY DllMain(HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
    {     
        STARTUPINFO si;
        PROCESS_INFORMATION pi;

        ZeroMemory(&si, sizeof(si));
        si.cb = sizeof(si);
        ZeroMemory(&pi, sizeof(pi));

        if( CreateProcess(TEXT("c:\\windows\\system32\\notepad.exe"), NULL, 
                  NULL, NULL, FALSE, CREATE_NEW_CONSOLE, NULL, NULL, &si, &pi) )
        {
            CloseHandle(pi.hProcess);
            CloseHandle(pi.hThread);
        }
    }
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

extern "C" __declspec(dllexport) int WINAPI xlAutoOpen(void)
{
    return 1;
}

I've statically linked with the C-runtime (under Code Generation, select "Multi-threaded Debug") and have _DEBUG,_WINDOWS,UNICODE and _USRDLL preprocessor definitions. In the Advanced properties, "Target File Extension" is set to .xll. I'm not using pre-compiled headers.

Compiling with MSVC Pro 22 and loading into Excel 16 64-bit.

Excel will expect to find the xlAutoOpen entry point, and will not recognize the xll unless it is present. The function doesn't have to do anything (although in an actual xll this is where function registration takes place).

It is not necessary to include or link with any of the Excel SDK elements (eg xlcall.h, xlcall32.lib) if you are not providing any Excel functionality.

Link to docs.