Crash in Win32 App while calling "GetOpenFileName"

Question

I have one win32 prog where i want to call GetOpenFileName() but every time it got crash and CommDlgExtendedError() return error code as 1008. Please help me out.

char Filestring[MAX_PATH] = "\0";
OPENFILENAME ofn = { 0 };
ofn.lStructSize = sizeof(OPENFILENAME);
ofn.lpstrFile = (LPWSTR)Filestring;
ofn.nMaxFile = MAX_PATH;
if (GetOpenFileName(&ofn) == TRUE)
{

}

else
{
    int err = CommDlgExtendedError();
}

Answer 1

You are using the TCHAR version of the GetOpenFileName() API, which maps to either the ANSI (GetOpenFileNameA()) or Unicode (GetOpenFileNameW()) API depending on whether UNICODE is defined.

Since ofn.lpstrFile is accepting a LPWSTR (wchar_t*) pointer, expecting it to point at a wchar[] buffer, your project clearly has UNICODE defined. But you are giving it a type-casted pointer to a char[] buffer instead, which has 1/2 the storage space than you are telling the API is available for it to write to.

Your type-cast on the ofn.lpstrFile field is lying to the API. The code is calling the Unicode API, so it will write Unicode data to your char[] buffer, giving you mojibake output, and risking a buffer overflow.

You should use a TCHAR[] buffer instead to match what the API is expecting, eg:

TCHAR Filestring[MAX_PATH] = TEXT("\0");
OPENFILENAME ofn = { 0 };
ofn.lStructSize = sizeof(ofn);
ofn.lpstrFile = Filestring;
ofn.nMaxFile = MAX_PATH;
if (GetOpenFileName(&ofn))
{
    ...
}
else
{
    int err = CommDlgExtendedError();
    ...
}

Otherwise, use the ANSI or Unicode API explicitly, depending on your desired character encoding, eg:

CHAR Filestring[MAX_PATH] = "\0";
OPENFILENAMEA ofn = { 0 };
ofn.lStructSize = sizeof(ofn);
ofn.lpstrFile = Filestring;
ofn.nMaxFile = MAX_PATH;
if (GetOpenFileNameA(&ofn))
{
    ...
}
else
{
    int err = CommDlgExtendedError();
    ...
}

WCHAR Filestring[MAX_PATH] = L"\0";
OPENFILENAMEW ofn = { 0 };
ofn.lStructSize = sizeof(ofn);
ofn.lpstrFile = Filestring;
ofn.nMaxFile = MAX_PATH;
if (GetOpenFileNameW(&ofn))
{
    ...
}
else
{
    int err = CommDlgExtendedError();
    ...
}