Problem getting correct parameters for C# P/Invoke call to C++ dll

Question

Trying to Interop a functionality from the Outside In API from Oracle. Have the following function:

SCCERR EXOpenExport(VTHDOC hDoc, VTDWORD dwOutputId, VTDWORD dwSpecType, 
                    VTLPVOID pSpec, VTDWORD dwFlags, VTSYSPARAM dwReserved,
                    VTLPVOID pCallbackFunc, VTSYSPARAM dwCallbackData, VTLPHEXPORT phExport);

From the header files I reduced the parameters to:

#define VTVOID void
typedef VTSYSPARAM VTHDOC, VTLPHDOC *;
typedef DWORD_PTR VTSYSPARAM;
typedef unsigned long DWORD_PTR;
typedef unsigned long VTDWORD;
typedef VTVOID* VTLPVOID;
typedef VTHDOC VTHEXPORT, *VTLPEXPORT;

These are for 32 bit windows.

Going through the header files, the example programs, and the documentation I found:

1. That pSpec could be a pointer to a buffer or NULL, so I set it to a IntPtr.Zero (documentation).
2. That dwFlags and dwReserved according to the documentation "Must be set by the developer to 0".
3. That pCallbackFunc can be set to NULL if I don't want to handle callbacks.
4. That the last two are based on structs that I wrote C# wrappers for using the [StructLayout(LayoutKind.Sequential)]. Then instatiated an instance and generated the parameters by first creating an IntPtr with Marshal.AllocHGlobal(Marshal.SizeOf(instance)), then getting the address value which is passed as a uint for dwCallbackData and a IntPtr for phExport.

The final parameter list is as follows:

1. phDoc as an IntPtr which was loaded with an address by the DAOpenDocument function called before.
2. dwOutputId as uint set to 1535 which represents FI_JPEGFIF
3. dwSpecType as int set to 2 which represents IOTYPE_ANSIPATH
4. pSpec as an IntPtr.Zero where the output will be written
5. dwFlags as uint set to 0 as directed
6. dwReserved as uint set to 0 as directed
7. pCallbackFunc as IntPtr set to NULL as I will handle results
8. dwCallBackData as uint the address of a buffer for a struct
9. phExport as IntPtr to another struct buffer

Still get an undefined error from the API. Meaning that the call returns a 961 which is not defined in any of the header files. In the past I have gotten this when my choice of parameter types are incorrect.

I started out using Interop Assistant which was helpful in learning how many of the parameter types get translated. It is however limited by how well I am able to glean the correct native type from the header files. For example the hDoc parameter used in the preceding function was defined as a non-filesytem handle, so attempted to use Marshal to create a handle, then used an IntPtr, and finally it turned out to be an int (actually it was &phDoc used here).

So is there a more scientific way of doing this, other than trial and error?

Jim

Answer 1

Your definition:

typedef unsigned long DWORD_PTR;

Is incorrect — DWORD_PTR is defined as:

typedef ULONG_PTR DWORD_PTR;

Which in turn is defined as:

typedef unsigned __int3264 ULONG_PTR;

The attribute __int3264 is documented here, I am reproducing the most important bit:

The keyword __int3264 specifies an integral type that has the following properties:

• It is 32-bit on 32-bit platforms
• It is 64-bit on 64-bit platforms
• It is 32-bit on the wire for backward compatibility. It gets truncated on the sending side and extended appropriately (signed or unsigned) on the receiving side.

When you are marshaling C "types" such as DWORD_PTR whose size depends on the platform pointer size which is often the case with the OS parameters (such as LPARAM, WPARAM, LRESULT, SIZE_T, etc) which the name VTSYSPARAM appears to suggest, you should use IntPtr, not uint.

Basically:

6. dwReserved as uint set to 0 as directed

8. dwCallBackData as uint the address of a buffer for a struct

Should be:

6. dwReserved as IntPtr set to IntPtr.Zero as directed

8. dwCallBackData as IntPtr the address of a buffer for a struct

Then it should work, and if it doesn't you need to make sure you didn't make a similar mistake when marshaling the necessary structures.