Overload COM class methods with methods from a standard module using VTable hacking

Question

Quick question - I've just been testing overwriting methods of a class by changing entries in its VTable using low level copy memory api.

Background

I've had some success, and can swap 2 entries in the VTable of a class if they have the same signature. So a class definition like this:

Option Explicit

Public Sub Meow()
    Debug.Print "Meow"
End Sub

Public Sub Woof()
    Debug.Print "Woof"
End Sub 

... generates a VTable like this:

... and I can swap the entries at positions 7 and 8 to make cls.Meow print Woof and vice-versa. I can also swap the entry from the VTable of one class with the VTable of an entirely different one (provided I don't try to dereference the implicit this pointer by calling Me.anything)

So I can make another class

Option Explicit

Public Sub Tweet()
    Debug.Print "Tweet"
End Sub

and swap the behaviour of Woof from one with Tweet from the other. Not too complicated, I can share the code if people need it.

What I can't do...

... however, is figure out how to swap a class method with a method from a standard module?

Based on this article, it appears that the COM machinery which VBA is built on requires 2 things of class methods which VBA hides:

• They have an implicit this pointer
• They return an HRESULT (typedef long)

So I thought

Public Sub Meow()

in a class module Class1 is equivalent to

Public Function Meow(ByVal this As LongPtr) As Long

I've also tried

Public Function Meow(ByRef meObj As Class1) As Long
Public Function Meow(ByRef meObj As Class1) As LongPtr 'but HResult is 32 bit int
Public Sub Meow(ByVal this As LongPtr)

etc. But VBA always crashes when I try to invoke the method from the VTable. So I'm at a bit of a loss. I wonder if things are different on a 64 bit computer, or if the standard module functions do something weird to the calling stack. The thing is I've seen examples of code where the entire VTable is assembled from standard module functions, so I know it's possible but just not sure how to convert the signatures correctly

How can I overwrite a VTable entry with a method defined in a standard module?

Answer 1

I was only partially corect in my comment on your question. I still believe that the Me keyword plays a role in preventing the 'redirection' of a class method to a method inside a standard .bas module. But that is only applicable to early binding.

IDispatch::Invoke can actually call a method inside a .bas module with no problem. Your initial method signature was correct:

Public Function Meow(ByRef meObj As Class1) As Long

Class1 code:

Option Explicit

Public Sub Meow()
    Debug.Print "Meow"
End Sub

Public Sub Woof()
    Debug.Print "Woof"
End Sub

Code in a standard .bas module:

Option Explicit

Sub Test()
    Dim c As Object 'Must be late-binded!
    Dim vTblPtr As LongPtr
    Dim vTblMeowPtr As LongPtr
    Dim originalMeow As LongPtr
    '
    Set c = New Class1
    c.Meow 'Prints "Meow" to the Immediate Window
    '
    'The address of the virtual table
    vTblPtr = MemLongPtr(ObjPtr(c))
    '
    'The address of the Class1.Meow method within the virtual table
    vTblMeowPtr = vTblPtr + 7 * PTR_SIZE
    '
    'The current address of the Class1.Meow method
    originalMeow = MemLongPtr(vTblMeowPtr)
    '
    'Replace the address of Meow with the one in a .bas module
    MemLongPtr(vTblMeowPtr) = VBA.Int(AddressOf Moew)
    '
    c.Meow 'Prints "Meow in .bas" to the Immediate Window
    '
    'Revert the original address
    MemLongPtr(vTblMeowPtr) = originalMeow
    '
    c.Meow 'Prints "Meow" to the Immediate Window
End Sub

Public Function Moew(ByVal this As Class1) As Long
    Debug.Print "Meow in .bas"
End Function

I've used LibMemory for the memory manipulation.

If you change the Meow class method to a Function instead of a Sub then you need to have an extra ByRef paramater at the end of the parameters list within the Meow method in the .bas module.

EDIT #1

I thought of the issue discussed in the comments below and the only reason I could come up with was that the IDispatch only works with a pointer to the IUnknown interface.

This means that:

Public Function Meow(ByRef this As Class1) As Long

will crash the Application

But, this works:

Public Function Moew(ByVal this As Class1) As Long
    Debug.Print "Meow in .bas"
End Function

because passing ByVal forces a QueryInterface and an AddRef on the IUnknown (with Release when exiting scope)

This also works:

Public Function Moew(ByRef this As IUnknown) As Long
    Debug.Print "Meow in .bas"
End Function

EDIT #2

Apologies for making another edit.

The Invoke method is not working with a pointer to IUnknown. It is working with a pointer to IDispatch. This can be checked with:

Public Function Moew(ByVal this As LongPtr) As Long
    Debug.Print this
    Debug.Print "Meow in .bas"
End Function

which will print the ptr to the IDispatch interface. So, why does ByRef this As Class1 fail? And why do ByVal this As Class1 and ByRef this As IUnknown work?

ByRef this As Class1
I believe the VarPtr(this) address is not accessible to VB, hence we are reading memory that we should not. It's not like there is an extra AddRef or Release on the IUnknown interface because the method never gets called using this declaration. The Application simply crashes when Invoke is trying to call the method.

ByVal this As Class1
The method simply creates a VB variable (on VB memory space) and calls AddRef

ByRef this As IUnknown
As this is not a dual interface, a call to QueryInterface and an AddRef is done. The memory address of 'this' is on local memory space, same as in the second example.