How to enable a menu that I added to external program on VB6

Question

DISCLAIMER: I USED THIS So. I tried to add an popup menu to another programs with VB6 And my Notepad test was successful. Then I tried same code with Paint but the menu item was grayed out, even if i tried alot of ways to enable a menu.

Here is code:

Option Explicit
 
Const MENUID = 56 'our unique identifier
Dim hWndSubMenu As Long
'Forgot these declares in a previous post
Const MF_ENABLED = &H0
Const MF_GRAYED = &H1
Const MF_DISABLED = &H2

Private Declare Function EnableMenuItem Lib "user32" ( _
    ByVal hMenu As Long, _
    ByVal wIDEnableItem As Long, _
    ByVal wEnable As Long _
) As Long

Private Type MENUITEMINFO
    cbSize As Long
    fMask As Long
    fType As Long
    fState As Long
    wID As Long
    hSubMenu As Long
    hbmpChecked As Long
    hbmpUnchecked As Long
    dwItemData As Long
    dwTypeData As String
    cch As Long
End Type
 
Private Const MIIM_STATE = &H1
Private Const MIIM_ID = &H2
Private Const MIIM_STRING = &H40
Private Const MIIM_FTYPE = &H100
 
Private Const MFT_SEPARATOR = &H800
Private Const MFT_STRING = &H0
Private Const MFS_ENABLED = &H0
Private Const MFS_CHECKED = &H8
 
Private Const WM_COMMAND = &H111
Private Const MF_BYCOMMAND = &H0&
 
Private Declare Function FindWindow Lib "user32" Alias "FindWindowA" ( _
    ByVal lpClassName As String, _
    ByVal lpWindowName As String _
) As Long
 
Private Declare Function GetMenu Lib "user32" ( _
    ByVal hwnd As Long _
) As Long
 
Private Declare Function GetSubMenu Lib "user32" ( _
    ByVal hMenu As Long, _
    ByVal nPos As Long _
) As Long
 
Private Declare Function RemoveMenu Lib "user32" ( _
    ByVal hMenu As Long, _
    ByVal nPosition As Long, _
    ByVal wFlags As Long _
) As Long
 
Private Declare Function GetMenuItemCount Lib "user32.dll" ( _
    ByVal hMenu As Long _
) As Long
 
Private Declare Function InsertMenuItem Lib "user32.dll" Alias "InsertMenuItemA" ( _
    ByVal hMenu As Long, _
    ByVal uItem As Long, _
    ByVal fByPosition As Long, _
    lpmii As MENUITEMINFO _
) As Long
 
Private Declare Function DrawMenuBar Lib "user32" ( _
    ByVal hwnd As Long _
) As Long

Const WM_INITMENUPOPUP = &H117

Private Declare Function SendMessage Lib "user32" Alias "SendMessageA" ( _
    ByVal hwnd As Long, _
    ByVal wMsg As Long, _
    ByVal wParam As Long, _
    ByVal lParam As Long _
) As Long
 
Private Sub Command1_Click()
Dim hWndNotepad As Long
Dim hWndMenu As Long
Dim count As Long
Dim mii As MENUITEMINFO
Dim appCaption As String
 
'Get a handle to Notepad
appCaption = "untitled - Paint"
hWndNotepad = FindWindow(vbNullString, appCaption)
While hWndNotepad = 0
    If MsgBox("Open Notepad", vbOKCancel) = vbCancel Then Exit Sub
    hWndNotepad = FindWindow(vbNullString, appCaption)
Wend
 
'Get handle to Notepad's main menu
hWndMenu = GetMenu(hWndNotepad)
 
'Get handle to 'File' submenu
hWndSubMenu = GetSubMenu(hWndMenu, 5)
 
'Get number of items in sub menu
count = GetMenuItemCount(hWndSubMenu)
 
'Setup data structure for InsertMenuItem call
    With mii
        .cbSize = Len(mii)
        .fMask = MIIM_STATE Or MIIM_ID Or MIIM_STRING Or MIIM_FTYPE
        .fType = MFT_STRING
        .fState = MFS_ENABLED
        .wID = MENUID 'our unique identifier
        .dwTypeData = "My New Menu"
        .cch = Len(.dwTypeData)
    End With
    
    ' Add this to the menu.
    Call InsertMenuItem(hWndSubMenu, count + 1, 1, mii)
    Call DrawMenuBar(hWndNotepad)
    
    'test shet
    Call SendMessage(hWndNotepad, WM_INITMENUPOPUP, hWndSubMenu, 3)
 Call EnableMenuItem(hWndSubMenu, MENUID, MF_BYCOMMAND Or MF_ENABLED)
     Call SendMessage(hWndNotepad, WM_INITMENUPOPUP, hWndSubMenu, 3)
      Call EnableMenuItem(hWndSubMenu, MENUID, MF_ENABLED)
      SendMessage hwnd, WM_COMMAND, 56, 0
            Call EnableMenuItem(hWndSubMenu, MENUID, MF_ENABLED)
'Now set up hook, we want to monitor the WM_COMMAND message
 With Hook
    .TargethWnd = hWndNotepad
    .AddMessage WM_COMMAND, "WM_COMMAND"
        .AddMessage WM_INITMENUPOPUP, "WM_INITMENUPOPUP"
    
    .SetHook
 End With
End Sub
 
 
Private Sub Form_Unload(Cancel As Integer)
'remove our hook
 Hook.RemoveAllHooks
'remove our menu item
 Call RemoveMenu(hWndSubMenu, MENUID, MF_BYCOMMAND)
End Sub
 
Private Sub Hook_PostedMessage(uMsg As Long, wParam As Long, lParam As Long)
'here is where our messages will arrive
'look for our special menu ID
If uMsg = WM_COMMAND Then
 If (wParam And &HFFFF) = MENUID Then
    'our menu item has been selected
    MsgBox "Why did you do that?"
 End If
 Else
 MsgBox wParam
 End If
End Sub

I tried to use the FState and EnableMenuItem functions to enable menu in diffrent ways. but none worked. Menu is grayed out

Answer 1

You cannot do that.

The entirety of the menus API (and supporting infrastructure) is designed and intended for private use inside an application. There is no public interface serving as a customization point across arbitrary applications.

The question asked has no general solution.

---

Menus roughly work like this: An application creates a menu and associates it with a window. When a user selects a menu item the system sends a WM_COMMAND message to the window associated with the menu.

A crucial part of making this work is the menu-item identifier:

Associated with each menu item is a unique, application-defined integer, called a menu-item identifier.

The code in question

.wID = MENUID 'our unique identifier

seems to have acknowledged that the ID needs to be unique but does nothing to make this a fact. Indeed, it cannot do this. While you can get the window procedure associated with a window, there's no API available to query for allocated menu-item IDs.

Ultimately, you cannot find a menu-item ID that is guaranteed to be unique (per window). Whatever you do from here on forward will always be subject to an ID collision, with a broken application following in lockstep.

If you are hell-bent on doing this you can get a second chance at updating a menu after the application's default processing is done. You'd need to set up a WH_CALLWNDPROCRET hook and retroactively re-enable the menu item(s) from your WM_INITMENUPOPUP handler.

The result is an application that is broken with a probability greater than zero. Breaking applications is not a sustainable engineering strategy.