Sorry for the long title but it summarises very well the bug I just discovered in MFC after a long debugging session...
The issue is as follows and is easily reproducible:
1/ Create a default MFC application (SDI or MDI, doesn't matter).
2/ Edit its default Menu resource and add the following 5 entries in one of the menus:
- Command1
- - Separator -
- Command2
- Command3
- - Separator -
- Command4
3/ Upon initializing the CFrameWnd-derived main window, set the IDs of Command2 and Command3 as non-permitted. This is done through CMFCToolbar::SetNonPermittedCommands.
4/ Now compile and launch, then click on the menu entry you've modified in the menu bar. You'll get the following results:
- Command1
- - Separator -
- - Separator -
- Command4
I was expecting to see only one Separator, which clearly indicates a bug.
Now, I've found where the problem lies. If you go to the MFC source files, in "afxpopupmenubar.cpp", you'll see the member function called "CMFCPopupMenuBar::ImportFromMenu". At first sight, it looks like the developers took care of successive separators with the use of BOOL bPrevWasSeparator. The problem is that it's reset to FALSE whenever a non-separator menu item is added even if it's been rejected because it's part of the non-permitted commands!
The culprit part lies after the "if (m_bTrackMode || bShowAllCommands || CMFCMenuBar::IsShowAllCommands() || !CMFCToolBar::IsCommandRarelyUsed(uiCmd) || m_bPaletteMode)" statement. In there, we've found a command ID that is not a separator. A few lines down, we add a button for this command to the popup menu bar through "iIndex = InsertButton(item);". This is where the magic of SetNonPermittedCommands comes into play: within this "InsertButton" function, we test if the command is permitted or not. If it is, the button is added and the index returned. If it isn't we return -1 as the index.
A few lines down again, we see that there's a check on iIndex to make sure the button has been added ("if (iIndex >= 0)"). Trouble is, the reset of BOOLs "bPrevWasSeparator" and "bFirstItem" is down outside of this "if" statement, which means that even if the button has not been added, we're considering that the previous item was not a separator!
The fix is easy, simply put the two BOOLs reset within the "if" statement, and the bug will be gone. Now, I realise we won't get a fix for this tomorrow so I'll have to find a way to recompile a custom version of MFC that fixes the issue...
Best regards,
Eric