Windows Server 2003 Version 3790 (Service Pack 2) UP Free x86 compatible
This is a MFC console application (MFC in a shared DLL) that was converted to call our
C# assembly using the technique described at this URL: http://support.microsoft.com/kb/828736/en-us
The /clr switch is used.
The text of the application popup window is as follows:
Application popup: MyProgram.exe - Application Error : The instruction at "0x7c82c912" referenced memory at "0x00000000". The memory could not be "read".
The server is not configured to capture .NET dumps, so while the application was hung - still displaying the popup message box,
I manually created the dump as follows: ntsd.exe -pvr -p 2812 -c ".dump /mfh /u c:\debug\user.dmp; q"
This issue is very intermittent and I have not been able to reproduce it in a dev environment. The programs that have the issue never seems to actually make any calls to the C# DLL and are only running for approx. 5 - 10 seconds based on its log file.
As a side note, the programs are kicked off by a multithreaded application (100% native) using the Windows "CreateProcess" function.
The call stack looks odd to me as "mswsock!_DllMainCRTStartup" is called after "clr!EEShutDown".
I have additional information if need be. I would appreciate any help as to how to determine what is causing this.
Here's the info from the dump:
**** ~*E !DUMPSTACK to output call stacks for managed threads
---------------------------------------------
OS Thread Id: 0x2534 (0)
Current frame: ntdll!KiFastSystemCallRet
ChildEBP RetAddr Caller,Callee
0012f040 7c8275c9 ntdll!ZwRaiseHardError+0xc
0012f044 77e828a8 kernel32!UnhandledExceptionFilter+0x51a , calling ntdll!NtRaiseHardError
0012f090 7c82a351 ntdll!RtlAllocateHeap+0x460 , calling ntdll!RtlLeaveCriticalSection
0012f094 7c829f6c ntdll!RtlAllocateHeap+0xee7 , calling ntdll!_SEH_epilog
0012f0d4 7c82a351 ntdll!RtlAllocateHeap+0x460 , calling ntdll!RtlLeaveCriticalSection
0012f0d8 7c829f6c ntdll!RtlAllocateHeap+0xee7 , calling ntdll!_SEH_epilog
0012f0ec 77e61d2b kernel32!WaitForSingleObjectEx+0xdc , calling kernel32!_SEH_epilog
0012f13c 7c829f94 ntdll!RtlpAllocateFromHeapLookaside+0x13 , calling ntdll!ExInterlockedPopEntrySList
0012f148 7c829f28 ntdll!RtlAllocateHeap+0x1dd , calling ntdll!RtlpAllocateFromHeapLookaside
0012f14c 7c829f6c ntdll!RtlAllocateHeap+0xee7 , calling ntdll!_SEH_epilog
0012f160 77c817ae rpcrt4!NdrpCheckCorrelation+0x71 , calling rpcrt4!NdrpValidateCorrelatedValue
0012f198 7c82fbfc ntdll!RtlIntegerToUnicode+0x126 , calling ntdll!_SEH_epilog
0012f1cc 791419a1 clr!StressLog::LogOn+0x1e , calling clr!StressLog::ETWLogOn
0012f1e0 793ba6e0 clr!InternalUnhandledExceptionFilter_Worker+0x19 , calling clr!StressLog::LogOn
0012f1e8 793ba719 clr!InternalUnhandledExceptionFilter_Worker+0x52 , calling clr!_SEH_epilog4
0012f210 603b4613 mscoreei!CLRUEFChainLockHolder::~CLRUEFChainLockHolder+0x14
0012f21c 603e478a mscoreei!CLRUEFManager::InvokeUEFCallbacks+0x62 , calling mscoreei!_EH_epilog3
0012f240 603e478a mscoreei!CLRUEFManager::InvokeUEFCallbacks+0x62 , calling mscoreei!_EH_epilog3
0012f244 7931b2c6 clr!InitGSCookie+0xca
0012f24c 7931b28f clr!InitGSCookie+0x93 , calling clr!_EH_epilog3
0012f26c 7c827589 ntdll!NtQueryVirtualMemory+0xc
0012f270 77e47f45 kernel32!_ValidateEH3RN+0xb6 , calling ntdll!ZwQueryVirtualMemory
0012f2b0 77e769c0 kernel32!BaseProcessStart+0x39 , calling kernel32!UnhandledExceptionFilter
0012f2b8 77e61aa9 kernel32!_except_handler3+0x61
0012f2e0 7c8285c2 ntdll!ExecuteHandler2+0x26
0012f304 7c828593 ntdll!ExecuteHandler+0x24 , calling ntdll!ExecuteHandler2
0012f328 7c831441 ntdll!RtlDispatchException+0x91 , calling ntdll!RtlpExecuteHandlerForException
0012f368 7c829dc9 ntdll!RtlFreeHeap+0x70f , calling ntdll!_SEH_epilog
0012f384 7c82cd6e ntdll!ARRAY_FITS+0x29 , calling ntdll!ULongAdd
0012f398 7c82cd20 ntdll!RtlpLocateActivationContextSection+0x1da , calling ntdll!ARRAY_FITS
0012f3ac 7c8283ce ntdll!KiUserExceptionDispatcher+0xe , calling ntdll!RtlDispatchException
0012f6ac 7c82c912 ntdll!RtlpCoalesceFreeBlocks+0x128 ====> Exception Code 0 cxr@12f3e0 exr@127000
0012f6b8 7c82a69b ntdll!RtlFreeHeap+0x38e , calling ntdll!RtlpCoalesceFreeBlocks
0012f75c 7c829dc9 ntdll!RtlFreeHeap+0x70f , calling ntdll!_SEH_epilog
0012f760 7c82fec8 ntdll!RtlpFreeDebugInfo+0x3c , calling ntdll!RtlFreeHeap
0012f774 7c82c8b8 ntdll!RtlDeleteCriticalSection+0xa5 , calling ntdll!RtlpFreeDebugInfo
0012f7a0 77bbcef6 msvcrt!free+0xc3 , calling ntdll!RtlFreeHeap
0012f7c0 71b215c0 mswsock!DllMain+0x190 , calling mswsock!_SEH_epilog
0012f7e8 71b260af mswsock!_CRT_INIT+0xa1 , calling msvcrt!free
0012f7f8 71b215dc mswsock!_DllMainCRTStartup+0x77 , calling mswsock!_CRT_INIT
0012f818 7c81a19a ntdll!LdrpCallInitRoutine+0x14
0012f838 7c830ce8 ntdll!LdrShutdownProcess+0x182 , calling ntdll!LdrpCallInitRoutine
0012f898 7c830cc7 ntdll!LdrShutdownProcess+0x15c , calling ntdll!RtlActivateActivationContextUnsafeFast
0012f8f0 77e6689b kernel32!_ExitProcess+0x43 , calling ntdll!LdrShutdownProcess
0012f904 7c829f94 ntdll!RtlpAllocateFromHeapLookaside+0x13 , calling ntdll!ExInterlockedPopEntrySList
0012f910 7c829f28 ntdll!RtlAllocateHeap+0x1dd , calling ntdll!RtlpAllocateFromHeapLookaside
0012f914 7c829f6c ntdll!RtlAllocateHeap+0xee7 , calling ntdll!_SEH_epilog
0012f968 603c4083 mscoreei!CQuickArrayBase<RuntimeDesc *>::ReSizeThrows+0x23 , calling mscoreei!CQuickMemoryBase<512,128>::_Alloc<1,1>
0012f978 603c4052 mscoreei!Enumerator<RuntimeDesc * const,SHash<RuntimeDescCache::DescSHashTraits>::Iterator>::operator+++0x2b , calling mscoreei!SHash<RuntimeDescCache::DescSHashTraits>::Index::Next
0012f984 603c4165 mscoreei!RuntimeDescCache::GetRuntimeDescList+0x44 , calling mscoreei!Enumerator<RuntimeDesc * const,SHash<RuntimeDescCache::DescSHashTraits>::Iterator>::operator!=
0012f988 603c41a1 mscoreei!RuntimeDescCache::GetRuntimeDescList+0x141 , calling mscoreei!_EH_epilog3
0012f9dc 77e668fd kernel32!ExitProcess+0x14 , calling kernel32!_ExitProcess
0012f9f0 603c42f0 mscoreei!RuntimeDesc::ShutdownAllActiveRuntimes+0x29c , calling kernel32!ExitProcess
0012fa98 7c827b69 ntdll!NtWaitForMultipleObjects+0xc
0012faac 77e6207f kernel32!WaitForMultipleObjectsEx+0x184 , calling ntdll!RtlDeactivateActivationContextUnsafeFast
0012fab0 77e6204e kernel32!WaitForMultipleObjectsEx+0x172 , calling kernel32!_SEH_epilog
0012fadc 77e62068 kernel32!WaitForMultipleObjectsEx+0x34 , calling ntdll!RtlActivateActivationContextUnsafeFast
0012fae0 77e6207f kernel32!WaitForMultipleObjectsEx+0x184 , calling ntdll!RtlDeactivateActivationContextUnsafeFast
0012faf4 7c827b69 ntdll!NtWaitForMultipleObjects+0xc
0012fb34 7c829f6c ntdll!RtlAllocateHeap+0xee7 , calling ntdll!_SEH_epilog
0012fb38 603b14a4 mscoreei!UtilExecutionEngine::ClrHeapAlloc+0x14 , calling ntdll!RtlAllocateHeap
0012fb4c 603b14cb mscoreei!ClrHeapAlloc+0x23
0012fb58 603b4e73 mscoreei!InternalUnknownImpl<CLRRuntimeHostInternalImpl,IID_ICLRRuntimeHostInternal,ICLRRuntimeHostInternal,IID_ICLRRuntimeHostInternal,ICLRRuntimeHostInternal>::InternalQueryInterface+0x73 , calling mscoreei!_EH_epilog3
0012fb9c 603b4e73 mscoreei!InternalUnknownImpl<CLRRuntimeHostInternalImpl,IID_ICLRRuntimeHostInternal,ICLRRuntimeHostInternal,IID_ICLRRuntimeHostInternal,ICLRRuntimeHostInternal>::InternalQueryInterface+0x73 , calling mscoreei!_EH_epilog3
0012fba0 603b4e90 mscoreei!CLRRuntimeHostInternalImpl::QueryInterface+0x16 , calling mscoreei!InternalUnknownImpl<CLRRuntimeHostInternalImpl,IID_ICLRRuntimeHostInternal,ICLRRuntimeHostInternal,IID_ICLRRuntimeHostInternal,ICLRRuntimeHostInternal>::InternalQueryInterface
0012fbb0 603b4cf5 mscoreei!CLRRuntimeInfoImpl::GetInterfaceInternal+0x208
0012fbb8 603b1253 mscoreei!_EH_epilog3, calling mscoreei!__security_check_cookie
0012fbbc 603b4cfa mscoreei!CLRRuntimeInfoImpl::GetInterfaceInternal+0x20d , calling mscoreei!_EH_epilog3_GS
0012fbd4 776e2f66 ole32!CoWaitForMultipleHandles+0x150 , calling ole32!CCliModalLoop::~CCliModalLoop
0012fbfc 603b1656 mscoreei!BaseWrapper<PEDecoder *,FunctionBase<PEDecoder *,&DoNothing<PEDecoder *>,&Delete<PEDecoder>,2>,0,&CompareDefault<PEDecoder *>,2>::~BaseWrapper<PEDecoder *,FunctionBase<PEDecoder *,&DoNothing<PEDecoder *>,&Delete<PEDecoder>,2>,0,&CompareDefault<PEDecoder *>,2>+0x2c , calling mscoreei!_EH_epilog3
0012fc14 603b4cfa mscoreei!CLRRuntimeInfoImpl::GetInterfaceInternal+0x20d , calling mscoreei!_EH_epilog3_GS
0012fc18 603b4dce mscoreei!CLRRuntimeInfoImpl::GetInterface+0xc5 , calling mscoreei!CLRRuntimeInfoImpl::GetInterfaceInternal
0012fc24 603b4de0 mscoreei!CLRRuntimeInfoImpl::GetInterface+0xd7 , calling mscoreei!_EH_epilog3
0012fc60 603c4321 mscoreei!CLRRuntimeHostInternalImpl::ShutdownAllRuntimesThenExit+0x15 , calling mscoreei!RuntimeDesc::ShutdownAllActiveRuntimes
0012fc74 792f865e clr!EEPolicy::ExitProcessViaShim+0x66
0012fcac 792f870d clr!SafeExitProcess+0x122 , calling clr!EEPolicy::ExitProcessViaShim
0012fcd4 792dea45 clr!Thread::DoAppropriateWaitWorker+0x321 , calling clr!_EH_epilog3
0012fd50 792dea45 clr!Thread::DoAppropriateWaitWorker+0x321 , calling clr!_EH_epilog3
0012fd54 792deaa8 clr!Thread::DoAppropriateWait+0x60 , calling clr!Thread::DoAppropriateWaitWorker
0012fd68 792deac2 clr!Thread::DoAppropriateWait+0x7a , calling clr!_SEH_epilog4
0012fdbc 792deac2 clr!Thread::DoAppropriateWait+0x7a , calling clr!_SEH_epilog4
0012fdc0 792deb27 clr!CLREvent::WaitEx+0x106 , calling clr!Thread::DoAppropriateWait
0012fdd8 79161722 clr!CLREvent::WaitEx+0x12b , calling clr!_EH_epilog3
0012fe10 79161722 clr!CLREvent::WaitEx+0x12b , calling clr!_EH_epilog3
0012fe14 7916173e clr!CLREvent::Wait+0x19 , calling clr!CLREvent::WaitEx
0012fe30 79141af9 clr!_EH_epilog3_catch_GS+0xa , calling clr!__security_check_cookie
0012fe34 7921cd76 clr!CoEEShutDownCOM+0x3a5 , calling clr!_EH_epilog3_catch_GS
0012fe70 77e65125 kernel32!CreateThread+0x1e , calling kernel32!CreateRemoteThread
0012fe90 791418ff clr!ClrFlsGetValue+0xb , calling 00ed1eb0
0012fe98 79163b98 clr!ClrFlsClearThreadType+0xd , calling clr!ClrFlsGetValue
0012fe9c 79163ba7 clr!ClrFlsClearThreadType+0x1c , calling 00ed1eb0
0012feac 792caf15 clr!EEShutDown+0x113 , calling clr!_EH_epilog3
0012fef0 792cad59 clr!DisableRuntime+0x120 , calling clr!SafeExitProcess
0012ff08 792cadb5 clr!EEPolicy::HandleExitProcess+0x5c , calling clr!DisableRuntime+0xc2
0012ff18 792ca080 clr!_CorExeMainInternal+0xdd , calling clr!EEPolicy::HandleExitProcess
0012ff30 603b5011 mscoreei!BaseWrapper<ShimFactory *,FunctionBase<ShimFactory *,&DoNothing<ShimFactory *>,&DoTheRelease<ShimFactory>,2>,0,&CompareDefault<ShimFactory *>,2>::~BaseWrapper<ShimFactory *,FunctionBase<ShimFactory *,&DoNothing<ShimFactory *>,&DoTheRelease<ShimFactory>,2>,0,&CompareDefault<ShimFactory *>,2>+0x30 , calling mscoreei!_EH_epilog3
0012ff64 7927cb38 clr!_CorExeMain+0x4e , calling clr!_CorExeMainInternal
0012ff9c 603b55ab mscoreei!_CorExeMain+0x38
0012ffa8 79007f16 mscoree!ShellShim__CorExeMain+0x99
0012ffb8 79004de3 mscoree!_CorExeMain_Exported+0x8 , calling mscoree!ShellShim__CorExeMain
0012ffc0 77e6f1eb kernel32!BaseProcessStart+0x23
0:000> !analyze -v -h
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
ntdll!RtlpCoalesceFreeBlocks+128
7c82c912 8b09 mov ecx,dword ptr [ecx]
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000007 (Wake debugger)
ExceptionFlags: 00000000
NumberParameters: 0
PROCESS_NAME: MyProgram.exe
ERROR_CODE: (NTSTATUS) 0x80000007 - {Kernel Debugger Awakened} the system debugger was awakened by an interrupt.
EXCEPTION_CODE: (HRESULT) 0x80000007 (2147483655) - Operation aborted
MOD_LIST: <ANALYSIS/>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x2534 (0)
Current frame:
ChildEBP RetAddr Caller,Callee
READ_ADDRESS: 00000000
FOLLOWUP_IP:
ntdll!KiFastSystemCallRet+0
7c82845c c3 ret
BUGCHECK_STR: ACCESS_VIOLATION
LOADERLOCK_OWNER_API: _ExitProcess:LdrShutdownProcess:LdrpCallInitRoutine:
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 26bc.2534 Unknown
WAIT_CHAIN_COMMAND: ~0s;k;;
BLOCKING_THREAD: 00002534
DEFAULT_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_WRONG_SYMBOLS
LAST_CONTROL_TRANSFER: from 7c8275c9 to 7c82845c
FAULTING_THREAD: 00000000
STACK_TEXT:
0012f040 7c8275c9 77e828a8 d0000144 00000004 ntdll!KiFastSystemCallRet
0012f044 77e828a8 d0000144 00000004 00000000 ntdll!ZwRaiseHardError+0xc
0012f2b0 77e769c0 0012f2d8 77e61aa9 0012f2e0 kernel32!UnhandledExceptionFilter+0x51a
0012f2b8 77e61aa9 0012f2e0 00000000 0012f2e0 kernel32!BaseProcessStart+0x39
0012f2e0 7c8285c2 0012f3c4 0012ffe0 0012f3e0 kernel32!_except_handler3+0x61
0012f304 7c828593 0012f3c4 0012ffe0 0012f3e0 ntdll!ExecuteHandler2+0x26
0012f3ac 7c8283ce 00127000 0012f3e0 0012f3c4 ntdll!ExecuteHandler+0x24
0012f3ac 7c82c912 00127000 0012f3e0 0012f3c4 ntdll!KiUserExceptionDispatcher+0xe
0012f6b8 7c82a69b 00000000 003d5b68 0012f780 ntdll!RtlpCoalesceFreeBlocks+0x128
0012f7a0 77bbcef6 003d0000 00000000 003d5b70 ntdll!RtlFreeHeap+0x38e
0012f7e8 71b260af 003d5b70 00000000 0012f818 msvcrt!free+0xc3
0012f7f8 71b215dc 71b20000 00000000 00000001 mswsock!_CRT_INIT+0xa1
0012f818 7c81a19a 71b20000 00000001 00000001 mswsock!_DllMainCRTStartup+0x77
0012f838 7c830ce8 71b21528 71b20000 00000000 ntdll!LdrpCallInitRoutine+0x14
0012f8f0 77e6689b 00000001 001632e0 00000000 ntdll!LdrShutdownProcess+0x182
0012f9dc 77e668fd 00000000 77e8f3b0 ffffffff kernel32!_ExitProcess+0x43
0012f9f0 603c42f0 00000000 79811c96 00000000 kernel32!ExitProcess+0x14
0012fc60 603c4321 00000000 001632e0 00000000 mscoreei!RuntimeDesc::ShutdownAllActiveRuntimes+0x29c
0012fc74 792f865e 00192800 00000000 7983025e mscoreei!CLRRuntimeHostInternalImpl::ShutdownAllRuntimesThenExit+0x15
0012fcac 792f870d 00000000 79830002 00000000 clr!EEPolicy::ExitProcessViaShim+0x66
0012fef0 792cad59 00000000 00000000 00000000 clr!SafeExitProcess+0x122
0012ff08 792cadb5 00000000 01000000 0012ff64 clr!DisableRuntime+0x120
0012ff18 792ca080 00000000 79830196 00000000 clr!EEPolicy::HandleExitProcess+0x5c
0012ff64 7927cb38 7983016e 00000000 77e63d6a clr!_CorExeMainInternal+0xdd
0012ff9c 603b55ab 7927cb1c 0012ffb8 79007f16 clr!_CorExeMain+0x4e
0012ffa8 79007f16 00000000 603b0000 0012fff0 mscoreei!_CorExeMain+0x38
0012ffb8 79004de3 00000000 77e6f1eb 00000000 mscoree!ShellShim__CorExeMain+0x99
0012ffc0 77e6f1eb 00000000 00000000 7ffde000 mscoree!_CorExeMain_Exported+0x8
0012fff0 00000000 79004ddb 00000000 78746341 kernel32!BaseProcessStart+0x23
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: ntdll!KiFastSystemCallRet+0
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ntdll
IMAGE_NAME: ntdll.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4ecbcdd1
STACK_COMMAND: ~0s ; kb
BUCKET_ID: ACCESS_VIOLATION_ntdll!KiFastSystemCallRet+0
FAILURE_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS_80000007_ntdll.dll!KiFastSystemCallRet