Introduction of issue symptom:
1. With ARR module in IIS global module list, w3wp.exe got 503 at first request - the w3wp.exe process crash consistently.
2. If remove ARR RequestRouter.dll from IIS global module list, then w3wp is up no issue.
3. The issue is only on servers in one domain - Could be domain GPOs caused the problem.
Time Travel Dump pinned the failed code path in the call stack below:
00 (Inline Function) --------`-------- rpcrt4!GENERIC_OBJECT::InvalidHandle
01 (Inline Function) --------`-------- rpcrt4!CheckHandleValidity+0x9
02 (Inline Function) --------`-------- rpcrt4!EnterBufApiFull+0x94
03 (Inline Function) --------`-------- rpcrt4!I_RpcNegotiateTransferSyntax+0x94
04 (Inline Function) --------`-------- rpcrt4!Ndr64ClientNegotiateTransferSyntax+0xf0
05 (Inline Function) --------`-------- rpcrt4!Ndr64pClientSetupTransferSyntax+0x43f
06 00000076`166fe6b0 00007ffe`f12d5a51 rpcrt4!NdrpClientCall3(void * pThis = 0x00000000`00000000, struct _MIDL_STUBLESS_PROXY_INFO * pProxyInfo = 0x00007ffe`f2ce32a8, unsigned long nProcNum = 8, void * pReturnValue = 0x00000076`166fe780, struct _NDR_PROC_CONTEXT * pContext = 0x00000076`166fea90, unsigned char * StartofStack = 0x00000076`166fee08 "")+0x574
07 00000076`166fea60 00007ffe`f2c96885 rpcrt4!NdrClientCall3(struct _MIDL_STUBLESS_PROXY_INFO * pProxyInfo = 0x00007ffe`f2ce32a8, unsigned long nProcNum = 8, void * pReturnValue = 0x00000000`00000000)+0xf1
08 (Inline Function) --------`-------- advapi32!ElfrRegisterEventSourceW+0x3a
09 00000076`166fedf0 00007ffe`f2c96776 advapi32!ElfRegisterEventSourceW(struct _UNICODE_STRING * UNCServerName = 0x00000076`166feee0 "", struct _UNICODE_STRING * ModuleName = 0x00000076`166feed0 "Application Request Routing", void ** LogHandle = 0x00000076`166fef10)+0xd5
0a 00000076`166feeb0 00007ffe`cdf182fc advapi32!RegisterEventSourceW(wchar_t * UNCServerName = 0x00000000`00000000 "", wchar_t * ModuleName = <Value unavailable error>)+0x36
0b 00000076`166fef00 00007ffe`cdf1f1d7 requestrouter!FORWARDING_HANDLER::StaticInitialize(void)+0x2bc
0c 00000076`166fef80 00007ffe`d4d18edf requestrouter!RegisterModule(unsigned long dwServerVersion = <Value unavailable error>, class IHttpModuleRegistrationInfo * pModuleInfo = <Value unavailable error>, class IHttpServer * pGlobalInfo = <Value unavailable error>)+0x4e7
0d (Inline Function) --------`-------- iiscore!MODULE_DLL::RegisterModule+0x19
0e (Inline Function) --------`-------- iiscore!VIRTUAL_MODULE::RegisterModule+0x2d
0f 00000076`166ff000 00007ffe`d4d188cf iiscore!W3_SERVER::LoadModulesFromConfig(class MULTISZ * pmszModuleConfig = 0x00000228`fdb734f8, int fActuallyLoadModules = 0n1)+0x393
10 00000076`166ff120 00007ffe`d4d18242 iiscore!W3_SERVER::InitializeGlobalModules(void)+0x3b
11 00000076`166ff160 00007ffe`d4d3119f iiscore!W3_SERVER::Initialize(class IWorkerProcessFramework * pWpFramework = <Value unavailable error>)+0xb36
12 00000076`166ff3d0 00007ffe`d4d3173c iiscore!IISCORE_PROTOCOL_MANAGER::InitializeGlobals(void)+0x24f
13 00000076`166ff670 00007ffe`deb1ab07 iiscore!IISCORE_PROTOCOL_MANAGER::StartListenerChannel(wchar_t * pProtocolId = <Value unavailable error>, class IWpfListenerChannelCallback * pListenerChannelCallback = 0x00000228`fdb82830)+0x5c
14 00000076`166ff6a0 00007ffe`deb1b9a2 w3wphost!LISTENER_CHANNEL::HandleStartListenerChannel(void)+0xdb
15 00000076`166ff6f0 00007ffe`deb146a6 w3wphost!LISTENER_CHANNEL_START_WORKITEM::ExecuteWorkItem(void)+0x12
16 00000076`166ff720 00007ffe`f3cdfb55 w3wphost!W3WP_HOST::ExecuteWorkItem(void * pvContext = 0x00000228`fd6b3410)+0x16
advapi32!RegisterEventSourceW then set LastError to "LastErrorValue: (Win32) 0x5 (5) - Access is denied.
LastStatusValue: (NTSTATUS) 0xc0000022 - {Access Denied} A process has requested access to an object, but has not been granted those access rights."
This is access denied error but a very unusual one – it does not show anything at file level or registry level access denied in ProcMon logs, and it does not give any clue when the access is denied in the time travel dump either.
The Code path above actually is trying to get a handle of the Application Event log, which is controlled by Application Event log ChannelAccess.
Application Event log ChannelAccess can be checked by Windows Command "wevtutil gl Application" in elevated Command Window.
The ChannelAccess right output will be something like string below:
O:BAG:SYD:(D;; 0xf0007 ;;;AN)(D;; 0xf0007 ;;;BG)(A;; 0xf0007 ;;;SY)(A;; 0x5 ;;;BA)(A;; 0x7 ;;;SO)(A;; 0x3 ;;;IU)(A;; 0x2 ;;;BA)(A;; 0x2 ;;;LS)(A;; 0x2 ;;;NS)
And the ChannelAccess can be written by registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\Application\customsd.
You can manually set this registry key or set it through GPOs.
A further detailed document you can reference to: https://learn.microsoft.com/en-us/troubleshoot/windows-server/group-policy/set-event-log-security-lo...
Windows out of box does not set the customsd registry key and the Application Event Log ChannelAccess has a default value. If you do set the customsd registry key, then the Application Event Log ChannelAccess will use the value in the customsd registry.
In the issue above, the customd is set via that domain GPO so all servers in that domain cannot grab a handle of Application Event Log in the ARR RequestRouter.dll registering global module code, thus it crashes the w3wp process when it hit the code path above.
When setting the customsd, you need to consider any App code will use the Event Log and check ChannelAcess and make sure certain rights are assigned to allow Apps work to access Event Log in code.