ObjectDisposedException intermittently crashes test process from DefaultHttpClientFactory.ExpiryTimer_Tick

[martincostello]

Description

The fix for #90386 does not appear to have resolved the issue but just changed the failure mode: #90386 (comment)

/cc @CarnaViire

Reproduction Steps

Clone martincostello/dependabot-helper@c8cbfc6 and run build.ps1 in the root of the repository.

Expected behavior

The tests pass.

Actual behavior

The test process crashes with an exception such as:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

Regression?

Yes.

Known Workarounds

None.

Configuration

.NET SDK 8.0.100-rc.2.23420.6 which contains .NET Runtime 8.0.0-rc.1.23414.4 @ e897946.

Other information

No response

Tagging subscribers to this area: @dotnet/area-extensions-dependencyinjection
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

The fix for #90386 does not appear to have resolved the issue but just changed the failure mode: #90386 (comment)

/cc @CarnaViire

Reproduction Steps

Clone martincostello/dependabot-helper@c8cbfc6 and run build.ps1 in the root of the repository.

Expected behavior

The tests pass.

Actual behavior

The test process crashes with an exception such as:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

Regression?

Yes.

Known Workarounds

None.

Configuration

.NET SDK 8.0.100-rc.2.23420.6 which contains .NET Runtime 8.0.0-rc.1.23414.4 @ e897946.

Other information

No response

Author:	martincostello
Assignees:	-
Labels:	untriaged, area-Extensions-DependencyInjection
Milestone:	-

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

The fix for #90386 does not appear to have resolved the issue but just changed the failure mode: #90386 (comment)

/cc @CarnaViire

Reproduction Steps

Clone martincostello/dependabot-helper@c8cbfc6 and run build.ps1 in the root of the repository.

Expected behavior

The tests pass.

Actual behavior

The test process crashes with an exception such as:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.<.ctor>b__14_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Http.DefaultHttpClientFactory.ExpiryTimer_Tick(Object state)
   at Microsoft.Extensions.Http.ActiveHandlerTrackingEntry.Timer_Tick()
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

Regression?

Yes.

Known Workarounds

None.

Configuration

.NET SDK 8.0.100-rc.2.23420.6 which contains .NET Runtime 8.0.0-rc.1.23414.4 @ e897946.

Other information

No response

Author:	martincostello
Assignees:	-
Labels:	untriaged, area-Extensions-HttpClientFactory
Milestone:	-

[CarnaViire]

.NET SDK 8.0.100-rc.2.23420.6 which contains .NET Runtime 8.0.0-rc.1.23414.4 @ e897946.

This build doesn't include my change yet 😅 you can check it here: https://github.com/dotnet/runtime/commits/2470610ea493b650177a5303684f87ee9fcec966

My commit was merged at 11:44 PM GMT+1, and the one your SDK is pointed at is from 5:34 PM GMT+1 😅

Can you please update to the latest daily build and try again?
E.g. .NET SDK 8.0.100-rc.2.23422.3 (.NET Runtime 8.0.0-rc.2.23418.14 @ 90b92bb) should already have the fix.

This issue has been marked needs-author-action and may be missing some important information.

[martincostello]

My bad, sorry about that 😅

I looked at the tree for e897946 and saw your name on the latest commit to HttpClientFactory, but I didn't pay enough attention to the commit message to see that it was the initial change that introduced the issue, rather than being the fix:

The fact the stack trace seemed to have changed made me think the fix had gone in too.

I've updated a bunch of repos to 8.0.100-rc.2.23422.31 just now and the problem does indeed appear to be resolved 🙇

martincostello/dependabot-helper#820