Intermitent error when using provided health checks

[ADringer]

Hi all,

I've added the health checks as documented here. My issue is with both health checks. Every couple of hours I get the error:

Object reference not set to an instance of an object. when calling the line var result = await handler.ProcessAsync(_httpContextAccessor.HttpContext);. I've added additional checks to make sure neither things are null:

var handler = _httpContextAccessor.HttpContext.RequestServices.GetRequiredService(endpoint.Handler) as IEndpointHandler;
if (handler != null)
{
    if(_httpContextAccessor.HttpContext == null)
        _logger.LogError("Discovery keys health check error: httpcontext is null");

    var result = await handler.ProcessAsync(_httpContextAccessor.HttpContext);
    if (result is JsonWebKeysResult)
    {
        return HealthCheckResult.Healthy();
    }

Any ideas what I should look at?

[wcabus]

Would you be able to share the stack trace when the NullReferenceException occurs? My immediate guess is that the exception is being thrown somewhere within the endpoint handler. When the health check is being performed by a background service in the same host, there's a high chance that some of HttpContext's properties are not or partially initialized by the caller of the health check.

Revisiting our documentation page there, I'm planning on doing a rewrite of the health check content. Different types of health checks can be implemented based on your deployment scenario, but perhaps calling an HTTP endpoint from inside the hosted service itself might not be the best idea. Within IdentityServer, you may rather want to check that the databases are ready, for example using EF Core's health check functionality:

services.AddHealthChecks()
    .AddDbContextCheck<ConfigurationDbContext>(tags: ["ready"])
    .AddDbContextCheck<PersistedGrantDbContext>(tags: ["ready"]);

app.MapHealthChecks("/health/live", new() { Predicate = _ => false });
app.MapHealthChecks("/health/ready", new() { Predicate = r => r.Tags.Contains("ready") });

[ADringer]

Yeah I think it's an intermittent issue that the context isn't always completly initialised. I don't have a full stack trace, but do have:

Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object. 
at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Method() 
at Duende.IdentityServer.Endpoints.DiscoveryKeyEndpoint.ProcessAsync(HttpContext context) in /_/identity-server/src/IdentityServer/Endpoints/DiscoveryKeyEndpoint.cs:line 40 
at UserManagementService.Api.HealthChecks.DiscoveryKeysHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /src/src/UserManagementService.Api/HealthChecks/DiscoveryKeysHealthCheck.cs:line 24 

which points towards an issue getting the HTTP method from the request message?

I also saw something similar when I was using Serilog:

An unhandled exception was thrown by the application. System.NullReferenceException: Object reference not set to an instance of an object. 
at Serilog.AspNetCore.RequestLoggingMiddleware.GetPath(HttpContext httpContext, Boolean includeQueryInRequestPath) 
at Serilog.AspNetCore.RequestLoggingMiddleware.Invoke(HttpContext httpContext) 
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) 

That had an issue getting the path, so think they are related?

I'll keep an eye out for the rewrite of the docs!