[API Proposal]: PinnedBlockMemoryPool metrics

[BrennanConroy]

Background and Motivation

Add metrics to the PinnedBlockMemoryPool that's used by Kestrel, IIS, and Http.Sys.

Proposed API

Microsoft.AspNetCore.MemoryPool

Name	Instrument Type	Unit	Description
aspnetcore.memorypool.current_memory	UpDownCounter	{bytes}	Number of bytes that are currently pooled by the pool.

Name	Instrument Type	Unit	Description
aspnetcore.memorypool.total_allocated	Counter	{bytes}	Total number of allocations made by the pool.

Name	Instrument Type	Unit	Description
aspnetcore.memorypool.evicted_memory	Counter	{bytes}	Total number of bytes that have been evicted.

Name	Instrument Type	Unit	Description
aspnetcore.memorypool.total_rented	Counter	{bytes}	Total number of rented bytes from the pool.

[JamesNK]

There can be multiple instances of memory pools. They will all write to the same counters so metrics will represent all metrics. For example, Kestrel HTTP layer could have a memory pool, Kestrel sockets layer could have a memory pool, a library could have created their own memory pool (SignalR, gRPC, etc).

There should be a way to view metrics for individual memory pools. The way to do that would be to add tags. A memory pool is given a name and that name is added as a tag when recording the value.

public void IncrementTotalMemory(int bytes)
{
    _totalAllocatedMemory.Add(bytes, new KeyValuePair<string, object>("aspnetcore.memorypool.owner", _owner));
}

This should be fine because the number of memory pools and owners should be low carindality. Although it might be worth calling out in the .NET API to avoid have many names to avoid exceeding metrics dimensions limits.

Name could be specified when creating a memory pool:

public KestrelThing(IMemoryPoolFactory<T> memoryPoolFactory)
{
    _memoryPool = memoryPoolFactory.Create(new MemoryPoolOptions { Owner = "Kestrel" });
}

[BrennanConroy]

_memoryPool = memoryPoolFactory.Create(new MemoryPoolOptions { Owner = "Kestrel" });

That's an interesting idea. Adding options to the Create call. What potential future options would there be? David was wondering about options for eviction delay and various thresholds in #61554 (comment) and #61554 (comment) but those seem specific to our implementation. We'd likely want options to be more generic and apply to "all" pool implementations.