Cache metadata more

[twmb]

grafana/mimir#8886 (comment)

@pracucci to reply to your latest message in the thread -- if we strengthen the caching within franz-go itself, then it addresses caching the metadata request you mention,

"""
The pt1 of the PR description refers to the Metadata request issued to discover the partitions:

franz-go/pkg/kadm/metadata.go

Lines 432 to 436 in 6b61d17

	func (cl *Client) listOffsets(ctx context.Context, isolation int8, timestamp int64, topics []string) (ListedOffsets, error) {
	tds, err := cl.ListTopics(ctx, topics...)
	if err != nil {
	return nil, err
	}

"""

My proposal covers caching that^^, at which point the only extra caching your PR would provide is the 5 extra seconds (your PR uses 15s caching, but also elsewhere in Mimir you use 10s MetadataMinAge)

[pracucci]

From the original comment, you proposed 3 options:

• Always use the mapped metadata cache for user issued Metadata requests
• Introduce a new API, RequestCachedMetadata(req *kmsg.MetadataRequest, expiry time.Duration) (*kmsg.MetadataResponse, error). If the metadata exists and is cached for less than expiry, return it. If not, issue the metadata request (and force the response through the cache)
• Introduce a new API, UseCache(context.Context) context.Context that can be used to query any internal cache when manually issuing a metadata request

My vote is the first or second bullet point. I think making these changes would avoid the need for this PR, but I'm not positive (not looking too closely).

As a user I would be prefer to be in control whether to use the cache and not. I guess option 1 and 2 wouldn't allow to have such control for an high-level request like ListOffsets() but, on the other side, I have no idea about option 3 complexity (which is also your least preferred one).

[twmb]

The more I look at this, the more I think this needs to be done via (2) or (3), not (1). Saving for the next next release.