feat: Skip writeback for chunks fetched by queriers older than a duration

[mveitas]

What this PR does / why we need it:
This change will prevent any chunks fetched by the querier to not be written back to the chunk cache that are before a given time range.

95% of the searches that our engineering teams execute are within a 7 day range and we have a cache enabled to support 14 days worth of chunks to aid in query performance. We recently found that we have compliance need to maintain searchable logs for 365 days. While these queries outside of the 7 day window are rare, they tend to be longer duration searches of up to 30 days for compliance, contain a lot of data, and tend to be one-off searches.

Currently these chunks fetched by queriers outside our target window are written back to the chunk cache and causing more recent data to be evicted from the cache and impacting the query performance for the most recent data.

Which issue(s) this PR fixes:
Fixes #14983

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[mveitas]

@ashwanthgoli This is the new PR that I created that replaces #15393 - it was easier to fix by creating a new branch

[ashwanthgoli]

lgtm

[ashwanthgoli]

@mveitas can you run make docs? Thanks again for the contribution!