feat(histograms): static config for controlling timer/histogram emission

[Groxx]

Now that the basics are working: time to allow turning off timers.

Expected uses are:

Wait for us to finish, then get everything all at once by default

The default config is currently set to "timers", which will not emit the new histograms.

This is primarily to:

• prevent you from being surprised by a large jump in your metrics cardinality
• prevent us from breaking your already-migrated alerts/dashboards/whatever due to us changing our mind on how to emit that data
  • for the most part though we only expect this for very-new histograms

When we're done creating all the new histograms and are happy with them, we'll decide on a rollout strategy and will change the default behavior to match that.
(this may just be "dual-emit everything all at once", or something more gradual)

Gradually migrate once things seem ready

Some time after we introduce a new histogram version of a timer metric, you can opt into it with config:

histograms:
  names:
    the_new_histogram_ns: true
    oops_wrong_name: true  # this will error, so you can't accidentally mis-spell things

Then you can change your alerts/dashboards/whatever to use the new data, and optionally disable the timer to stop emitting it:

histograms:
  names:
    the_old_timer: false
    the_new_histogram_ns: true

This may also be useful for handling the final "we're changing the default!" migration more gradually, as you can change the default and opt in piece by piece:

histograms:
  default: timers
  names:
    the_old_timer: false        # stop emitting just this timer
    the_new_histogram_ns: true  # opt into just this histogram

Join us on the bumpy road

Internally, to migrate quicker and to get a feel for if our changes are correct (and change things if it seems necessary), we're going to be dual-emitting ASAP and removing ASAP:

histograms:
  default: both
  names:
    the_old_timer: false  # already migrated this one, so turn it off

We will probably also change the sample development yamls to use this, to serve as a sample and an early warning for people who use them.

---

If you have custom timers to migrate in wrapping code

No problem! Just add to the static list of metrics being migrated, and you can use this config too:

package main

import "github.com/uber/cadence/common/metrics"

func init() {
  metrics.HistogramMigrationMetrics["your_custom_timer"] = struct{}{}
  metrics.HistogramMigrationMetrics["your_custom_histogram"] = struct{}{}
}
func main() {
  // ... service startup stuff
}

histograms:
  names:
    your_custom_timer: false     # this will error unless you
    your_custom_histogram: true  # change the metrics map keys

We will eventually remove timer APIs from our metrics.Scope though, so you should definitely tackle this migration "soon" after we switch the default mode (or earlier).

---

I'd originally considered doing this in dynamic config, but I doubt we'll need to make quick changes, and static config is significantly less of a pain, both internally for Reasons™ and in code due to the key-ID verbosity.

Unrelatedly: this PR is a near-ideal example of why dependency injection frameworks are nice 😢

[Groxx]

config.Config is provided by config.New during service startup, already exists

[Groxx]

surprisingly, as tests show, empty values don't call this unmarshal func at all. so they don't "need" to be handled.

that seems rather Bad™, but I don't think it'll change while we're on yaml/v2. and tests should catch if/when it changes in later major versions / different libraries.

[Groxx]

an explicitly-empty value seems a bit clearer about intent than NewTestStopwatch() or NewStopwatch(), though they're essentially identical (just with time.Now() instead of a zero-valued Time). happy to change if anyone disagrees.

[davidporter-id-au]

Do you mind updating the diff description to reference something about the overall intent of the project? I added https://github.com/orgs/cadence-workflow/projects/1?pane=issue&itemId=132424271 as a draft, but maybe ther's something better already?

[davidporter-id-au]

nit: I think this is out of x now

[Groxx]

ah, it's in 1.23. I thought it was 1.24 for some reason (and we're still in a bit of a struggle to upgrade to 1.24)

I may make a minor followup to replace these, in that case.