The metastore offers replicas > 1 but no HA ist configured

[maltesander]

Currently, the Hive metastore can be configured with multiple replicas > 1 per role group. This results in the feeling for high availability. The service discovery to offers multiple FQDN's to different (and completely unrelated) metastores (#145).

We should consolidate the behavior and have two options:

1. Fix replicas to 1 and offer a warning if set > 1. Add in the documentation that metastores from different rolegroups do not offer HA or interact with each other
2. Implement HA for the metastore (e.g. using ZooKeeperTokenStore)?

TODO: They all share the same database, Metastore should still be stateless, ZK discovery does exist but not sure what uses that. This is for Jim to research

[sbernauer]

I've set up metastores in production on k8s in the past.
We've used 3 instances with 3 stable service names (meta-primary, meta-secondary and meta-tertiary or so). All of them shared the same postgres and all our metastore clients (e.g. Hive servers and Trinos) got the list of the 3 clients. I also enabled hive.metastore.uri.selection=RANDOM to evenly balance the load. (We used a beefy Postgres)
As a side note i also deployed 3 hive servers and enabled thrift over HTTP (instead of plain tcp) and was able to simply put a http loadbalancer-service in front - no need for Zookeeper at all. The hive servers did only serve create external table and alter table commands, no actual queries.
As the last thing i put a PodDisruptionBudget of max 1 unavailable for all the 6 Pods and antiaffinity between instances and affinity between HMS and hive.
All resulted in really smooth operation without any problems :)

[lfrancke]

We need to make sure to not allow more than 1 replica when using Derby.

We need to check if the discovery config map already exposes all instances in the correct format.