Possible race condition in getting the database credentials

[tonyandrewmeyer]

Feedback from an onboarder working through the tutorial - they get an error on config-changed/pebble-ready:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/./src/charm.py", line 192, in <module>
    ops.main(FastAPIDemoCharm)
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/venv/ops/main.py", line 563, in __call__
    return main(charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/venv/ops/main.py", line 551, in main
    manager.run()
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/venv/ops/main.py", line 530, in run
    self._emit()
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/venv/ops/main.py", line 516, in _emit
    self.framework.reemit()
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/venv/ops/framework.py", line 870, in reemit
    self._reemit()
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/./src/charm.py", line 65, in _on_config_changed
    self._update_layer_and_restart(None)
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/./src/charm.py", line 84, in _update_layer_and_restart
    new_layer = self._pebble_layer.to_dict()
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/./src/charm.py", line 167, in _pebble_layer
    "environment": self.app_environment,
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/./src/charm.py", line 109, in app_environment
    db_data = self.fetch_postgres_relation_data()
  File "/var/lib/juju/agents/unit-demo-api-charm-0/charm/./src/charm.py", line 141, in fetch_postgres_relation_data
    "db_username": data["username"],
KeyError: 'username'

My best guess here is that there's a point in time where the lib's get_relation_data will be non-empty, but the secrets won't have been created yet (or are otherwise somehow not returned in the relation data). Digging into the lib code should be able to confirm that.

If that's the case, then the guard that currently returns {} if there's no relation data should also handle the case where there's relation data, but no credentials (and use an empty environment). My assumption (again, would be good to confirm this) is that the custom DatabaseCreated event will only be triggered after the credentials are available, so eventually the holistic handler would provide the full environment.