oauth relation-departed hook fails when trying to remove juju secret

[natalian98]

Bug Description

When the charm related to hydra via oauth relation is removed, hydra gets into ErrorState. Its relation-departed hook fails when it tries to remove a juju secret it no longer has access to (or doesn't exist anymore).

To Reproduce

1. Deploy postgresql-k8s and hydra, relate them
2. Deploy grafana (currently a locally built charm with oauth relation added)
3. juju relate hydra:oauth grafana-k8s:oauth
4. Wait until the charms get active
5. juju remove grafana-k8s

Environment

microk8s 1.26-strict/stable
juju 3.1/stable (3.1.2)

Relevant log output

unit-hydra-0: 09:35:53 INFO juju.worker.uniter.operation ran "pg-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-hydra-0: 09:39:23 ERROR unit.hydra/0.juju-log oauth:7: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2616, in _run
    result = run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-hydra-0/secret-get', '--label', 'client_secret_7', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2967, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2618, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: ERROR consumer label "client_secret_7" not found


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 285, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2970, in secret_get
    raise SecretNotFoundError() from e
ops.model.SecretNotFoundError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2616, in _run
    result = run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-hydra-0/secret-info-get', '--label', 'client_secret_7', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2977, in _run_for_secret
    return self._run(*args, return_output=return_output, use_json=use_json)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2618, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: ERROR secret "client_secret_7" not found


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-hydra-0/charm/./src/charm.py", line 757, in <module>
    main(HydraCharm)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/framework.py", line 354, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/framework.py", line 830, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/framework.py", line 919, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-hydra-0/charm/lib/charms/hydra/v0/oauth.py", line 615, in _on_relation_departed
    self._delete_juju_secret(event.relation)
  File "/var/lib/juju/agents/unit-hydra-0/charm/lib/charms/hydra/v0/oauth.py", line 626, in _delete_juju_secret
    secret = self.model.get_secret(label=self._get_secret_label(relation))
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 290, in get_secret
    info = self._backend.secret_info_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2991, in secret_info_get
    result = self._run_for_secret('secret-info-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2980, in _run_for_secret
    raise SecretNotFoundError() from e
ops.model.SecretNotFoundError
unit-hydra-0: 09:39:24 ERROR juju.worker.uniter.operation hook "oauth-relation-departed" (via hook dispatching script: dispatch) failed: exit status 1
unit-hydra-0: 09:39:24 INFO juju.worker.uniter awaiting error resolution for "relation-departed" hook
unit-hydra-0: 09:39:29 INFO juju.worker.uniter awaiting error resolution for "relation-departed" hook
unit-hydra-0: 09:39:32 ERROR unit.hydra/0.juju-log oauth:7: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2616, in _run
    result = run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-hydra-0/secret-get', '--label', 'client_secret_7', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2967, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-hydra-0/charm/venv/ops/model.py", line 2618, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: ERROR consumer label "client_secret_7" not found

Additional context

No response