Objects still apear in listing even if not authorized

[IamSAB]

I am using an adapted flex type. A user is only allowed a action if he is owner of the object or manager of the flex type and has the proper permission.

isAuthorizedOverridemethod of the GenericFlexObject class:

  protected function isAuthorizedOverride(UserInterface $user, string $action, string $scope, bool $isMe): ?bool
  {
    // Check if the action has been denied in the flex type configuration.
    $directory = $this instanceof FlexDirectory ? $this : $this->getFlexDirectory();
    $config = $directory->getConfig();
    $allowed = $config->get("{$scope}.actions.{$action}") ?? $config->get("actions.{$action}") ?? true;
    if (false === $allowed) {
      return false;
    }

    // TODO: Not needed anymore with flex users, remove in 2.0.
    $auth = $user instanceof FlexObjectInterface ? null : $user->authorize('admin.super');
    if (true === $auth) {
      return true;
    }

    // check if user is authorized for action
    $isAuthorized = $user->authorize($this->getAuthorizeRule($scope, $action), !$isMe ? 'test' : null);

    // check if user is owner
    $owners = (array) $this->getNestedProperty('x-accounts', []);
    $isOwner = in_array($user->username, $owners);

    // check if user can mange flex type
    $isManager = $user->authorize($this->getAuthorizeRule($scope, 'manage'), !$isMe ? 'test' : null);

    return $isAuthorized && ($isOwner || $isManager);
  }

It works so far, but objects are still listed, even if the user is not their owner or manager. They are listed, but without the ability to edit or delete them. From debugging I know, these objects have no read or list access. Is the listing controller via the directory?
How can I remove/hide these objects from the list?