Cluster server client - TLS certificate is read once per-transaction.

[thomaschristopherking]

Description

When opening multiple transactions, in parallel, a UNIX system can raise an 'OSError: [Errno 24] Too many open files' error. This is because the TLS certificate file is opened once per-transaction, instead of once per-client.

Additional information

I am foregoing the reproducible steps, and instead going to provide some code changes I think are sufficient.

In typedb.connection.cluster we have the following method:

def _new_channel(self) -> grpc.Channel:
    if self._credential.tls_root_ca_path() is not None:
        with open(self._credential.tls_root_ca_path(), 'rb') as root_ca:
            channel_credentials = grpc.ssl_channel_credentials(root_ca.read())
    else:
        channel_credentials = grpc.ssl_channel_credentials()
    combined_credentials = grpc.composite_channel_credentials(
        channel_credentials,
        grpc.metadata_call_credentials(_CredentialAuth(credential=self._credential, token_fn=lambda: self._stub.token()))
    )
    return grpc.secure_channel(self._address, combined_credentials)

The line with open(self._credential.tls_root_ca_path(), 'rb') as root_ca is causing the issue.

I propose instead, making this the init:

def __init__(self, address: str, credential: TypeDBCredential, parallelisation: int = 2):
        super(_ClusterServerClient, self).__init__(address, parallelisation)
        self._credential = credential
        self._channel, self._stub = self.new_channel_and_stub()
        self._databases = _TypeDBDatabaseManagerImpl(self.stub())

        if self._credential.tls_root_ca_path() is not None:
            with open(self._credential.tls_root_ca_path(), 'rb') as root_ca:
                self._root_ca = root_ca.read()

And replacing the _new_channel method with this:

def _new_channel(self) -> grpc.Channel:
        if self._root_ca is not None:
            with open(self._credential.tls_root_ca_path(), 'rb') as root_ca:
                channel_credentials = grpc.ssl_channel_credentials(root_ca.read())
        else:
            channel_credentials = grpc.ssl_channel_credentials()
        combined_credentials = grpc.composite_channel_credentials(
            channel_credentials,
            grpc.metadata_call_credentials(_CredentialAuth(credential=self._credential, token_fn=lambda: self._stub.token()))
        )
        return grpc.secure_channel(self._address, combined_credentials)