Add support for pyexasol part 2/2 (#320)

* Add hooks for websocket based exasol dialect
* Add support for enabling and disabling certificate validation
* Add hooks for type conversions between pyexasol and dbapi2 cursor
* Add websocket dialect/connector to ci/cd matrix
* Update dependencies
* Update README.md
* Rework dbapi2 types
* Restructure the dbapi2 module of the websocket driver
* Disable remaining failing SQLA compliance tests
   - #341
   - #342

---
Co-authored-by: Christoph Pirkl <[email protected]>
Co-authored-by: Sebastian Bär <[email protected]>
Co-authored-by: Torsten Kilias <[email protected]>

Author: 4 people

.github/workflows/ci-cd.yml

@@ -64,6 +64,7 @@ jobs:
         connector:
           - pyodbc
           - turbodbc
+          - websocket
         exasol_version:
           - 7.1.17
           - 7.0.20

.github/workflows/ci.yaml

@@ -68,6 +68,7 @@ jobs:
         connector:
           - pyodbc
           - turbodbc
+          - websocket
         exasol_version:
           - 7.1.17
           - 7.0.20

CHANGELOG.rst

@@ -3,6 +3,17 @@
 Unreleased
 ==========
 
+* Add Beta version of websocket based dialect
+
+     **🚨 Attention:**
+
+        This feature is currently in Beta, therefore it should not be used in production.
+        We also recommend to have a look into the known issues, before you start using it.
+
+        If you encounter any problem, please `create an issue <https://github.com/exasol/sqlalchemy-exasol/issues/new?assignees=&labels=bug&projects=&template=bug.md&title=%F0%9F%90%9E+%3CInsert+Title%3E>`_.
+        With your feedback, we will be able stabilize this feature more quickly.
+
+
 .. _changelog-4.4.0:
 
 4.4.0 — 2023-05-16

README.rst

@@ -26,7 +26,7 @@ SQLAlchemy Dialect for EXASOL DB
     :target: https://pycqa.github.io/isort/
     :alt: Formatter - Isort
 
-.. image:: https://img.shields.io/badge/pylint-5.8-yellow
+.. image:: https://img.shields.io/badge/pylint-6.4-yellowgreen
     :target: https://github.com/PyCQA/pylint
     :alt: Pylint
 
@@ -49,17 +49,6 @@ How to get started
 We assume you have a good understanding of (unix)ODBC. If not, make sure you
 read their documentation carefully - there are lot's of traps 🪤 to step into.
 
-Meet the system requirements
-````````````````````````````
-
-On Linux/Unix like systems you need:
-
-- Python
-- An Exasol DB (e.g. `docker-db <test_docker_image_>`_ or a `cloud instance <test_drive_>`_)
-- The packages unixODBC and unixODBC-dev >= 2.2.14
-- The Exasol `ODBC driver <odbc_driver_>`_
-- The ODBC.ini and ODBCINST.ini configurations files setup
-
 Turbodbc support
 ````````````````
 
@@ -72,6 +61,18 @@ Turbodbc support
   `test/test_update.py <test/test_update.py>`_ for an example
 
 
+
+Meet the system requirements
+````````````````````````````
+
+On Linux/Unix like systems you need:
+
+- Python
+- An Exasol DB (e.g. `docker-db <test_docker_image_>`_ or a `cloud instance <test_drive_>`_)
+- The packages unixODBC and unixODBC-dev >= 2.2.14
+- The Exasol `ODBC driver <odbc_driver_>`_
+- The ODBC.ini and ODBCINST.ini configurations files setup
+
 Setup your python project and install sqlalchemy-exasol
 ```````````````````````````````````````````````````````
 
@@ -109,18 +110,17 @@ The dialect supports two types of connection urls creating an engine. A DSN (Dat
 
 .. list-table::
 
-   * - Type
-     - Example
-   * - DSN URL
-     - 'exa+pyodbc://USER:PWD@exa_test'
-   * - HOST URL
-     - 'exa+pyodbc://USER:[email protected]:1234/my_schema?parameter'
+    * - Type
+      - Example
+    * - DSN URL
+      - 'exa+pyodbc://USER:PWD@exa_test'
+    * - HOST URL
+      - 'exa+pyodbc://USER:[email protected]:1234/my_schema?parameter'
 
 Features
 ++++++++
 
 - SELECT, INSERT, UPDATE, DELETE statements
-- you can even use the MERGE statement (see unit tests for examples)
 
 Notes
 +++++
@@ -139,3 +139,95 @@ Development & Testing
 `````````````````````
 See `developer guide`_
 
+Websocket support
+-----------------
+
+.. attention::
+
+    The Websocket support is currently in Beta, therefore it should not be used in production.
+    We also recommend to have a look into the known issues, before you start using it.
+
+    If you encounter any issue, please `create an issue <https://github.com/exasol/sqlalchemy-exasol/issues/new?assignees=&labels=bug&projects=&template=bug.md&title=%F0%9F%90%9E+%3CInsert+Title%3E>`_.
+    With your feedback, we will be able to stabilize this feature more quickly.
+
+What is Websocket support?
+``````````````````````````
+In the context of SQLA and Exasol, Websocket support means that an SQLA dialect
+supporting the `Exasol Websocket Protocol <https://github.com/exasol/websocket-api>`_
+is provided.
+
+Using the websocket based protocol instead over ODBC will provide various advantages:
+
+* Less System Dependencies
+* Easier to use than ODBC based driver(s)
+* Lock free metadata calls etc.
+
+For further details `Why a Websockets API  <https://github.com/exasol/websocket-api#why-a-websockets-api>`_.
+
+Examples Usage(s)
+`````````````````
+
+.. code-block:: python
+
+    from sqla import create_engine
+
+    engine = create_engine("exa+websocket://sys:[email protected]:8888")
+    with engine.connect() as con:
+        ...
+
+.. code-block:: python
+
+    from sqla import create_engine
+
+    # ATTENTION:
+    # In terms of security it is NEVER a good idea to turn of certificate validation!!
+    # In rare cases it may be handy for non-security related reasons.
+    # That said, if you are not a 100% sure about your scenario, stick with the
+    # secure defaults.
+    # In most cases, having a valid certificate and/or configuring the truststore(s)
+    # appropriately is the best/correct solution.
+    engine = create_engine("exa+websocket://sys:[email protected]:8888?SSLCertificate=SSL_VERIFY_NONE")
+    with engine.connect() as con:
+        ...
+
+Supported Connection Parameters
+```````````````````````````````
+.. list-table::
+
+   * - Parameter
+     - Values
+     - Comment
+   * - ENCRYPTION
+     - Y, Yes, N, No
+     - Y or Yes Enable Encryption (TLS) default, N or No disable Encryption
+   * - SSLCertificate
+     - SSL_VERIFY_NONE
+     - Disable certificate validation
+
+
+Known Issues
+````````````
+
+* Literal casts within prepared statements do not work
+    - :code:`INSERT INTO t (x) VALUES (CAST(? AS VARCHAR(50)));`
+* Various conversions regarding float, decimals
+    - Certain scenarios still yield a :code:`string` type instead :code:`float` or :code:`decimal` type.
+* Insert
+    - Insert multiple rows via prepared statements does not work in all cases
+    - Insert from SELECT does not work
+* For some prepared statements, the wss protocol type conversion does not work properly
+    - Error messages usually state some JSON type mismatch, e.g.: '... getString: JSON value is not a string ...'
+* Known failing tests of the SQLA compliance test suite
+    - FAILED test/integration/sqlalchemy/test_suite.py::CastTypeDecoratorTest_exasol+exasol_driver_websocket_dbapi2::test_special_type - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
+    - FAILED test/integration/sqlalchemy/test_suite.py::ExistsTest_exasol+exasol_driver_websocket_dbapi2::test_select_exists - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
+    - FAILED test/integration/sqlalchemy/test_suite.py::ExistsTest_exasol+exasol_driver_websocket_dbapi2::test_select_exists_false - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
+    - FAILED test/integration/sqlalchemy/test_suite.py::InsertBehaviorTest_exasol+exasol_driver_websocket_dbapi2::test_empty_insert_multiple - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
+    - ERROR  test/integration/sqlalchemy/test_suite.py::InsertBehaviorTest_exasol+exasol_driver_websocket_dbapi2::test_empty_insert_multiple_teardown - ERROR
+    - FAILED test/integration/sqlalchemy/test_suite.py::InsertBehaviorTest_exasol+exasol_driver_websocket_dbapi2::test_insert_from_select - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
+    - FAILED test/integration/sqlalchemy/test_suite.py::InsertBehaviorTest_exasol+exasol_driver_websocket_dbapi2::test_insert_from_select_with_defaults - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
+    - FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_float_as_decimal - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
+    - FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_float_as_float - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
+    - FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_float_coerce_round_trip - AssertionError: '15.7563' != 15.7563
+    - FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_float_custom_scale - sqlalchemy.exc.DBAPIError: (exasol.driver.websocket._errors.Error)
+    - FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_numeric_as_float - AssertionError: {'15.7563'} != {15.7563}
+    - FAILED test/integration/sqlalchemy/test_suite.py::NumericTest_exasol+exasol_driver_websocket_dbapi2::test_render_literal_numeric_asfloat - AssertionError: assert '15.7563' in [15.7563]