Create Apache TinkerPop provider (apache#47446)

* tests passed

* narrow down CI

* fix host

* add drill for testing

* add docker inspect

* create a funtion

* minor changes

* change the inspect

* add status condition

* change host to zeros

* rewrite the pipeline

* push image and reuse

* remove tag

* rename tag

* change to locally pushed image

* add sha to image name

* add sha to image name

* add debug and login

* change token name

* change token name

* remove login

* remove login

* remove login

* move token to hight level

* copy from ci

* change to secrets

* test workflow

* test workflow

* rerun on CI login

* rerun on CI login

* logout

* revert logout

* revert login

* remove cleanup

* readd cleanup

* change docker pass

* add login to action

* move up

* chage token

* one job

* one job

* remove action

* delete if condition

* rename image

* fix tar name

* fix tar name

* fix tar name

* remove status

* remove stdin

* add logs and hostname

* removed restart

* add chmod

* create entry sh

* create entry sh

* add status

* remove failure

* add user

* revert back all ci yamls

* revert back shell script

* fix tinkerpop

* change back to gremlin in the providers list

* change docs

* change docs

* add conn_name_attr

* move system test

* change to cap

* change to 1.0.0

* fix sh

* removed serializer and some changes

* fixing docs

* update provider

* change to 2.9.0 and add spell check

* ran breeze release management

* add doc strings

* add asterisk

* moved operators doc

* fixed docs

* fixed pyproject

* minor change to docs

* add conf.py

* fixed docs

* fix toml

* changed to 2.10

* remove fab from tinkerpop

* fix prov info

* add close method

* fix integration

* change gremlin host

* change gremlin host back

* remove async

* remove package

* fix pyproject

* fix test

---------

Co-authored-by: Jarek Potiuk <[email protected]>

Author: ahmadtfarhan

.github/ISSUE_TEMPLATE/airflow_providers_bug_report.yml

@@ -41,6 +41,7 @@ body:
         - apache-pig
         - apache-pinot
         - apache-spark
+        - apache-tinkerpop
         - apprise
         - arangodb
         - asana

.github/boring-cyborg.yml

@@ -72,6 +72,9 @@ labelPRBasedOnFilePath:
   provider:apache-spark:
     - providers/apache/spark/**
 
+  provider:apache-tinkerpop:
+    - providers/apache/tinkerpop/**
+
   provider:apprise:
     - providers/apprise/**
 

.pre-commit-config.yaml

@@ -246,6 +246,9 @@ repos:
         name: Check that executables have shebang
       - id: check-xml
         name: Check XML files with xmllint
+        exclude: >
+          (?x)
+          ^scripts/ci/docker-compose/gremlin/.
       - id: trailing-whitespace
         name: Remove trailing whitespace at end of line
         exclude: ^airflow-core/docs/img/.*\.dot$|^dev/breeze/doc/images/output.*$
@@ -970,6 +973,7 @@ repos:
         exclude: >
           (?x)
           ^scripts/ci/docker-compose/grafana/.|
+          ^scripts/ci/docker-compose/gremlin/.|
           ^scripts/ci/docker-compose/.+-config\.ya?ml$
         require_serial: true
         additional_dependencies: ['jsonschema>=3.2.0,<5.0', 'pyyaml>=6.0.2', 'requests==2.32.3', 'rich>=12.4.4']

airflow-core/docs/extra-packages-ref.rst

@@ -198,6 +198,8 @@ custom bash/python providers).
 +---------------------+-----------------------------------------------------+------------------------------------------------+
 | apache-spark        | ``pip install 'apache-airflow[apache-spark]'``      | All Spark related operators & hooks            |
 +---------------------+-----------------------------------------------------+------------------------------------------------+
+| apache-tinkerpop    | ``pip install apache-airflow[apache-tinkerpop]``    | Apache-tinkerpop hooks and operators           |
++---------------------+-----------------------------------------------------+------------------------------------------------+
 | apache-webhdfs      | ``pip install 'apache-airflow[apache-webhdfs]'``    | HDFS hooks and operators                       |
 +---------------------+-----------------------------------------------------+------------------------------------------------+
 

airflow-core/src/airflow/utils/db.py

@@ -352,6 +352,15 @@ def create_default_connections(session: Session = NEW_SESSION):
         ),
         session,
     )
+    merge_conn(
+        Connection(
+            conn_id="gremlin_default",
+            conn_type="gremlin",
+            host="gremlin",
+            port=8182,
+        ),
+        session,
+    )
     merge_conn(
         Connection(
             conn_id="hive_cli_default",

contributing-docs/testing/integration_tests.rst

@@ -64,6 +64,8 @@ core or provider type of test.
 +--------------+-------------------------------------------------------+
 | drill        | Integration required for drill operator and hook.     |
 +--------------+-------------------------------------------------------+
+| gremlin      | Integration required for gremlin operator and hook.   |
++--------------+-------------------------------------------------------+
 | kafka        | Integration required for Kafka hooks.                 |
 +--------------+-------------------------------------------------------+
 | kerberos     | Integration that provides Kerberos authentication.    |