GCP Dataproc suddently not accepting current config default values

[sudohainguyen]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

apache-airflow-providers-google          8.6.0

Apache Airflow version

2.3.2

Operating System

ubuntu 20.04

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

The incident occurred 2 days ago, all of DataprocCreateCluster thrown this error

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Network 'default' requires explicit subnetworks for instance creation. Specify a 'legacy' or 'auto' network, or specify subnetwork explicitly."
	debug_error_string = "UNKNOWN:Error received from peer ipv4:172.217.194.95:443 {grpc_message:"Network \'default\' requires explicit subnetworks for instance creation. Specify a \'legacy\' or \'auto\' network, or specify subnetwork explicitly.", grpc_status:3, created_time:"2022-12-30T09:48:37.000807792+00:00"}"

What you think should happen instead

After a few tries with gcloud and pure python client, I found out it was not because of code implementation, maybe on server side, GCP had a few changes.

How to reproduce

1. Create ClusterGenerator object with a few required params, left the rest as default
2. Create a DAG containing DataprocCreateClusterOperator with the config
3. Execute the DAG

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[sudohainguyen]

my team resolved this issue by assigning specific value to subnetwork_uri attributes like this

cluster = {
        "project_id": project_id,
        "cluster_name": cluster_name,
        "config": {
...
            "gce_cluster_config": {
                "subnetwork_uri": f"projects/{project_id}/regions/{region}/subnetworks/default",
            },
...
        },
    }

[sudohainguyen]

my team resolved this issue by assigning specific value to subnetwork_uri attributes like this

cluster = {
        "project_id": project_id,
        "cluster_name": cluster_name,
        "config": {
...
            "gce_cluster_config": {
                "subnetwork_uri": f"projects/{project_id}/regions/{region}/subnetworks/default",
            },
...
        },
    }

so my question is, should we put it as default value of ClusterGenerator on behalf of airflow users?

[potiuk]

I believe so, but I think this is more question to Dataproc team - seem that they have changed the API. cc: @bhirsz @bjankie1 @VladaZakharova - and whether there is somethign to be done in the google provider about it.

[potiuk]

Converted it into a discussion so that it can be discussed there whether anything should be done about it.

[kunaljubce]

@potiuk I see this discussion is still open. Was there any conclusion on this? Because I am facing the same issue when I run the example_dataproc_batch.py job as part of my testing for PR #40630.

pytest --system google tests/system/providers/google/cloud/dataproc/example_dataproc_batch.py

[2024-07-14 06:12:05,700] {taskinstance.py:2996} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='dataproc_batch' AIRFLOW_CTX_TASK_ID='create_batch_2' AIRFLOW_CTX_EXECUTION_DATE='2024-07-14T06:12:03.902848+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-07-14T06:12:03.902848+00:00'
[2024-07-14T06:12:05.700+0000] {taskinstance.py:2996} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='dataproc_batch' AIRFLOW_CTX_TASK_ID='create_batch_2' AIRFLOW_CTX_EXECUTION_DATE='2024-07-14T06:12:03.902848+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-07-14T06:12:03.902848+00:00'
[2024-07-14T06:12:05.704+0000] {taskinstance.py:727} INFO - ::endgroup::
[2024-07-14T06:12:05.706+0000] {connection.py:277} WARNING - Connection schemes (type: google_cloud_platform) shall not contain '_' according to RFC3986.
[2024-07-14T06:12:05.707+0000] {base.py:84} INFO - Using connection ID 'google_cloud_default' for task execution.
[2024-07-14 06:12:05,707] {dataproc.py:3033} INFO - Creating batch batch-a1b2c3d4-dataproc-batch-2
[2024-07-14T06:12:05.707+0000] {dataproc.py:3033} INFO - Creating batch batch-a1b2c3d4-dataproc-batch-2
[2024-07-14 06:12:05,707] {dataproc.py:3034} INFO - Once started, the batch job will be available at https://console.cloud.google.com/dataproc/batches/asia-south2/batch-a1b2c3d4-dataproc-batch-2/monitoring?project=airflow-gc-03072024
[2024-07-14T06:12:05.707+0000] {dataproc.py:3034} INFO - Once started, the batch job will be available at https://console.cloud.google.com/dataproc/batches/asia-south2/batch-a1b2c3d4-dataproc-batch-2/monitoring?project=airflow-gc-03072024
[2024-07-14T06:12:05.708+0000] {credentials_provider.py:402} INFO - Getting connection using `google.auth.default()` since no explicit credentials are provided.
[2024-07-14 06:12:09,681] {taskinstance.py:3179} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1181, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "Network 'default' requires explicit subnetworks for instance creation. Specify a 'legacy' or 'auto' network, or specify subnetwork explicitly."
        debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.195.234:443 {created_time:"2024-07-14T06:12:09.676267379+00:00", grpc_status:3, grpc_message:"Network \'default\' requires explicit subnetworks for instance creation. Specify a \'legacy\' or \'auto\' network, or specify subnetwork explicitly."}"

[kunaljubce]

I got the same error when I ran example_dataproc_cluster_generator.py without subnet parameter. When I add the subnet parameter subnet_uri="my_subnet" while instantiating the ClusterGenerator(), the job runs fine and I can see the cluster being created on GCP UI.

[kunaljubce]

So it seems I can make it somehow work when I add subnet_uri to the batch attribute, as mentioned here -

BATCH_CONFIG = {
    "environment_config": {
        "execution_config": {
            "subnetwork_uri": "sample-subnet"
        }
    },
    "spark_batch": {
        "jar_file_uris": ["file:///usr/lib/spark/examples/jars/spark-examples.jar"],
        "main_class": "org.apache.spark.examples.SparkPi",
    },
}