How to disable public ips when invoking the dataflow job using airflow DataflowStartFlexTemplateOperator Operator

[Harshith246]

When I am trying to invoke the dataflow job using DataflowStartFlexTemplateOperator from airflow, I am facing error as
Failed to start the VM, launcher-xxxxx, used for launching because of status code: FAILED_PRECONDITION, reason: Error: Message: Constraint constraints/compute.vmExternalIpAccess violated for project 646962485257. Add instance projects/projectxx/zones/us-central1-a/instances/launcher-xxxx to the constraint to use external IP with it. HTTP Code: 412.

Here is the code template am using, please advise how to get rid of that error ?

    from airflow import DAG
    from airflow.providers.google.cloud.operators.dataflow import DataflowStartFlexTemplateOperator
    from airflow.utils.dates import days_ago, timedelta
    
    default_args = {"owner": "XX",
		        "depends_on_past": False,
                "start_date": days_ago(3),
		        "email": "[email protected]",
		        "email_on_failure": True,
		        "email_on_retry": False,
		        "retries": 1,
		        "retry_delay": timedelta(minutes=5)}
    
    with DAG('dataflow_dag', default_args=default_args, schedule_interval='@once') as dag:
        dataflow_task = DataflowStartFlexTemplateOperator(
            task_id='dataflow_example', 
            location = 'location',      
            project_id = 'xxxx',
            body={
                "launchParameter":{
                    "jobName": "dataflow-airflow-test",
                    "containerSpecGcsPath": "gs://sample/sample-template.json",
                    "parameters":{  
                        'project':'xxxx',
                        'region':'us-central1',
                        'subnetwork':'https://www.googleapis.com/compute/v1/projects/project/regions/us-central1/subnetworks/xxxx',
                        'disable-public-ips':'True',
                        'use_public_ips':'False',
                        'no_use_public_ips': 'True',
                        'max-workers':'1',
                        'temp-location':'gs://framework-test/temp'
                    },
                }
            },
            impersonation_chain="[email protected]",
            dag=dag
        )
    
        dataflow_task

Note : tried all these three parameters ('disable-public-ips':'True' , 'use_public_ips':'False', 'no_use_public_ips': 'True',) but no luck

@Taragolis Appreciate your advice here

[luisoviedo007]

I had the same problem. To fix it I added this:"environment" : {"ipConfiguration": "WORKER_IP_PRIVATE"} inside launchParameter.

Here's an example:

    body = {
        "launchParameter": {              
            "jobName": "dataflow-airflow", 
            "parameters": {                          
                "input": input_data,                 
                "output": output_data
            },
            "containerSpecGcsPath": my_flex_template, 

            "environment" : {
            "ipConfiguration": "WORKER_IP_PRIVATE"
        }
        },
        
        "validateOnly": False             
    }

Documentation:
https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.flexTemplates/launch#request-body