[Feature] Override credential validation to run commands like dbt run-operation

[yu-iskw]

Is this a regression in a recent version of dbt-bigquery?

• I believe this is a regression in dbt-bigquery functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

dbt-bigquery 1.9 requires the application default credentials even if the executed macro doesn't need to use the credentials.

WARNING:google.auth._default:Authentication failed using Compute Engine authentication due to unavailable metadata server.
08:57:43  Encountered an error while running operation: Database Error
  Runtime Error
    Failed to authenticate with supplied credentials
    error:
    Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

Expected/Previous Behavior

I expect we can execute a macro which don't need to access BigQuery without the application default credetials.

Steps To Reproduce

1. Install dbt-core 1.9 and dbt-bigquery 1.9 in the jaffle_shop project
2. Implement a dbt Macro named macros/echo.sql

   {% macro echo() %}
       {% do log("Hello, world!", info=True) %}
   {% endmacro %}
   

3. Execute dbt run-operator echo without the appliction default credentials
   • We can revoke the application default credentials by gcloud auth application-default revoke

Relevant log output

09:25:29  Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x107536e60>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x110dd5de0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x110dd6290>]}
09:25:29  Running with dbt=1.9.0
09:25:29  running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '/Users/yu/local/src/github/jaffle_shop', 'version_check': 'True', 'debug': 'True', 'log_path': '/Users/yu/local/src/github/jaffle_shop/logs', 'fail_fast': 'False', 'warn_error': 'None', 'use_colors': 'False', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'empty': 'None', 'log_format': 'default', 'static_parser': 'True', 'invocation_command': 'dbt --no-use-colors run-operation echo', 'introspect': 'True', 'target_path': 'None', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'send_anonymous_usage_stats': 'True'}
09:25:29  Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '20281131-e5da-48d0-8bec-3908076ecc3e', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1370653c0>]}
09:25:29  Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '20281131-e5da-48d0-8bec-3908076ecc3e', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x136efe890>]}
09:25:29  Registered adapter: bigquery=1.9.0
09:25:29  checksum: c99e828bba267739642b5a3ce85f17518764ea526e0e6c4fdc649171c1a66bff, vars: {}, profile: , target: , version: 1.9.0
09:25:29  Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
09:25:29  Partial parsing enabled, no changes found, skipping parsing
09:25:29  [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 2 unused configuration paths:
- models.jaffle_shop
- models.jaffle_shop.staging
09:25:29  Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '20281131-e5da-48d0-8bec-3908076ecc3e', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x13743c130>]}
09:25:29  Wrote artifact WritableManifest to /Users/yu/local/src/github/jaffle_shop/target/manifest.json
09:25:29  Wrote artifact SemanticManifest to /Users/yu/local/src/github/jaffle_shop/target/semantic_manifest.json
09:25:29  Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': '20281131-e5da-48d0-8bec-3908076ecc3e', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x137411780>]}
09:25:29  Found 2 operations, 489 macros
09:25:29  Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '20281131-e5da-48d0-8bec-3908076ecc3e', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x136acf640>]}
09:25:29  Acquiring new bigquery connection 'macro_echo'
09:25:29  Hello, world!
09:25:29  Opening a new connection, currently in state init
09:25:39  BigQuery adapter: Got an error when attempting to create a bigquery " "client: 'Runtime Error
  Failed to authenticate with supplied credentials
  error:
  Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.'
09:25:39  Encountered an error while running operation: Database Error
  Runtime Error
    Failed to authenticate with supplied credentials
    error:
    Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.
09:25:39  Traceback (most recent call last):
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/bigquery/connections.py", line 195, in open
    connection.handle = create_bigquery_client(connection.credentials)
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/bigquery/clients.py", line 24, in create_bigquery_client
    return _create_bigquery_client(credentials)
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
    result = target()
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/bigquery/clients.py", line 61, in _create_bigquery_client
    create_google_credentials(credentials),
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/bigquery/credentials.py", line 178, in create_google_credentials
    return _create_google_credentials(credentials)
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/bigquery/credentials.py", line 197, in _create_google_credentials
    creds, _ = _create_bigquery_defaults(scopes=credentials.scopes)
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/bigquery/credentials.py", line 239, in _create_bigquery_defaults
    raise DbtConfigError(f"Failed to authenticate with supplied credentials\nerror:\n{e}")
dbt_common.exceptions.base.DbtConfigError: Runtime Error
  Failed to authenticate with supplied credentials
  error:
  Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/task/run_operation.py", line 67, in run
    self._run_unsafe(package_name, macro_name)
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/task/run_operation.py", line 46, in _run_unsafe
    with adapter.connection_named("macro_{}".format(macro_name)):
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/base/impl.py", line 374, in connection_named
    self.release_connection()
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/base/impl.py", line 343, in release_connection
    self.connections.release()
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/base/connections.py", line 296, in release
    self.close(conn)
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/bigquery/connections.py", line 155, in close
    connection.handle.close()
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/contracts/connection.py", line 96, in handle
    self._handle.resolve(self)
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/contracts/connection.py", line 120, in resolve
    return self.opener(connection)
  File "/Users/yu/anaconda2/envs/jaffle-shop/lib/python3.10/site-packages/dbt/adapters/bigquery/connections.py", line 203, in open
    raise FailedToConnectError(str(e))
dbt.adapters.exceptions.connection.FailedToConnectError: Database Error
  Runtime Error
    Failed to authenticate with supplied credentials
    error:
    Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

09:25:39  Wrote artifact RunResultsArtifact to /Users/yu/local/src/github/jaffle_shop/target/run_results.json
09:25:39  Resource report: {"command_name": "run-operation", "command_success": false, "command_wall_clock_time": 9.877516, "process_in_blocks": "0", "process_kernel_time": 0.204768, "process_mem_max_rss": "201310208", "process_out_blocks": "0", "process_user_time": 1.248639}
09:25:39  Command `dbt run-operation` failed at 18:25:39.018344 after 9.88 seconds
09:25:39  Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x107536e60>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x136efe890>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x137355180>]}
09:25:39  Flushing usage events
09:25:39  An error was encountered while trying to flush usage events

### Environment

```markdown
- OS:
- Python: 3.11
- dbt-core (working version): 1.8
- dbt-bigquery (working version): 1.8
- dbt-core (regression version): 1.9
- dbt-bigquery (regression version): 1.9

Additional Context

No response

[amychen1776]

Hello @yu-iskw ! Curious what your use case here is as to why you would like us to support this?

[yu-iskw]

Hi @amychen1776,

I don't think a regular dbt project should work without the application default credentials. But, some dbt Packages might need to run dbt run-operation without the application default credentials for unit tests.

I posted an article about how to do unit testing dbt Package in addition to integration tests to dbt Developer Blog before. We can implement unit tests utilizing dbt run-operation, as we implement unit tests in regular programming languages. That approach enables us to directly make sure behaviors dbt Macros. We can also run unit tests in CI as I implemented in yu-iskw/dbt-unittest.

https://docs.getdbt.com/blog/unit-testing-dbt-packages

We consider if a dbt Package supports multiple major warehouses as BigQuery, Snowflake ans so on. And the developers want to run unit tests not only in CI jobs but also their local machines. For instance, first, the repository owner who doesn't mainly use BigQuery on a daily basis has to set the application default credentials in CI jobs. Second, the contributos have to set the application default credentials, even if he/she wants to enhance parts for a warehouse other than BigQuery. I suppose it can be hassle for them. That's why I want dbt-bigquery 1.9 to act as before. That would be helpful, though the behaviro might be directly related to some dbt Package develoeprs.

[amychen1776]

Thanks you @yu-iskw for the context! It was very helpful.

At this time, we do expect users to execute dbt with default credentials so I do not consider this a regression but validating what we expect to the current expected behavior. Due to this - I'm going to update this issue to a feature request to support overriding this validation. At this time, this is not a priority for our current roadmap but I will leave this issue open for others to also validate that they would like this.

[yu-iskw]

@amychen1776 Thank you for sharing your thought. I totally understand the less prioritized issue to resolve this. I would love to contribute to the enhancement. Please let me know, if there is anything I can help.