Skip to content

Commit

Permalink
[dataproc] adds project to args for submit
Browse files Browse the repository at this point in the history
Currently, if you try to pass `--project` to `hailctl dataproc submit`,
it still tries to use the project configured via `gcloud config`, and
passes through the value specified for `--project` in the
`pass_through_args`.

This change allows the user to specify which project their Dataproc
cluster is in as part of the command.

An example command where the project is ignored on `main` but used
correctly on this branch is:

```bash
hailctl dataproc submit test.py --region us-central1 --project broad-ctsa
```
  • Loading branch information
iris-garden committed Jul 16, 2024
1 parent 49a13e0 commit 2442764
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 1 deletion.
12 changes: 11 additions & 1 deletion hail/python/hailtop/hailctl/dataproc/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,7 @@ def submit(
] = None,
dry_run: DryRunOption = False,
region: Ann[Optional[str], Opt(help='Compute region for the cluster.')] = None,
project: Ann[Optional[str], Opt(help='GCP project for the cluster.')] = None,
arguments: Ann[
Optional[List[str]], Arg(help='You should use -- if you want to pass option-like arguments through.')
] = None,
Expand All @@ -334,7 +335,16 @@ def submit(
"""
dataproc_submit(
name, script, files, pyfiles, properties, gcloud_configuration, dry_run, region, [*(arguments or []), *ctx.args]
name,
script,
files,
pyfiles,
properties,
gcloud_configuration,
dry_run,
region,
project,
[*(arguments or []), *ctx.args],
)


Expand Down
4 changes: 4 additions & 0 deletions hail/python/hailtop/hailctl/dataproc/submit.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ def submit(
gcloud_configuration: Optional[str],
dry_run: bool,
region: Optional[str],
project: Optional[str],
pass_through_args: List[str],
):
print("Submitting to cluster '{}'...".format(name))
Expand Down Expand Up @@ -67,6 +68,9 @@ def submit(
if region:
cmd.append('--region={}'.format(region))

if project:
cmd.append('--project={}'.format(project))

# append arguments to pass to the Hail script
if pass_through_args:
cmd.append('--')
Expand Down

0 comments on commit 2442764

Please sign in to comment.