Refactor: run DBT from devcontainer #3515

thekaveman · 2024-10-25T21:59:04Z

Description

After working on #3468 and trying to get things running on a new local environment, I thought it would be nice to encode the dependencies and tools into the devcontainer configuration. This makes it much easier to get DBT running from a fresh environment, and is system/platform agnostic.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation

How has this been tested?

See the README updates for steps to test/run this locally.

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

No action required
Actions required (specified below)

angela-tran · 2024-10-31T18:27:26Z

Starting a review of this. Gonna try opening up the devcontainer!

angela-tran · 2024-10-31T18:49:14Z

The devcontainer was created and opened for me 🎉 I think I'm now at the point of setting up my dbt profile (probably the command in postAttach.sh at line 13), and it seems to be hanging at a part where a browser is supposed to open for me to login:

Snippets of my output:

18:28:24  Running with dbt=1.5.1
18:28:24  Setting up your profile.
schema (usually your name; will be added as a prefix to schemas e.g. <schema>_mart_gtfs): angela
maximum_bytes_billed (the maximum number of bytes allowed per BigQuery query; default is 2 TB) [2000000000000]:
18:34:10  Profile calitp_warehouse written to /home/calitp/.dbt/profiles.yml using project's profile_template.yml and your supplied value
s. Run 'dbt debug' to validate the connection.
18:34:15  Running with dbt=1.5.1
18:34:15  dbt version: 1.5.1
18:34:15  python version: 3.9.20

...

18:34:18  1 check failed:
18:34:18  dbt was unable to connect to the specified database.
The database returned the following error:

  >Database Error
  Runtime Error

    dbt encountered an error while trying to read your profiles.yml file.

    Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/
external/set-up-adc for more information.


Check your database credentials and try again. For more information, visit:
https://docs.getdbt.com/docs/configure-your-profile


Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

You can skip diagnostics next time by using the following flag:
  gcloud init --skip-diagnostics

Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

You must sign in to continue. Would you like to sign in (Y/n)?  Y

Your browser has been opened to visit:
...

Nothing happens here. 🤔

angela-tran · 2024-10-31T18:52:37Z

I tried copying the URL from the output and opening that in a browser manually, and that did show me a Google consent screen for Google Cloud SDK, but then when I finished going through the consent screens, it redirected me to a localhost:8085 URL. I guess this is not too surprising when you look at the parameters of the URL I pasted into my browser.

Screenshot of the localhost redirect

My devcontainer terminal is still waiting for me to log in. 🤔

Screenshot of devcontainer terminal

thekaveman · 2024-10-31T20:16:54Z

That second screenshot with the links in the console, can you click from there? That's what I had to do, and it worked after I logged in to my Google account on the browser.

angela-tran · 2024-10-31T20:41:37Z

That second screenshot with the links in the console, can you click from there? That's what I had to do, and it worked after I logged in to my Google account on the browser.

It doesn't let me click the full link, unfortunately. For some reason, it's getting cut off maybe by a line break or something. If it'd be helpful for me to get on a screenshare with you, let me know

lalver1 · 2024-11-01T15:18:58Z

The devcontainer was created and it opened for me too 🙂. I did notice an error when postAttach.sh ran, but it didn't prevent the rest of the script from running 🤔:

[1722 ms] Start: Run in container: /bin/bash .devcontainer/postAttach.sh
pre-commit installed at .git/hooks/pre-commit
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/isort.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pycqa/bandit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/home/calitp/.cache/pre-commit/repoz_qq3lxw/py_env-python3/bin/python', '-mpip', 'install', '.')
return code: 1
stdout:
    Processing /home/calitp/.cache/pre-commit/repoz_qq3lxw
      Preparing metadata (setup.py): started
      Preparing metadata (setup.py): finished with status 'done'
    Collecting GitPython>=1.0.1 (from bandit==0.0.0)
      Downloading GitPython-3.1.43-py3-none-any.whl.metadata (13 kB)
    Collecting PyYAML>=5.3.1 (from bandit==0.0.0)
      Using cached PyYAML-6.0.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (2.1 kB)
    Collecting stevedore>=1.20.0 (from bandit==0.0.0)
      Downloading stevedore-5.3.0-py3-none-any.whl.metadata (2.3 kB)
    Collecting rich (from bandit==0.0.0)
      Using cached rich-13.9.3-py3-none-any.whl.metadata (18 kB)
    Collecting gitdb<5,>=4.0.1 (from GitPython>=1.0.1->bandit==0.0.0)
      Downloading gitdb-4.0.11-py3-none-any.whl.metadata (1.2 kB)
    Collecting pbr>=2.0.0 (from stevedore>=1.20.0->bandit==0.0.0)
      Using cached pbr-6.1.0-py2.py3-none-any.whl.metadata (3.4 kB)
    Collecting markdown-it-py>=2.2.0 (from rich->bandit==0.0.0)
      Using cached markdown_it_py-3.0.0-py3-none-any.whl.metadata (6.9 kB)
    Collecting pygments<3.0.0,>=2.13.0 (from rich->bandit==0.0.0)
      Using cached pygments-2.18.0-py3-none-any.whl.metadata (2.5 kB)
    Collecting typing-extensions<5.0,>=4.0.0 (from rich->bandit==0.0.0)
      Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
    Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython>=1.0.1->bandit==0.0.0)
      Downloading smmap-5.0.1-py3-none-any.whl.metadata (4.3 kB)
    Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich->bandit==0.0.0)
      Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
    Downloading GitPython-3.1.43-py3-none-any.whl (207 kB)
    Using cached PyYAML-6.0.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (720 kB)
    Downloading stevedore-5.3.0-py3-none-any.whl (49 kB)
    Using cached rich-13.9.3-py3-none-any.whl (242 kB)
    Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)
    Using cached markdown_it_py-3.0.0-py3-none-any.whl (87 kB)
    Using cached pbr-6.1.0-py2.py3-none-any.whl (108 kB)
    Using cached pygments-2.18.0-py3-none-any.whl (1.2 MB)
    Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)
    Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
    Downloading smmap-5.0.1-py3-none-any.whl (24 kB)
    Building wheels for collected packages: bandit
      Building wheel for bandit (setup.py): started
      Building wheel for bandit (setup.py): finished with status 'done'
      Created wheel for bandit: filename=bandit-0.0.0-py3-none-any.whl size=119913 sha256=4507954499c0c4f40fbbd8c35cbfa4a920ad4494375a71ba501d3f0649f2fe1a
      Stored in directory: /tmp/pip-ephem-wheel-cache-8gp8q082/wheels/6a/4d/55/ae8cdabeab19e9f4ffc7f4dab60e605c1b4201fce4f3a9d5b4
    Successfully built bandit
    Installing collected packages: typing-extensions, smmap, PyYAML, pygments, pbr, mdurl, stevedore, markdown-it-py, gitdb, rich, GitPython, bandit
stderr:
    ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device: '/home/calitp/.cache/pre-commit/repoz_qq3lxw/py_env-python3/lib/python3.9/site-packages/pbr'
Check the log at /home/calitp/.cache/pre-commit/pre-commit.log
14:36:10  Running with dbt=1.5.1
14:36:10  Setting up your profile.

Then I was able to create my profile using my name as schema, use the default maximum_bytes_billed, and sign in with my @compiler.la email after going through the Google Cloud SDK consent screens. I then was able to select Pick cloud project to use.

thekaveman · 2024-11-04T18:48:22Z

@lalver1 I see in your output:

ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device

This seems strange? Did your Docker VM run out of virtual hard disk?

lalver1 · 2024-11-04T23:33:25Z

Looks like the problem was with my Docker Engine @thekaveman. I cleaned up several dangling images I had and rebuilt the container and postAttach.sh ran fine with no errors 👍

angela-tran

I moved from Windows 10 to Linux (for reasons unrelated to this PR). I no longer experience the issue with authentication and am able to get through the dbt profile set up now 🎉

Ran docker compose run dbt debug and got successful output:

- remove node install - add google-cloud-cli install - use non-root user

- extra mounted volumes - env vars modernize devcontainer.json syntax use postAttach to ensure dbt and gcloud are configured

script seems unused

evansiroky · 2024-11-13T20:56:21Z

Interesting. I'll defer to @vevetron and/or @tiffanychu90 for review.

vevetron

Ran it all on my mac and worked like a dream.

vevetron · 2024-11-18T22:28:02Z

There was one moment towards the end of the install where it seems like it froze:

Credentials saved to file: [/home/calitp/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "cal-itp-data-infra-staging" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.

I'm still learning to use it in VS code but lines like
docker compose run dbt run -s +fct_service_alerts_messages_unnested+ from the .devcontainer folder work great.

thekaveman · 2024-11-18T22:38:55Z

@vevetron

There was one moment towards the end of the install where it seems like it froze

Did it ever come back? Or was it just hung here?

Was this on your very first time launching the devcontainer in VS Code? What about on subsequent launches?

vevetron · 2024-11-18T22:40:34Z

It didn't come back after I restarted. When I launched it again it jumped to the post-hook and then hit a terminal inside the container.

thekaveman · 2024-11-18T22:48:15Z

Hmmm, do you see the "success" output from the postattach hook? Something like:

vevetron · 2024-11-18T23:35:53Z

Hmmm, do you see the "success" output from the postattach hook? Something like:

Yes I do see the "success" output.

thekaveman · 2024-11-18T23:41:37Z

Awesome! Then I'll merge this if that's OK with you?

vevetron · 2024-11-20T00:21:11Z

Yes please merge it in!

thekaveman force-pushed the refactor/devcontainer branch from 404f963 to 5fb0819 Compare October 28, 2024 04:03

thekaveman self-assigned this Oct 28, 2024

thekaveman marked this pull request as ready for review October 28, 2024 17:21

thekaveman requested review from evansiroky, vevetron, tiffanychu90 and hunterowens as code owners October 28, 2024 17:21

thekaveman mentioned this pull request Oct 29, 2024

Fix: reorder Benefits mart fields #3518

Merged

6 tasks

thekaveman force-pushed the refactor/devcontainer branch from 5fb0819 to da3a916 Compare November 4, 2024 18:46

angela-tran approved these changes Nov 5, 2024

View reviewed changes

thekaveman added 4 commits November 12, 2024 16:26

refactor(devcontainer): use Dockerfile.local as starting point

6974933

- remove node install - add google-cloud-cli install - use non-root user

refactor(devcontainer): use compose.yml to launch/config

5cb3ea3

- extra mounted volumes - env vars modernize devcontainer.json syntax use postAttach to ensure dbt and gcloud are configured

chore(warehouse): replace dbt script with compose service

47ca6a7

script seems unused

docs: add steps for running devcontainer

6263e2b

thekaveman force-pushed the refactor/devcontainer branch from da3a916 to 6263e2b Compare November 13, 2024 00:26

vevetron approved these changes Nov 18, 2024

View reviewed changes

Merge branch 'main' into refactor/devcontainer

033a3b8

thekaveman merged commit 12e3b01 into main Nov 20, 2024
4 checks passed

thekaveman deleted the refactor/devcontainer branch November 20, 2024 00:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor: run DBT from devcontainer #3515

Refactor: run DBT from devcontainer #3515

thekaveman commented Oct 25, 2024 •

edited

Loading

angela-tran commented Oct 31, 2024

angela-tran commented Oct 31, 2024 •

edited

Loading

angela-tran commented Oct 31, 2024 •

edited

Loading

thekaveman commented Oct 31, 2024

angela-tran commented Oct 31, 2024

lalver1 commented Nov 1, 2024

thekaveman commented Nov 4, 2024

lalver1 commented Nov 4, 2024

angela-tran left a comment

evansiroky commented Nov 13, 2024

vevetron left a comment

vevetron commented Nov 18, 2024

thekaveman commented Nov 18, 2024

vevetron commented Nov 18, 2024

thekaveman commented Nov 18, 2024

vevetron commented Nov 18, 2024

thekaveman commented Nov 18, 2024

vevetron commented Nov 20, 2024

Refactor: run DBT from devcontainer #3515

Refactor: run DBT from devcontainer #3515

Conversation

thekaveman commented Oct 25, 2024 • edited Loading

Description

Type of change

How has this been tested?

Post-merge follow-ups

angela-tran commented Oct 31, 2024

angela-tran commented Oct 31, 2024 • edited Loading

angela-tran commented Oct 31, 2024 • edited Loading

thekaveman commented Oct 31, 2024

angela-tran commented Oct 31, 2024

lalver1 commented Nov 1, 2024

thekaveman commented Nov 4, 2024

lalver1 commented Nov 4, 2024

angela-tran left a comment

Choose a reason for hiding this comment

evansiroky commented Nov 13, 2024

vevetron left a comment

Choose a reason for hiding this comment

vevetron commented Nov 18, 2024

thekaveman commented Nov 18, 2024

vevetron commented Nov 18, 2024

thekaveman commented Nov 18, 2024

vevetron commented Nov 18, 2024

thekaveman commented Nov 18, 2024

vevetron commented Nov 20, 2024

thekaveman commented Oct 25, 2024 •

edited

Loading

angela-tran commented Oct 31, 2024 •

edited

Loading

angela-tran commented Oct 31, 2024 •

edited

Loading