Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎉 Add origins as a default to walkthrough #1369

Merged
merged 4 commits into from
Aug 7, 2023

Conversation

Marigold
Copy link
Collaborator

@Marigold Marigold commented Jul 21, 2023

  • refactor how we use cookiecutter in walkthrough and backport-migrate
  • use origins in walkthrough snapshot instead of source
  • remove YAML generation file from meadow step, YAML should be ideally used only in garden step
  • add all metadata options to dummy step (will be updated over time)

I tested all steps on a dummy dataset by running walkthrough snapshot --dummy-data

This should ideally be merged only after merging origins branch to master and rebasing.

@Marigold Marigold force-pushed the 2395-add-origins-as-a-default-to-walkthrough branch 3 times, most recently from f073bb9 to d37d0a5 Compare July 21, 2023 12:47
@danyx23 danyx23 force-pushed the 2395-add-origins-as-a-default-to-walkthrough branch from d37d0a5 to 21e5421 Compare July 21, 2023 14:09
@danyx23 danyx23 force-pushed the 2395-add-origins-as-a-default-to-walkthrough branch from 21e5421 to d55dc5a Compare July 31, 2023 14:47
@danyx23
Copy link
Contributor

danyx23 commented Jul 31, 2023

@Marigold I rebased this branch and force pushed it after some changes and brought this branch in line with the last field renames

@danyx23 danyx23 force-pushed the 2395-add-origins-as-a-default-to-walkthrough branch 3 times, most recently from 4834b7f to b7b2575 Compare August 1, 2023 16:32
Base automatically changed from origins to master August 3, 2023 07:29
@Marigold Marigold force-pushed the 2395-add-origins-as-a-default-to-walkthrough branch 2 times, most recently from cec3079 to c3f4d12 Compare August 3, 2023 10:33
@Marigold Marigold changed the base branch from master to refactor-walkthrough-2 August 3, 2023 10:34
@Marigold Marigold force-pushed the 2395-add-origins-as-a-default-to-walkthrough branch from c3f4d12 to e120d45 Compare August 3, 2023 11:04
@pabloarosado
Copy link
Contributor

Hi @Marigold I'm going through a small dataset, and the snapshot step went well! Now, when running the meadow step, I unticked the box to create a playground notebook, and got:

Traceback (most recent call last):
  File "/Users/prosado/Documents/owid/repos/etl/.venv/lib/python3.11/site-packages/pywebio/session/threadbased.py", line 86, in main_task
    target()
  File "/Users/prosado/Documents/owid/repos/etl/walkthrough/cli.py", line 67, in <lambda>
    apps_to_start = {app_name: lambda app=app: app(run_checks=run_checks) for app_name, app in apps.items()}
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prosado/Documents/owid/repos/etl/walkthrough/meadow.py", line 139, in app
    os.remove(notebook_path)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/prosado/Documents/owid/repos/etl/etl/steps/data/meadow/animal_welfare/2023-08-03/playground.ipynb'

The meadow step was successfully created though, but I guess the above error is a bug (I haven't looked into it, just informing you).

@Marigold
Copy link
Collaborator Author

Marigold commented Aug 4, 2023

It was a bug, fixed. Thanks for spotting it.

Copy link
Contributor

@pabloarosado pabloarosado left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I will make some small changes in a future PR, but for now let's merge to avoid blocking other things, thanks!

etl/steps/data/meadow/dummy/2020-01-01/dummy.py Outdated Show resolved Hide resolved
Base automatically changed from refactor-walkthrough-2 to master August 7, 2023 06:10
@Marigold Marigold force-pushed the 2395-add-origins-as-a-default-to-walkthrough branch from da95151 to f0d9704 Compare August 7, 2023 06:40
@Marigold Marigold merged commit 4015e52 into master Aug 7, 2023
@Marigold Marigold linked an issue Aug 7, 2023 that may be closed by this pull request
@Marigold Marigold deleted the 2395-add-origins-as-a-default-to-walkthrough branch August 7, 2023 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add origins as a default to walkthrough
3 participants