Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revised USA for changed upstream format - partial fix for #430 #431

Closed
wants to merge 6 commits into from
Closed

Revised USA for changed upstream format - partial fix for #430 #431

wants to merge 6 commits into from

Conversation

RichardMN
Copy link
Collaborator

This adjusts the download and clean_level_2 for the USA data so that it will calculate the number of years (!) of data to download and then download each separate yearly file from the @nytimes repo.

I suspect that the filename and directory structure may change before we get much beyond 2025 but this is "future-proofed" in some sense.

One of the columns appears to have been renamed in the level 2 data upstream but I think this is fixed too.

@RichardMN RichardMN requested a review from seabbs October 30, 2021 19:54
@RichardMN RichardMN linked an issue Oct 30, 2021 that may be closed by this pull request
Upstream data changes break our regional code - Colombia, Cuba, India, United States #430
Closed
5 tasks
@RichardMN RichardMN changed the title Revised USA code for now upstream format - partial fix for #430 Revised USA for changed upstream format - partial fix for #430 Oct 30, 2021
seabbs
seabbs approved these changes Dec 1, 2021
View reviewed changes
Copy link
Contributor

@seabbs seabbs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me (another apology for delayed response). Just needs a news update and then is good to go.

@RichardMN RichardMN removed the request for review from kathsherratt December 3, 2021 22:00
@RichardMN
Squashed commit of the following:
9195c3e 
commit 950e625
Author: Sam Abbott <[email protected]>
Date:   Fri Dec 3 16:37:48 2021 +0000

    Update NEWS.md

commit b93e19d
Merge: f6c9759 007b535
Author: Sam Abbott <[email protected]>
Date:   Fri Dec 3 16:36:05 2021 +0000

    Merge pull request epiforecasts#437 from epiforecasts/fewer-deps

    Reduce the number of `Imports`

commit f6c9759
Merge: 4509d81 ac3e63e
Author: Sam Abbott <[email protected]>
Date:   Fri Dec 3 16:33:49 2021 +0000

    Merge pull request epiforecasts#439 from epiforecasts/pr-checklist-news

    Update pr-checklist.yaml to include news nudge

commit 4509d81
Merge: 04009e7 b24f2b6
Author: Sam Abbott <[email protected]>
Date:   Fri Dec 3 16:33:08 2021 +0000

    Merge pull request epiforecasts#429 from epiforecasts/alternate-switzerland

    New data source for Switzerland - fix for epiforecasts#412

commit b24f2b6
Author: Richard Martin-Nielsen <[email protected]>
Date:   Wed Dec 1 19:50:08 2021 +0200

    Update all_country_data.rda

commit ed0f6c0
Author: Richard Martin-Nielsen <[email protected]>
Date:   Wed Dec 1 19:36:35 2021 +0200

    Fixed documentation links to FOPH as source of Switzerland data, updated News

commit 2a7f105
Merge: aea9b96 04009e7
Author: Sam Abbott <[email protected]>
Date:   Wed Dec 1 15:23:21 2021 +0000

    Merge branch 'master' into alternate-switzerland

commit 04009e7
Merge: d1630f5 68200f9
Author: Sam Abbott <[email protected]>
Date:   Wed Dec 1 14:27:05 2021 +0000

    Merge pull request epiforecasts#443 from epiforecasts/feature-boostrap-5-in-docs

    Switch to bootstrap 5 for pkgdown

commit 68200f9
Author: Sam Abbott <[email protected]>
Date:   Wed Dec 1 13:52:07 2021 +0000

    switch to bootstrap 5 for pkgdown

commit d1630f5
Merge: 71d7b1f ddbdb8e
Author: Sam Abbott <[email protected]>
Date:   Wed Dec 1 13:48:12 2021 +0000

    Merge pull request epiforecasts#442 from epiforecasts/fix-logo

    Fix giant logo by specifying width instead of height

commit ddbdb8e
Merge: a19643f 71d7b1f
Author: Sam Abbott <[email protected]>
Date:   Wed Dec 1 13:47:52 2021 +0000

    Merge branch 'master' into fix-logo

commit a19643f
Author: Hugo Gruson <[email protected]>
Date:   Wed Dec 1 14:30:11 2021 +0100

    Fix giant logo by specifying width instead of height
    fix epiforecasts#440

commit aea9b96
Merge: 0d8a6e5 71d7b1f
Author: Sam Abbott <[email protected]>
Date:   Wed Dec 1 10:52:16 2021 +0000

    Merge branch 'master' into alternate-switzerland

commit ac3e63e
Author: Sam Abbott <[email protected]>
Date:   Wed Dec 1 10:42:33 2021 +0000

    Update pr-checklist.yaml

commit 71d7b1f
Merge: 2e5cabc fb10488
Author: Sam Abbott <[email protected]>
Date:   Wed Dec 1 10:21:14 2021 +0000

    Merge pull request epiforecasts#438 from epiforecasts/rm-joss-action

    Remove action re-building JOSS paper

commit 007b535
Author: Hugo Gruson <[email protected]>
Date:   Wed Dec 1 10:43:26 2021 +0100

    Run devtools::document()

commit fe79373
Author: Hugo Gruson <[email protected]>
Date:   Wed Dec 1 10:43:16 2021 +0100

    Get rid of tidyselect

commit 3f5c115
Author: Hugo Gruson <[email protected]>
Date:   Wed Dec 1 10:27:44 2021 +0100

    Remove withr dependency as well

commit fb10488
Author: Hugo Gruson <[email protected]>
Date:   Tue Nov 30 19:09:00 2021 +0100

    Remove action re-building JOSS paper

commit 79857e9
Author: Hugo Gruson <[email protected]>
Date:   Tue Nov 30 19:03:17 2021 +0100

    Use dplyr::tibble in tests as well

commit d3c150b
Author: Hugo Gruson <[email protected]>
Date:   Tue Nov 30 18:59:59 2021 +0100

    Remove broken rlang::.data import
    to prevent it from being understood as a re-export

commit e711456
Author: Hugo Gruson <[email protected]>
Date:   Tue Nov 30 12:53:15 2021 +0100

    Run devtools::document()

commit 4032abd
Author: Hugo Gruson <[email protected]>
Date:   Tue Nov 30 12:43:43 2021 +0100

    Use stringr everywhere instead of stringi

commit 1c4ede0
Author: Hugo Gruson <[email protected]>
Date:   Tue Nov 30 12:34:20 2021 +0100

    Get tibble functions from dplyr

commit a499cc0
Author: Hugo Gruson <[email protected]>
Date:   Tue Nov 30 12:10:05 2021 +0100

    Re-export pipe from dplyr

commit 2e5cabc
Author: Sam Abbott <[email protected]>
Date:   Mon Nov 29 13:43:29 2021 +0000

    Update NEWS.md

commit d1350d9
Merge: 8c77b82 782256c
Author: Sam Abbott <[email protected]>
Date:   Mon Nov 29 13:41:00 2021 +0000

    Merge pull request epiforecasts#436 from epiforecasts/fix-germany

    fix Germany case/death data

commit 782256c
Author: Sebastian Funk <[email protected]>
Date:   Mon Nov 29 07:40:01 2021 +0000

    add missing sum

commit 4c7cb8c
Author: Sebastian Funk <[email protected]>
Date:   Mon Nov 29 07:19:09 2021 +0000

    fix Germany case/death data

commit 0d8a6e5
Author: Richard Martin-Nielsen <[email protected]>
Date:   Mon Oct 11 22:19:24 2021 +0300

    Fixes to keep lintr happier

commit 4eb12da
Author: Richard Martin-Nielsen <[email protected]>
Date:   Mon Oct 11 22:05:46 2021 +0300

    Cleaning code added and running. Passes tests.

    Removed AltSwitzerland code, generated data file of sample data and list of data sets.

commit 76e4c44
Author: Richard Martin-Nielsen <[email protected]>
Date:   Mon Oct 4 22:16:44 2021 +0300

    Moved old Switzerland code to SwitzerlandAlt, written functional 2-stage download method
@RichardMN
Update all_country_data.rda
db5f667
@github-actions
Copy link

github-actions bot commented Feb 2, 2022

This PR has been flagged as stale due to lack of activity

@seabbs seabbs self-assigned this Feb 4, 2022
@Bisaloo Bisaloo mentioned this pull request Feb 4, 2022
Closed
5 tasks
@Bisaloo
Copy link
Contributor

Bisaloo commented Feb 4, 2022

Commented about it in #430 (comment). Briefly, it doesn't look like the data source will be changing in this case. It was just the result of a confusingly worded announcement. If @RichardMN agrees, we can then close this PR.

@RichardMN
Copy link
Collaborator Author

As @Bisaloo has pointed out in the #430 conversation my PR had been based on a misunderstanding of the upstream data and isn't necessary.

@RichardMN RichardMN closed this Feb 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seabbs seabbs seabbs approved these changes

Assignees

@seabbs seabbs

Labels
no-pr-activity
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Upstream data changes break our regional code - Colombia, Cuba, India, United States
3 participants
@RichardMN @Bisaloo @seabbs