From 25c11b9e1136152c4caa456fc79b2c5493849662 Mon Sep 17 00:00:00 2001 From: "Zhian N. Kamvar" Date: Mon, 18 Sep 2023 09:44:52 -0700 Subject: [PATCH] [automation] transform lesson to sandpaper --- .editorconfig | 26 ++ .github/workflows/README.md | 198 ++++++++++ .github/workflows/pr-close-signal.yaml | 23 ++ .github/workflows/pr-comment.yaml | 185 +++++++++ .github/workflows/pr-post-remove-branch.yaml | 32 ++ .github/workflows/pr-preflight.yaml | 39 ++ .github/workflows/pr-receive.yaml | 131 +++++++ .github/workflows/sandpaper-main.yaml | 61 +++ .github/workflows/sandpaper-version.txt | 1 + .github/workflows/update-cache.yaml | 125 ++++++ .github/workflows/update-workflows.yaml | 66 ++++ .github/workflows/workbench-beta-phase.yml | 60 +++ .gitignore | 55 +++ CODE_OF_CONDUCT.md | 13 + CONTRIBUTING.md | 123 ++++++ LICENSE.md | 79 ++++ README.md | 12 +- _extras/data.md | 73 ---- config.yaml | 83 ++++ index.md | 121 +++--- instructors/data.md | 79 ++++ .../instructor-notes.md | 19 +- reference.md => learners/reference.md | 5 +- learners/setup.md | 363 ++++++++++++++++++ profiles/learner-profiles.md | 5 + setup.md | 353 ----------------- site/README.md | 2 + 27 files changed, 1838 insertions(+), 494 deletions(-) create mode 100644 .editorconfig create mode 100755 .github/workflows/README.md create mode 100755 .github/workflows/pr-close-signal.yaml create mode 100755 .github/workflows/pr-comment.yaml create mode 100755 .github/workflows/pr-post-remove-branch.yaml create mode 100755 .github/workflows/pr-preflight.yaml create mode 100755 .github/workflows/pr-receive.yaml create mode 100755 .github/workflows/sandpaper-main.yaml create mode 100644 .github/workflows/sandpaper-version.txt create mode 100755 .github/workflows/update-cache.yaml create mode 100755 .github/workflows/update-workflows.yaml create mode 100644 .github/workflows/workbench-beta-phase.yml create mode 100644 .gitignore create mode 100644 CODE_OF_CONDUCT.md create mode 100644 CONTRIBUTING.md create mode 100644 LICENSE.md delete mode 100644 _extras/data.md create mode 100644 config.yaml create mode 100644 instructors/data.md rename _extras/guide.md => instructors/instructor-notes.md (70%) rename reference.md => learners/reference.md (51%) create mode 100644 learners/setup.md create mode 100644 profiles/learner-profiles.md delete mode 100644 setup.md create mode 100644 site/README.md diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 0000000..5bf4860 --- /dev/null +++ b/.editorconfig @@ -0,0 +1,26 @@ +root = true + +[*] +charset = utf-8 +insert_final_newline = true +trim_trailing_whitespace = true + +[*.md] +indent_size = 2 +indent_style = space +max_line_length = 100 # Please keep this in sync with bin/lesson_check.py! +trim_trailing_whitespace = false # keep trailing spaces in markdown - 2+ spaces are translated to a hard break (
) + +[*.r] +max_line_length = 80 + +[*.py] +indent_size = 4 +indent_style = space +max_line_length = 79 + +[*.sh] +end_of_line = lf + +[Makefile] +indent_style = tab diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100755 index 0000000..d6edf88 --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,198 @@ +# Carpentries Workflows + +This directory contains workflows to be used for Lessons using the {sandpaper} +lesson infrastructure. Two of these workflows require R (`sandpaper-main.yaml` +and `pr-recieve.yaml`) and the rest are bots to handle pull request management. + +These workflows will likely change as {sandpaper} evolves, so it is important to +keep them up-to-date. To do this in your lesson you can do the following in your +R console: + +```r +# Install/Update sandpaper +options(repos = c(carpentries = "https://carpentries.r-universe.dev/", + CRAN = "https://cloud.r-project.org")) +install.packages("sandpaper") + +# update the workflows in your lesson +library("sandpaper") +update_github_workflows() +``` + +Inside this folder, you will find a file called `sandpaper-version.txt`, which +will contain a version number for sandpaper. This will be used in the future to +alert you if a workflow update is needed. + +What follows are the descriptions of the workflow files: + +## Deployment + +### 01 Build and Deploy (sandpaper-main.yaml) + +This is the main driver that will only act on the main branch of the repository. +This workflow does the following: + + 1. checks out the lesson + 2. provisions the following resources + - R + - pandoc + - lesson infrastructure (stored in a cache) + - lesson dependencies if needed (stored in a cache) + 3. builds the lesson via `sandpaper:::ci_deploy()` + +#### Caching + +This workflow has two caches; one cache is for the lesson infrastructure and +the other is for the the lesson dependencies if the lesson contains rendered +content. These caches are invalidated by new versions of the infrastructure and +the `renv.lock` file, respectively. If there is a problem with the cache, +manual invaliation is necessary. You will need maintain access to the repository +and you can either go to the actions tab and [click on the caches button to find +and invalidate the failing cache](https://github.blog/changelog/2022-10-20-manage-caches-in-your-actions-workflows-from-web-interface/) +or by setting the `CACHE_VERSION` secret to the current date (which will +invalidate all of the caches). + +## Updates + +### Setup Information + +These workflows run on a schedule and at the maintainer's request. Because they +create pull requests that update workflows/require the downstream actions to run, +they need a special repository/organization secret token called +`SANDPAPER_WORKFLOW` and it must have the `public_repo` and `workflow` scope. + +This can be an individual user token, OR it can be a trusted bot account. If you +have a repository in one of the official Carpentries accounts, then you do not +need to worry about this token being present because the Carpentries Core Team +will take care of supplying this token. + +If you want to use your personal account: you can go to + +to create a token. Once you have created your token, you should copy it to your +clipboard and then go to your repository's settings > secrets > actions and +create or edit the `SANDPAPER_WORKFLOW` secret, pasting in the generated token. + +If you do not specify your token correctly, the runs will not fail and they will +give you instructions to provide the token for your repository. + +### 02 Maintain: Update Workflow Files (update-workflow.yaml) + +The {sandpaper} repository was designed to do as much as possible to separate +the tools from the content. For local builds, this is absolutely true, but +there is a minor issue when it comes to workflow files: they must live inside +the repository. + +This workflow ensures that the workflow files are up-to-date. The way it work is +to download the update-workflows.sh script from GitHub and run it. The script +will do the following: + +1. check the recorded version of sandpaper against the current version on github +2. update the files if there is a difference in versions + +After the files are updated, if there are any changes, they are pushed to a +branch called `update/workflows` and a pull request is created. Maintainers are +encouraged to review the changes and accept the pull request if the outputs +are okay. + +This update is run ~~weekly or~~ on demand. + +### 03 Maintain: Update Package Cache (update-cache.yaml) + +For lessons that have generated content, we use {renv} to ensure that the output +is stable. This is controlled by a single lockfile which documents the packages +needed for the lesson and the version numbers. This workflow is skipped in +lessons that do not have generated content. + +Because the lessons need to remain current with the package ecosystem, it's a +good idea to make sure these packages can be updated periodically. The +update cache workflow will do this by checking for updates, applying them in a +branch called `updates/packages` and creating a pull request with _only the +lockfile changed_. + +From here, the markdown documents will be rebuilt and you can inspect what has +changed based on how the packages have updated. + +## Pull Request and Review Management + +Because our lessons execute code, pull requests are a secruity risk for any +lesson and thus have security measures associted with them. **Do not merge any +pull requests that do not pass checks and do not have bots commented on them.** + +This series of workflows all go together and are described in the following +diagram and the below sections: + +![Graph representation of a pull request](https://carpentries.github.io/sandpaper/articles/img/pr-flow.dot.svg) + +### Pre Flight Pull Request Validation (pr-preflight.yaml) + +This workflow runs every time a pull request is created and its purpose is to +validate that the pull request is okay to run. This means the following things: + +1. The pull request does not contain modified workflow files +2. If the pull request contains modified workflow files, it does not contain + modified content files (such as a situation where @carpentries-bot will + make an automated pull request) +3. The pull request does not contain an invalid commit hash (e.g. from a fork + that was made before a lesson was transitioned from styles to use the + workbench). + +Once the checks are finished, a comment is issued to the pull request, which +will allow maintainers to determine if it is safe to run the +"Receive Pull Request" workflow from new contributors. + +### Recieve Pull Request (pr-recieve.yaml) + +**Note of caution:** This workflow runs arbitrary code by anyone who creates a +pull request. GitHub has safeguarded the token used in this workflow to have no +priviledges in the repository, but we have taken precautions to protect against +spoofing. + +This workflow is triggered with every push to a pull request. If this workflow +is already running and a new push is sent to the pull request, the workflow +running from the previous push will be cancelled and a new workflow run will be +started. + +The first step of this workflow is to check if it is valid (e.g. that no +workflow files have been modified). If there are workflow files that have been +modified, a comment is made that indicates that the workflow is not run. If +both a workflow file and lesson content is modified, an error will occurr. + +The second step (if valid) is to build the generated content from the pull +request. This builds the content and uploads three artifacts: + +1. The pull request number (pr) +2. A summary of changes after the rendering process (diff) +3. The rendered files (build) + +Because this workflow builds generated content, it follows the same general +process as the `sandpaper-main` workflow with the same caching mechanisms. + +The artifacts produced are used by the next workflow. + +### Comment on Pull Request (pr-comment.yaml) + +This workflow is triggered if the `pr-recieve.yaml` workflow is successful. +The steps in this workflow are: + +1. Test if the workflow is valid and comment the validity of the workflow to the + pull request. +2. If it is valid: create an orphan branch with two commits: the current state + of the repository and the proposed changes. +3. If it is valid: update the pull request comment with the summary of changes + +Importantly: if the pull request is invalid, the branch is not created so any +malicious code is not published. + +From here, the maintainer can request changes from the author and eventually +either merge or reject the PR. When this happens, if the PR was valid, the +preview branch needs to be deleted. + +### Send Close PR Signal (pr-close-signal.yaml) + +Triggered any time a pull request is closed. This emits an artifact that is the +pull request number for the next action + +### Remove Pull Request Branch (pr-post-remove-branch.yaml) + +Tiggered by `pr-close-signal.yaml`. This removes the temporary branch associated with +the pull request (if it was created). diff --git a/.github/workflows/pr-close-signal.yaml b/.github/workflows/pr-close-signal.yaml new file mode 100755 index 0000000..9b129d5 --- /dev/null +++ b/.github/workflows/pr-close-signal.yaml @@ -0,0 +1,23 @@ +name: "Bot: Send Close Pull Request Signal" + +on: + pull_request: + types: + [closed] + +jobs: + send-close-signal: + name: "Send closing signal" + runs-on: ubuntu-latest + if: ${{ github.event.action == 'closed' }} + steps: + - name: "Create PRtifact" + run: | + mkdir -p ./pr + printf ${{ github.event.number }} > ./pr/NUM + - name: Upload Diff + uses: actions/upload-artifact@v3 + with: + name: pr + path: ./pr + diff --git a/.github/workflows/pr-comment.yaml b/.github/workflows/pr-comment.yaml new file mode 100755 index 0000000..bb2eb03 --- /dev/null +++ b/.github/workflows/pr-comment.yaml @@ -0,0 +1,185 @@ +name: "Bot: Comment on the Pull Request" + +# read-write repo token +# access to secrets +on: + workflow_run: + workflows: ["Receive Pull Request"] + types: + - completed + +concurrency: + group: pr-${{ github.event.workflow_run.pull_requests[0].number }} + cancel-in-progress: true + + +jobs: + # Pull requests are valid if: + # - they match the sha of the workflow run head commit + # - they are open + # - no .github files were committed + test-pr: + name: "Test if pull request is valid" + runs-on: ubuntu-latest + if: > + github.event.workflow_run.event == 'pull_request' && + github.event.workflow_run.conclusion == 'success' + outputs: + is_valid: ${{ steps.check-pr.outputs.VALID }} + payload: ${{ steps.check-pr.outputs.payload }} + number: ${{ steps.get-pr.outputs.NUM }} + msg: ${{ steps.check-pr.outputs.MSG }} + steps: + - name: 'Download PR artifact' + id: dl + uses: carpentries/actions/download-workflow-artifact@main + with: + run: ${{ github.event.workflow_run.id }} + name: 'pr' + + - name: "Get PR Number" + if: ${{ steps.dl.outputs.success == 'true' }} + id: get-pr + run: | + unzip pr.zip + echo "NUM=$(<./NR)" >> $GITHUB_OUTPUT + + - name: "Fail if PR number was not present" + id: bad-pr + if: ${{ steps.dl.outputs.success != 'true' }} + run: | + echo '::error::A pull request number was not recorded. The pull request that triggered this workflow is likely malicious.' + exit 1 + - name: "Get Invalid Hashes File" + id: hash + run: | + echo "json<> $GITHUB_OUTPUT + - name: "Check PR" + id: check-pr + if: ${{ steps.dl.outputs.success == 'true' }} + uses: carpentries/actions/check-valid-pr@main + with: + pr: ${{ steps.get-pr.outputs.NUM }} + sha: ${{ github.event.workflow_run.head_sha }} + headroom: 3 # if it's within the last three commits, we can keep going, because it's likely rapid-fire + invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} + fail_on_error: true + + # Create an orphan branch on this repository with two commits + # - the current HEAD of the md-outputs branch + # - the output from running the current HEAD of the pull request through + # the md generator + create-branch: + name: "Create Git Branch" + needs: test-pr + runs-on: ubuntu-latest + if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + env: + NR: ${{ needs.test-pr.outputs.number }} + permissions: + contents: write + steps: + - name: 'Checkout md outputs' + uses: actions/checkout@v3 + with: + ref: md-outputs + path: built + fetch-depth: 1 + + - name: 'Download built markdown' + id: dl + uses: carpentries/actions/download-workflow-artifact@main + with: + run: ${{ github.event.workflow_run.id }} + name: 'built' + + - if: ${{ steps.dl.outputs.success == 'true' }} + run: unzip built.zip + + - name: "Create orphan and push" + if: ${{ steps.dl.outputs.success == 'true' }} + run: | + cd built/ + git config --local user.email "actions@github.com" + git config --local user.name "GitHub Actions" + CURR_HEAD=$(git rev-parse HEAD) + git checkout --orphan md-outputs-PR-${NR} + git add -A + git commit -m "source commit: ${CURR_HEAD}" + ls -A | grep -v '^.git$' | xargs -I _ rm -r '_' + cd .. + unzip -o -d built built.zip + cd built + git add -A + git commit --allow-empty -m "differences for PR #${NR}" + git push -u --force --set-upstream origin md-outputs-PR-${NR} + + # Comment on the Pull Request with a link to the branch and the diff + comment-pr: + name: "Comment on Pull Request" + needs: [test-pr, create-branch] + runs-on: ubuntu-latest + if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + env: + NR: ${{ needs.test-pr.outputs.number }} + permissions: + pull-requests: write + steps: + - name: 'Download comment artifact' + id: dl + uses: carpentries/actions/download-workflow-artifact@main + with: + run: ${{ github.event.workflow_run.id }} + name: 'diff' + + - if: ${{ steps.dl.outputs.success == 'true' }} + run: unzip ${{ github.workspace }}/diff.zip + + - name: "Comment on PR" + id: comment-diff + if: ${{ steps.dl.outputs.success == 'true' }} + uses: carpentries/actions/comment-diff@main + with: + pr: ${{ env.NR }} + path: ${{ github.workspace }}/diff.md + + # Comment if the PR is open and matches the SHA, but the workflow files have + # changed + comment-changed-workflow: + name: "Comment if workflow files have changed" + needs: test-pr + runs-on: ubuntu-latest + if: ${{ always() && needs.test-pr.outputs.is_valid == 'false' }} + env: + NR: ${{ github.event.workflow_run.pull_requests[0].number }} + body: ${{ needs.test-pr.outputs.msg }} + permissions: + pull-requests: write + steps: + - name: 'Check for spoofing' + id: dl + uses: carpentries/actions/download-workflow-artifact@main + with: + run: ${{ github.event.workflow_run.id }} + name: 'built' + + - name: 'Alert if spoofed' + id: spoof + if: ${{ steps.dl.outputs.success == 'true' }} + run: | + echo 'body<> $GITHUB_ENV + echo '' >> $GITHUB_ENV + echo '## :x: DANGER :x:' >> $GITHUB_ENV + echo 'This pull request has modified workflows that created output. Close this now.' >> $GITHUB_ENV + echo '' >> $GITHUB_ENV + echo 'EOF' >> $GITHUB_ENV + + - name: "Comment on PR" + id: comment-diff + uses: carpentries/actions/comment-diff@main + with: + pr: ${{ env.NR }} + body: ${{ env.body }} + diff --git a/.github/workflows/pr-post-remove-branch.yaml b/.github/workflows/pr-post-remove-branch.yaml new file mode 100755 index 0000000..62c2e98 --- /dev/null +++ b/.github/workflows/pr-post-remove-branch.yaml @@ -0,0 +1,32 @@ +name: "Bot: Remove Temporary PR Branch" + +on: + workflow_run: + workflows: ["Bot: Send Close Pull Request Signal"] + types: + - completed + +jobs: + delete: + name: "Delete branch from Pull Request" + runs-on: ubuntu-latest + if: > + github.event.workflow_run.event == 'pull_request' && + github.event.workflow_run.conclusion == 'success' + permissions: + contents: write + steps: + - name: 'Download artifact' + uses: carpentries/actions/download-workflow-artifact@main + with: + run: ${{ github.event.workflow_run.id }} + name: pr + - name: "Get PR Number" + id: get-pr + run: | + unzip pr.zip + echo "NUM=$(<./NUM)" >> $GITHUB_OUTPUT + - name: 'Remove branch' + uses: carpentries/actions/remove-branch@main + with: + pr: ${{ steps.get-pr.outputs.NUM }} diff --git a/.github/workflows/pr-preflight.yaml b/.github/workflows/pr-preflight.yaml new file mode 100755 index 0000000..d0d7420 --- /dev/null +++ b/.github/workflows/pr-preflight.yaml @@ -0,0 +1,39 @@ +name: "Pull Request Preflight Check" + +on: + pull_request_target: + branches: + ["main"] + types: + ["opened", "synchronize", "reopened"] + +jobs: + test-pr: + name: "Test if pull request is valid" + if: ${{ github.event.action != 'closed' }} + runs-on: ubuntu-latest + outputs: + is_valid: ${{ steps.check-pr.outputs.VALID }} + permissions: + pull-requests: write + steps: + - name: "Get Invalid Hashes File" + id: hash + run: | + echo "json<> $GITHUB_OUTPUT + - name: "Check PR" + id: check-pr + uses: carpentries/actions/check-valid-pr@main + with: + pr: ${{ github.event.number }} + invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} + fail_on_error: true + - name: "Comment result of validation" + id: comment-diff + if: ${{ always() }} + uses: carpentries/actions/comment-diff@main + with: + pr: ${{ github.event.number }} + body: ${{ steps.check-pr.outputs.MSG }} diff --git a/.github/workflows/pr-receive.yaml b/.github/workflows/pr-receive.yaml new file mode 100755 index 0000000..371ef54 --- /dev/null +++ b/.github/workflows/pr-receive.yaml @@ -0,0 +1,131 @@ +name: "Receive Pull Request" + +on: + pull_request: + types: + [opened, synchronize, reopened] + +concurrency: + group: ${{ github.ref }} + cancel-in-progress: true + +jobs: + test-pr: + name: "Record PR number" + if: ${{ github.event.action != 'closed' }} + runs-on: ubuntu-latest + outputs: + is_valid: ${{ steps.check-pr.outputs.VALID }} + steps: + - name: "Record PR number" + id: record + if: ${{ always() }} + run: | + echo ${{ github.event.number }} > ${{ github.workspace }}/NR # 2022-03-02: artifact name fixed to be NR + - name: "Upload PR number" + id: upload + if: ${{ always() }} + uses: actions/upload-artifact@v3 + with: + name: pr + path: ${{ github.workspace }}/NR + - name: "Get Invalid Hashes File" + id: hash + run: | + echo "json<> $GITHUB_OUTPUT + - name: "echo output" + run: | + echo "${{ steps.hash.outputs.json }}" + - name: "Check PR" + id: check-pr + uses: carpentries/actions/check-valid-pr@main + with: + pr: ${{ github.event.number }} + invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} + + build-md-source: + name: "Build markdown source files if valid" + needs: test-pr + runs-on: ubuntu-latest + if: ${{ needs.test-pr.outputs.is_valid == 'true' }} + env: + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: ~/.local/share/renv/ + CHIVE: ${{ github.workspace }}/site/chive + PR: ${{ github.workspace }}/site/pr + MD: ${{ github.workspace }}/site/built + steps: + - name: "Check Out Main Branch" + uses: actions/checkout@v3 + + - name: "Check Out Staging Branch" + uses: actions/checkout@v3 + with: + ref: md-outputs + path: ${{ env.MD }} + + - name: "Set up R" + uses: r-lib/actions/setup-r@v2 + with: + use-public-rspm: true + install-r: false + + - name: "Set up Pandoc" + uses: r-lib/actions/setup-pandoc@v2 + + - name: "Setup Lesson Engine" + uses: carpentries/actions/setup-sandpaper@main + with: + cache-version: ${{ secrets.CACHE_VERSION }} + + - name: "Setup Package Cache" + uses: carpentries/actions/setup-lesson-deps@main + with: + cache-version: ${{ secrets.CACHE_VERSION }} + + - name: "Validate and Build Markdown" + id: build-site + run: | + sandpaper::package_cache_trigger(TRUE) + sandpaper::validate_lesson(path = '${{ github.workspace }}') + sandpaper:::build_markdown(path = '${{ github.workspace }}', quiet = FALSE) + shell: Rscript {0} + + - name: "Generate Artifacts" + id: generate-artifacts + run: | + sandpaper:::ci_bundle_pr_artifacts( + repo = '${{ github.repository }}', + pr_number = '${{ github.event.number }}', + path_md = '${{ env.MD }}', + path_pr = '${{ env.PR }}', + path_archive = '${{ env.CHIVE }}', + branch = 'md-outputs' + ) + shell: Rscript {0} + + - name: "Upload PR" + uses: actions/upload-artifact@v3 + with: + name: pr + path: ${{ env.PR }} + + - name: "Upload Diff" + uses: actions/upload-artifact@v3 + with: + name: diff + path: ${{ env.CHIVE }} + retention-days: 1 + + - name: "Upload Build" + uses: actions/upload-artifact@v3 + with: + name: built + path: ${{ env.MD }} + retention-days: 1 + + - name: "Teardown" + run: sandpaper::reset_site() + shell: Rscript {0} diff --git a/.github/workflows/sandpaper-main.yaml b/.github/workflows/sandpaper-main.yaml new file mode 100755 index 0000000..e17707a --- /dev/null +++ b/.github/workflows/sandpaper-main.yaml @@ -0,0 +1,61 @@ +name: "01 Build and Deploy Site" + +on: + push: + branches: + - main + - master + schedule: + - cron: '0 0 * * 2' + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build?' + required: true + default: 'Maintainer (via GitHub)' + reset: + description: 'Reset cached markdown files' + required: false + default: false + type: boolean +jobs: + full-build: + name: "Build Full Site" + runs-on: ubuntu-latest + permissions: + checks: write + contents: write + pages: write + env: + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: ~/.local/share/renv/ + steps: + + - name: "Checkout Lesson" + uses: actions/checkout@v3 + + - name: "Set up R" + uses: r-lib/actions/setup-r@v2 + with: + use-public-rspm: true + install-r: false + + - name: "Set up Pandoc" + uses: r-lib/actions/setup-pandoc@v2 + + - name: "Setup Lesson Engine" + uses: carpentries/actions/setup-sandpaper@main + with: + cache-version: ${{ secrets.CACHE_VERSION }} + + - name: "Setup Package Cache" + uses: carpentries/actions/setup-lesson-deps@main + with: + cache-version: ${{ secrets.CACHE_VERSION }} + + - name: "Deploy Site" + run: | + reset <- "${{ github.event.inputs.reset }}" == "true" + sandpaper::package_cache_trigger(TRUE) + sandpaper:::ci_deploy(reset = reset) + shell: Rscript {0} diff --git a/.github/workflows/sandpaper-version.txt b/.github/workflows/sandpaper-version.txt new file mode 100644 index 0000000..54d1a4f --- /dev/null +++ b/.github/workflows/sandpaper-version.txt @@ -0,0 +1 @@ +0.13.0 diff --git a/.github/workflows/update-cache.yaml b/.github/workflows/update-cache.yaml new file mode 100755 index 0000000..676d742 --- /dev/null +++ b/.github/workflows/update-cache.yaml @@ -0,0 +1,125 @@ +name: "03 Maintain: Update Package Cache" + +on: + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build (enter github username to tag yourself)?' + required: true + default: 'monthly run' + schedule: + # Run every tuesday + - cron: '0 0 * * 2' + +jobs: + preflight: + name: "Preflight Check" + runs-on: ubuntu-latest + outputs: + ok: ${{ steps.check.outputs.ok }} + steps: + - id: check + run: | + if [[ ${{ github.event_name }} == 'workflow_dispatch' ]]; then + echo "ok=true" >> $GITHUB_OUTPUT + echo "Running on request" + # using single brackets here to avoid 08 being interpreted as octal + # https://github.com/carpentries/sandpaper/issues/250 + elif [ `date +%d` -le 7 ]; then + # If the Tuesday lands in the first week of the month, run it + echo "ok=true" >> $GITHUB_OUTPUT + echo "Running on schedule" + else + echo "ok=false" >> $GITHUB_OUTPUT + echo "Not Running Today" + fi + + check_renv: + name: "Check if We Need {renv}" + runs-on: ubuntu-latest + needs: preflight + if: ${{ needs.preflight.outputs.ok == 'true'}} + outputs: + needed: ${{ steps.renv.outputs.exists }} + steps: + - name: "Checkout Lesson" + uses: actions/checkout@v3 + - id: renv + run: | + if [[ -d renv ]]; then + echo "exists=true" >> $GITHUB_OUTPUT + fi + + check_token: + name: "Check SANDPAPER_WORKFLOW token" + runs-on: ubuntu-latest + needs: check_renv + if: ${{ needs.check_renv.outputs.needed == 'true' }} + outputs: + workflow: ${{ steps.validate.outputs.wf }} + repo: ${{ steps.validate.outputs.repo }} + steps: + - name: "validate token" + id: validate + uses: carpentries/actions/check-valid-credentials@main + with: + token: ${{ secrets.SANDPAPER_WORKFLOW }} + + update_cache: + name: "Update Package Cache" + needs: check_token + if: ${{ needs.check_token.outputs.repo== 'true' }} + runs-on: ubuntu-latest + env: + GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} + RENV_PATHS_ROOT: ~/.local/share/renv/ + steps: + + - name: "Checkout Lesson" + uses: actions/checkout@v3 + + - name: "Set up R" + uses: r-lib/actions/setup-r@v2 + with: + use-public-rspm: true + install-r: false + + - name: "Update {renv} deps and determine if a PR is needed" + id: update + uses: carpentries/actions/update-lockfile@main + with: + cache-version: ${{ secrets.CACHE_VERSION }} + + - name: Create Pull Request + id: cpr + if: ${{ steps.update.outputs.n > 0 }} + uses: carpentries/create-pull-request@main + with: + token: ${{ secrets.SANDPAPER_WORKFLOW }} + delete-branch: true + branch: "update/packages" + commit-message: "[actions] update ${{ steps.update.outputs.n }} packages" + title: "Update ${{ steps.update.outputs.n }} packages" + body: | + :robot: This is an automated build + + This will update ${{ steps.update.outputs.n }} packages in your lesson with the following versions: + + ``` + ${{ steps.update.outputs.report }} + ``` + + :stopwatch: In a few minutes, a comment will appear that will show you how the output has changed based on these updates. + + If you want to inspect these changes locally, you can use the following code to check out a new branch: + + ```bash + git fetch origin update/packages + git checkout update/packages + ``` + + - Auto-generated by [create-pull-request][1] on ${{ steps.update.outputs.date }} + + [1]: https://github.com/carpentries/create-pull-request/tree/main + labels: "type: package cache" + draft: false diff --git a/.github/workflows/update-workflows.yaml b/.github/workflows/update-workflows.yaml new file mode 100755 index 0000000..288bcd1 --- /dev/null +++ b/.github/workflows/update-workflows.yaml @@ -0,0 +1,66 @@ +name: "02 Maintain: Update Workflow Files" + +on: + workflow_dispatch: + inputs: + name: + description: 'Who triggered this build (enter github username to tag yourself)?' + required: true + default: 'weekly run' + clean: + description: 'Workflow files/file extensions to clean (no wildcards, enter "" for none)' + required: false + default: '.yaml' + schedule: + # Run every Tuesday + - cron: '0 0 * * 2' + +jobs: + check_token: + name: "Check SANDPAPER_WORKFLOW token" + runs-on: ubuntu-latest + outputs: + workflow: ${{ steps.validate.outputs.wf }} + repo: ${{ steps.validate.outputs.repo }} + steps: + - name: "validate token" + id: validate + uses: carpentries/actions/check-valid-credentials@main + with: + token: ${{ secrets.SANDPAPER_WORKFLOW }} + + update_workflow: + name: "Update Workflow" + runs-on: ubuntu-latest + needs: check_token + if: ${{ needs.check_token.outputs.workflow == 'true' }} + steps: + - name: "Checkout Repository" + uses: actions/checkout@v3 + + - name: Update Workflows + id: update + uses: carpentries/actions/update-workflows@main + with: + clean: ${{ github.event.inputs.clean }} + + - name: Create Pull Request + id: cpr + if: "${{ steps.update.outputs.new }}" + uses: carpentries/create-pull-request@main + with: + token: ${{ secrets.SANDPAPER_WORKFLOW }} + delete-branch: true + branch: "update/workflows" + commit-message: "[actions] update sandpaper workflow to version ${{ steps.update.outputs.new }}" + title: "Update Workflows to Version ${{ steps.update.outputs.new }}" + body: | + :robot: This is an automated build + + Update Workflows from sandpaper version ${{ steps.update.outputs.old }} -> ${{ steps.update.outputs.new }} + + - Auto-generated by [create-pull-request][1] on ${{ steps.update.outputs.date }} + + [1]: https://github.com/carpentries/create-pull-request/tree/main + labels: "type: template and tools" + draft: false diff --git a/.github/workflows/workbench-beta-phase.yml b/.github/workflows/workbench-beta-phase.yml new file mode 100644 index 0000000..2faa25d --- /dev/null +++ b/.github/workflows/workbench-beta-phase.yml @@ -0,0 +1,60 @@ +name: "Deploy to AWS" + +on: + workflow_run: + workflows: ["01 Build and Deploy Site"] + types: + - completed + workflow_dispatch: + +jobs: + preflight: + name: "Preflight Check" + runs-on: ubuntu-latest + outputs: + ok: ${{ steps.check.outputs.ok }} + folder: ${{ steps.check.outputs.folder }} + steps: + - id: check + run: | + if [[ -z "${{ secrets.DISTRIBUTION }}" || -z "${{ secrets.AWS_ACCESS_KEY_ID }}" || -z "${{ secrets.AWS_SECRET_ACCESS_KEY }}" ]]; then + echo ":information_source: No site configured" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo 'To deploy the preview on AWS, you need the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `DISTRIBUTION` secrets set up' >> $GITHUB_STEP_SUMMARY + else + echo "::set-output name=folder::"$(sed -E 's^.+/(.+)^\1^' <<< ${{ github.repository }}) + echo "::set-output name=ok::true" + fi + + full-build: + name: "Deploy to AWS" + needs: [preflight] + if: ${{ needs.preflight.outputs.ok }} + runs-on: ubuntu-latest + steps: + + - name: "Checkout site folder" + uses: actions/checkout@v3 + with: + ref: 'gh-pages' + path: 'source' + + - name: "Deploy to Bucket" + uses: jakejarvis/s3-sync-action@v0.5.1 + with: + args: --acl public-read --follow-symlinks --delete --exclude '.git/*' + env: + AWS_S3_BUCKET: preview.carpentries.org + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + SOURCE_DIR: 'source' + DEST_DIR: ${{ needs.preflight.outputs.folder }} + + - name: "Invalidate CloudFront" + uses: chetan/invalidate-cloudfront-action@master + env: + PATHS: /* + AWS_REGION: 'us-east-1' + DISTRIBUTION: ${{ secrets.DISTRIBUTION }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b8ab706 --- /dev/null +++ b/.gitignore @@ -0,0 +1,55 @@ +# sandpaper files +episodes/*html +site/* +!site/README.md + +# History files +.Rhistory +.Rapp.history +# Session Data files +.RData +# User-specific files +.Ruserdata +# Example code in package build process +*-Ex.R +# Output files from R CMD build +/*.tar.gz +# Output files from R CMD check +/*.Rcheck/ +# RStudio files +.Rproj.user/ +# produced vignettes +vignettes/*.html +vignettes/*.pdf +# OAuth2 token, see https://github.com/hadley/httr/releases/tag/v0.3 +.httr-oauth +# knitr and R markdown default cache directories +*_cache/ +/cache/ +# Temporary files created by R markdown +*.utf8.md +*.knit.md +# R Environment Variables +.Renviron +# pkgdown site +docs/ +# translation temp files +po/*~ +# renv detritus +renv/sandbox/ +*.pyc +*~ +.DS_Store +.ipynb_checkpoints +.sass-cache +.jekyll-cache/ +.jekyll-metadata +__pycache__ +_site +.Rproj.user +.bundle/ +.vendor/ +vendor/ +.docker-vendor/ +Gemfile.lock +.*history diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..f19b804 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,13 @@ +--- +title: "Contributor Code of Conduct" +--- + +As contributors and maintainers of this project, +we pledge to follow the [The Carpentries Code of Conduct][coc]. + +Instances of abusive, harassing, or otherwise unacceptable behavior +may be reported by following our [reporting guidelines][coc-reporting]. + + +[coc-reporting]: https://docs.carpentries.org/topic_folders/policies/incident-reporting.html +[coc]: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..6c2b81c --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,123 @@ +## Contributing + +[The Carpentries][cp-site] ([Software Carpentry][swc-site], [Data +Carpentry][dc-site], and [Library Carpentry][lc-site]) are open source +projects, and we welcome contributions of all kinds: new lessons, fixes to +existing material, bug reports, and reviews of proposed changes are all +welcome. + +### Contributor Agreement + +By contributing, you agree that we may redistribute your work under [our +license](LICENSE.md). In exchange, we will address your issues and/or assess +your change proposal as promptly as we can, and help you become a member of our +community. Everyone involved in [The Carpentries][cp-site] agrees to abide by +our [code of conduct](CODE_OF_CONDUCT.md). + +### How to Contribute + +The easiest way to get started is to file an issue to tell us about a spelling +mistake, some awkward wording, or a factual error. This is a good way to +introduce yourself and to meet some of our community members. + +1. If you do not have a [GitHub][github] account, you can [send us comments by + email][contact]. However, we will be able to respond more quickly if you use + one of the other methods described below. + +2. If you have a [GitHub][github] account, or are willing to [create + one][github-join], but do not know how to use Git, you can report problems + or suggest improvements by [creating an issue][repo-issues]. This allows us + to assign the item to someone and to respond to it in a threaded discussion. + +3. If you are comfortable with Git, and would like to add or change material, + you can submit a pull request (PR). Instructions for doing this are + [included below](#using-github). For inspiration about changes that need to + be made, check out the [list of open issues][issues] across the Carpentries. + +Note: if you want to build the website locally, please refer to [The Workbench +documentation][template-doc]. + +### Where to Contribute + +1. If you wish to change this lesson, add issues and pull requests here. +2. If you wish to change the template used for workshop websites, please refer + to [The Workbench documentation][template-doc]. + + +### What to Contribute + +There are many ways to contribute, from writing new exercises and improving +existing ones to updating or filling in the documentation and submitting [bug +reports][issues] about things that do not work, are not clear, or are missing. +If you are looking for ideas, please see [the list of issues for this +repository][repo-issues], or the issues for [Data Carpentry][dc-issues], +[Library Carpentry][lc-issues], and [Software Carpentry][swc-issues] projects. + +Comments on issues and reviews of pull requests are just as welcome: we are +smarter together than we are on our own. **Reviews from novices and newcomers +are particularly valuable**: it's easy for people who have been using these +lessons for a while to forget how impenetrable some of this material can be, so +fresh eyes are always welcome. + +### What *Not* to Contribute + +Our lessons already contain more material than we can cover in a typical +workshop, so we are usually *not* looking for more concepts or tools to add to +them. As a rule, if you want to introduce a new idea, you must (a) estimate how +long it will take to teach and (b) explain what you would take out to make room +for it. The first encourages contributors to be honest about requirements; the +second, to think hard about priorities. + +We are also not looking for exercises or other material that only run on one +platform. Our workshops typically contain a mixture of Windows, macOS, and +Linux users; in order to be usable, our lessons must run equally well on all +three. + +### Using GitHub + +If you choose to contribute via GitHub, you may want to look at [How to +Contribute to an Open Source Project on GitHub][how-contribute]. In brief, we +use [GitHub flow][github-flow] to manage changes: + +1. Create a new branch in your desktop copy of this repository for each + significant change. +2. Commit the change in that branch. +3. Push that branch to your fork of this repository on GitHub. +4. Submit a pull request from that branch to the [upstream repository][repo]. +5. If you receive feedback, make changes on your desktop and push to your + branch on GitHub: the pull request will update automatically. + +NB: The published copy of the lesson is usually in the `main` branch. + +Each lesson has a team of maintainers who review issues and pull requests or +encourage others to do so. The maintainers are community volunteers, and have +final say over what gets merged into the lesson. + +### Other Resources + +The Carpentries is a global organisation with volunteers and learners all over +the world. We share values of inclusivity and a passion for sharing knowledge, +teaching and learning. There are several ways to connect with The Carpentries +community listed at including via social +media, slack, newsletters, and email lists. You can also [reach us by +email][contact]. + +[repo]: https://example.com/FIXME +[repo-issues]: https://example.com/FIXME/issues +[contact]: mailto:team@carpentries.org +[cp-site]: https://carpentries.org/ +[dc-issues]: https://github.com/issues?q=user%3Adatacarpentry +[dc-lessons]: https://datacarpentry.org/lessons/ +[dc-site]: https://datacarpentry.org/ +[discuss-list]: https://carpentries.topicbox.com/groups/discuss +[github]: https://github.com +[github-flow]: https://guides.github.com/introduction/flow/ +[github-join]: https://github.com/join +[how-contribute]: https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github +[issues]: https://carpentries.org/help-wanted-issues/ +[lc-issues]: https://github.com/issues?q=user%3ALibraryCarpentry +[swc-issues]: https://github.com/issues?q=user%3Aswcarpentry +[swc-lessons]: https://software-carpentry.org/lessons/ +[swc-site]: https://software-carpentry.org/ +[lc-site]: https://librarycarpentry.org/ +[template-doc]: https://carpentries.github.io/workbench/ diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 0000000..7632871 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,79 @@ +--- +title: "Licenses" +--- + +## Instructional Material + +All Carpentries (Software Carpentry, Data Carpentry, and Library Carpentry) +instructional material is made available under the [Creative Commons +Attribution license][cc-by-human]. The following is a human-readable summary of +(and not a substitute for) the [full legal text of the CC BY 4.0 +license][cc-by-legal]. + +You are free: + +- to **Share**---copy and redistribute the material in any medium or format +- to **Adapt**---remix, transform, and build upon the material + +for any purpose, even commercially. + +The licensor cannot revoke these freedoms as long as you follow the license +terms. + +Under the following terms: + +- **Attribution**---You must give appropriate credit (mentioning that your work + is derived from work that is Copyright (c) The Carpentries and, where + practical, linking to ), provide a [link to the + license][cc-by-human], and indicate if changes were made. You may do so in + any reasonable manner, but not in any way that suggests the licensor endorses + you or your use. + +- **No additional restrictions**---You may not apply legal terms or + technological measures that legally restrict others from doing anything the + license permits. With the understanding that: + +Notices: + +* You do not have to comply with the license for elements of the material in + the public domain or where your use is permitted by an applicable exception + or limitation. +* No warranties are given. The license may not give you all of the permissions + necessary for your intended use. For example, other rights such as publicity, + privacy, or moral rights may limit how you use the material. + +## Software + +Except where otherwise noted, the example programs and other software provided +by The Carpentries are made available under the [OSI][osi]-approved [MIT +license][mit-license]. + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +## Trademark + +"The Carpentries", "Software Carpentry", "Data Carpentry", and "Library +Carpentry" and their respective logos are registered trademarks of [Community +Initiatives][ci]. + +[cc-by-human]: https://creativecommons.org/licenses/by/4.0/ +[cc-by-legal]: https://creativecommons.org/licenses/by/4.0/legalcode +[mit-license]: https://opensource.org/licenses/mit-license.html +[ci]: https://communityin.org/ +[osi]: https://opensource.org diff --git a/README.md b/README.md index 98144be..5d17ec0 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,10 @@ -[![DOI](https://zenodo.org/badge/124151544.svg)](https://zenodo.org/badge/latestdoi/124151544) [![Create a Slack Account with us](https://img.shields.io/badge/Create_Slack_Account-The_Carpentries-071159.svg)](https://slack-invite.carpentries.org/) - [![Slack Status](https://img.shields.io/badge/Slack_Channel-dc--geospatial-E01563.svg)](https://carpentries.slack.com/messages/C9ME7G5RD) +> **ATTENTION** This is an experimental test of [The Carpentries Workbench](https://carpentries.github.io/workbench) lesson infrastructure. +> It was automatically converted from the source lesson via [the lesson transition script](https://github.com/carpentries/lesson-transition/). +> +> If anything seems off, please contact Zhian Kamvar [zkamvar@carpentries.org](mailto:zkamvar@carpentries.org) +[![DOI](https://zenodo.org/badge/124151544.svg)](https://zenodo.org/badge/latestdoi/124151544) [![Create a Slack Account with us](https://img.shields.io/badge/Create_Slack_Account-The_Carpentries-071159.svg)](https://slack-invite.carpentries.org/) +[![Slack Status](https://img.shields.io/badge/Slack_Channel-dc--geospatial-E01563.svg)](https://carpentries.slack.com/messages/C9ME7G5RD) # Geospatial Workshop @@ -8,7 +12,7 @@ Overview of the Geospatial workshop. ## Code of Conduct -All participants should agree to abide by the [Data Carpentry Code of Conduct](http://www.datacarpentry.org/code-of-conduct/). +All participants should agree to abide by the [Data Carpentry Code of Conduct](https://www.datacarpentry.org/code-of-conduct/). ## Authors @@ -19,3 +23,5 @@ The Geospatial workshop overview is authored and maintained by the Carpentry com Please cite as: Geospatial Workshop. June 2018. + + diff --git a/_extras/data.md b/_extras/data.md deleted file mode 100644 index 8a1bfbd..0000000 --- a/_extras/data.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -layout: page -title: "Workshop data" ---- - -## NEON Spatio-temporal Teaching Data Subset - -Available on FigShare: -[NEON Spatio-temporal Teaching Data Subset](https://figshare.com/articles/Spatio_temporal_Series_Teaching_Data_Subsets/2009586). - -**CITATION:** Data Skills Teaching Data Subsets, NEON; Wasser, Leah; Jones, Megan A. (2016): -NEON Spatio-temporal Teaching Data Subset. figshare. Fileset. https://doi.org/10.6084/m9.figshare.2009586.v10 - -The data and lessons in this workshop were originally developed through a hackathon funded by the [National Ecological Observatory Network (NEON)](https://www.neonscience.org/) - an NSF funded observatory in Boulder, Colorado - in collaboration with Data Carpentry, SESYNC and CYVERSE. NEON is collecting data for 30 years to help scientists understand -how our aquatic and terrestrial ecosystems are changing. The data used in these lessons cover two NEON field sites: -* Harvard Forest (HARV) - Massachusetts, USA - [fieldsite description](https://www.neonscience.org/field-sites/field-sites-map/HARV) -* San Joaquin Experimental Range (SJER) - California, USA - [fieldsite description](https://www.neonscience.org/field-sites/field-sites-map/SJER) - -You can download all of the data used in this workshop by clicking -[this download link](https://ndownloader.figshare.com/articles/2009586/versions/10). -Clicking the download link will automatically download all of the files to your default download directory as a single compressed -(`.zip`) file. To expand this file, double click the folder icon in your file navigator application (for Macs, this is the Finder -application). - -These data files represent teaching version of the data, with sufficient complexity to teach many aspects of data analysis and -management, but with many complexities removed to allow students to focus on the core ideas and skills being taught. - - | Dataset | File name | Description | - | ---- | ------| ---- | - | Site layout shapefiles | NEON-DS-Site-Layout-Files.zip | A set of shapefiles for the NEON's Harvard Forest field site and US and (some) state boundary layers. | - | Meteorological data | NEON-DS-Met-Time-Series.zip | Precipitation, temperature and other variables collected from a flux tower at the NEON Harvard Forest site - | Airborne remote sensing data | NEON-DS-Airborne-RemoteSensing.zip | LiDAR data collected by the NEON Airborne Observation Platform (AOP) and processed at NEON including a canopy height model, digital elevation model and digital surface model for NEON's Harvard Forest and San Joaquin Experimental Range field sites. | - | Landstat 7 NDVI raster data | NEON-DS-Landsat-NDVI.zip | 2011 NDVI data product derived from Landsat 7 and processed by USGS cropped to NEON's Harvard Forest and San Joaquin Experimental Range field sites | - -### About the Site Layout Shapefiles -These vector data provide information on the site characterization and infrastructure at the -[National Ecological Observatory Network's](https://www.neonscience.org/) -[Harvard Forest](https://www.neonscience.org/field-sites/field-sites-map/HARV) field site. -The Harvard Forest shapefiles are from the [Harvard Forest GIS & Map](http://harvardforest.fas.harvard.edu/gis-maps/) archives. -US Country and State Boundary layers are from the [US Census Bureau](https://www.census.gov/geo/maps-data/data/tiger-cart-boundary.html). - -### About the Meteorological Data -The data used in this lesson were collected at the [National Ecological Observatory Network's](https://www.neonscience.org/) -[Harvard Forest](https://www.neonscience.org/field-sites/field-sites-map/HARV). These data are proxy data for what will be available -for 30 years on the [NEON data portal](http://data.neonscience.org/home) for the Harvard Forest and other field sites located across -the United States. - -### About the Airborne Remote Sensing Data -This data was collected at the [National Ecological Observatory Network's](https://www.neonscience.org/) -[Harvard Forest](https://www.neonscience.org/field-sites/field-sites-map/HARV) and -[San Joaquin Experimental Range](https://www.neonscience.org/field-sites/field-sites-map/SJER) field sites. - -Data are collected during peak greenness at each field site and are processed to provide useful data products to the community. The following NEON data products are used in these lessons: - -Data Created from Discrete Return (point clouds) Lidar Data: - -* DSM (Digital Surface Model; full mosaic) -* DTM (Digital Terrain Model; full mosaic) -* CHM (Canopy Height Model; full mosaic; Harvard Forest site only) -and -* RGB imagery (Harvard Forest site only) derived from the RGB Camera - -Additional information about airborne remote sensing data, including other data types for these and other sites can -be found on [NEON’s Airborne Data](https://www.neonscience.org/data-collection/airborne-remote-sensing) page. - -### About the Landstat 7 NDVI Raster Data -The imagery data used to create this raster teaching data subset were collected over the -[National Ecological Observatory Network's](https://www.neonscience.org/) -[Harvard Forest](https://www.neonscience.org/field-sites/field-sites-map/HARV) -and [San Joaquin Experimental Range](https://www.neonscience.org/field-sites/field-sites-map/SJER) field sites. -The imagery was created by the U.S. Geological Survey (USGS) using a -[multispectral scanner](http://eros.usgs.gov/#/Find_Data/Products_and_Data_Available/MSS) on a -[Landsat Satellite](http://landsat.usgs.gov/). The data files are Geographic Tagged Image-File Format (GeoTIFF). diff --git a/config.yaml b/config.yaml new file mode 100644 index 0000000..c1ba5ba --- /dev/null +++ b/config.yaml @@ -0,0 +1,83 @@ +#------------------------------------------------------------ +# Values for this lesson. +#------------------------------------------------------------ + +# Which carpentry is this (swc, dc, lc, or cp)? +# swc: Software Carpentry +# dc: Data Carpentry +# lc: Library Carpentry +# cp: Carpentries (to use for instructor training for instance) +# incubator: The Carpentries Incubator +carpentry: 'dc' + +# Overall title for pages. +title: 'Geospatial Workshop Overview' + +# Date the lesson was created (YYYY-MM-DD, this is empty by default) +created: ~ # FIXME + +# Comma-separated list of keywords for the lesson +keywords: 'software, data, lesson, The Carpentries' # FIXME + +# Life cycle stage of the lesson +# possible values: pre-alpha, alpha, beta, stable +life_cycle: 'stable' + +# License of the lesson +license: 'CC-BY 4.0' + +# Link to the source repository for this lesson +source: 'https://github.com/fishtree-attempt/geospatial-workshop/' + +# Default branch of your lesson +branch: 'main' + +# Who to contact if there are any issues +contact: 'team@carpentries.org' + +# Navigation ------------------------------------------------ +# +# Use the following menu items to specify the order of +# individual pages in each dropdown section. Leave blank to +# include all pages in the folder. +# +# Example ------------- +# +# episodes: +# - introduction.md +# - first-steps.md +# +# learners: +# - setup.md +# +# instructors: +# - instructor-notes.md +# +# profiles: +# - one-learner.md +# - another-learner.md + +# Order of episodes in your lesson +episodes: +- introduction.Rmd + +# Information for Learners +learners: + +# Information for Instructors +instructors: + +# Learner Profiles +profiles: + +# Customisation --------------------------------------------- +# +# This space below is where custom yaml items (e.g. pinning +# sandpaper and varnish versions) should live + + +url: 'https://preview.carpentries.org/geospatial-workshop' +analytics: 'carpentries' +lang: 'en' +workbench-beta: true +overview: true diff --git a/index.md b/index.md index a781ef5..c0933c9 100644 --- a/index.md +++ b/index.md @@ -1,68 +1,79 @@ --- -layout: lesson -root: . +site: sandpaper::sandpaper_site --- -Data Carpentry's aim is to teach researchers basic concepts, skills, and tools for working with data so that they can get more done in less time, and with less pain. +> **ATTENTION** This is an experimental test of [The Carpentries Workbench](https://carpentries.github.io/workbench) lesson infrastructure. +> It was automatically converted from the source lesson via [the lesson transition script](https://github.com/carpentries/lesson-transition/). +> +> If anything seems off, please contact Zhian Kamvar [zkamvar@carpentries.org](mailto:zkamvar@carpentries.org) +Data Carpentry's aim is to teach researchers basic concepts, skills, and tools for working with data so that they can get more done in less time, and with less pain. Interested in teaching these materials? We have an [onboarding video](https://www.youtube.com/watch?v=Qtnb_eeHt7E) available to prepare Instructors to teach these lessons. After watching this video, please contact [team@carpentries.org](mailto:team@carpentries.org) so that we can record your status as an onboarded Instructor. Instructors who have completed onboarding will be given priority status for teaching at centrally-organized Data Carpentry Geospatial workshops. +:::::::::::::::::::::::::::::::::::::::::: prereq -> ## Getting Started -> -> Data Carpentry’s teaching is hands-on, so participants are encouraged to use -> their own computers to ensure the proper setup of tools for an efficient -> workflow. To most effectively use these materials, please make sure to download -> the data and install everything before working through this lesson. -> -> This workshop assumes no prior experience with the tools covered in the workshop. However, learners with prior -> experience working with geospatial data may be able to skip the -> [Geospatial Project Organization and Management](https://www.datacarpentry.org/organization-geospatial/) lesson. -> Similarly, learners who have prior experience with the `R` programming language may wish to skip the -> [Introduction to R for Geospatial Data](https://www.datacarpentry.org/r-intro-geospatial/) lesson. -> -> To get started, follow the directions in the [Setup](setup.html) tab to -> get access to the required software and data for this workshop. -{: .prereq} - -> ## Data -> -> The data and lessons in this workshop were originally developed through a hackathon funded by the -> [National Ecological Observatory Network (NEON)](https://www.neonscience.org/) - an NSF funded observatory in Boulder, Colorado - in -> collaboration with Data Carpentry, SESYNC and CYVERSE. NEON is collecting data for 30 years to help scientists understand -> how aquatic and terrestrial ecosystems are changing. The data used in these lessons cover two NEON field sites: -> * Harvard Forest (HARV) - Massachusetts, USA - [fieldsite description](https://www.neonscience.org/field-sites/field-sites-map/HARV) -> * San Joaquin Experimental Range (SJER) - California, USA - [fieldsite description](https://www.neonscience.org/field-sites/field-sites-map/SJER) -> -> There are four data sets included, all of which are available -> [on Figshare](https://figshare.com/articles/Spatio_temporal_Series_Teaching_Data_Subsets/2009586) -> under a CC-BY license. You can download all of the data used in this workshop by clicking -> [this download link](https://ndownloader.figshare.com/articles/2009586/versions/10). -> Clicking the download link will download all of the files as a single compressed -> (`.zip`) file. To expand this file, double click the folder icon in your file navigator application (for Macs, this is the Finder -> application). -> -> These data files represent the teaching version of the data, with sufficient complexity to teach many aspects of data analysis and -> management, but with many complexities removed to allow students to focus on the core ideas and skills being taught. -> -> | Dataset | File name | Description | -> | ---- | ------| ---- | -> | Site layout shapefiles | NEON-DS-Site-Layout-Files.zip | A set of shapefiles for the NEON's Harvard Forest field site and US and (some) state boundary layers. | -> | Meteorological data | NEON-DS-Met-Time-Series.zip | Precipitation, temperature and other variables collected from a flux tower at the NEON Harvard Forest site -> | Airborne remote sensing data | NEON-DS-Airborne-RemoteSensing.zip | LiDAR data collected by the NEON Airborne Observation Platform (AOP) and processed at NEON including a canopy height model, digital elevation model and digital surface model for NEON's Harvard Forest and San Joaquin Experimental Range field sites. | -> | Landstat 7 NDVI raster data | NEON-DS-Landsat-NDVI.zip | 2011 NDVI data product derived from Landsat 7 and processed by USGS cropped to NEON's Harvard Forest and San Joaquin Experimental Range field sites | -> -> [More information on this dataset](data) -> -{: .prereq} +## Getting Started + +Data Carpentry's teaching is hands-on, so participants are encouraged to use +their own computers to ensure the proper setup of tools for an efficient +workflow. To most effectively use these materials, please make sure to download +the data and install everything before working through this lesson. + +This workshop assumes no prior experience with the tools covered in the workshop. However, learners with prior +experience working with geospatial data may be able to skip the +[Geospatial Project Organization and Management](https://www.datacarpentry.org/organization-geospatial/) lesson. +Similarly, learners who have prior experience with the `R` programming language may wish to skip the +[Introduction to R for Geospatial Data](https://www.datacarpentry.org/r-intro-geospatial/) lesson. + +To get started, follow the directions in the [Setup](learners/setup.md) tab to +get access to the required software and data for this workshop. + + +:::::::::::::::::::::::::::::::::::::::::::::::::: + +:::::::::::::::::::::::::::::::::::::::::: prereq + +## Data + +The data and lessons in this workshop were originally developed through a hackathon funded by the +[National Ecological Observatory Network (NEON)](https://www.neonscience.org/) - an NSF funded observatory in Boulder, Colorado - in +collaboration with Data Carpentry, SESYNC and CYVERSE. NEON is collecting data for 30 years to help scientists understand +how aquatic and terrestrial ecosystems are changing. The data used in these lessons cover two NEON field sites: + +- Harvard Forest (HARV) - Massachusetts, USA - [fieldsite description](https://www.neonscience.org/field-sites/field-sites-map/HARV) +- San Joaquin Experimental Range (SJER) - California, USA - [fieldsite description](https://www.neonscience.org/field-sites/field-sites-map/SJER) + +There are four data sets included, all of which are available +[on Figshare](https://figshare.com/articles/Spatio_temporal_Series_Teaching_Data_Subsets/2009586) +under a CC-BY license. You can download all of the data used in this workshop by clicking +[this download link](https://ndownloader.figshare.com/articles/2009586/versions/10). +Clicking the download link will download all of the files as a single compressed +(`.zip`) file. To expand this file, double click the folder icon in your file navigator application (for Macs, this is the Finder +application). + +These data files represent the teaching version of the data, with sufficient complexity to teach many aspects of data analysis and +management, but with many complexities removed to allow students to focus on the core ideas and skills being taught. + +| Dataset | File name | Description | +| ---------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Site layout shapefiles | NEON-DS-Site-Layout-Files.zip | A set of shapefiles for the NEON's Harvard Forest field site and US and (some) state boundary layers. | +| Meteorological data | NEON-DS-Met-Time-Series.zip | Precipitation, temperature and other variables collected from a flux tower at the NEON Harvard Forest site | +| Airborne remote sensing data | NEON-DS-Airborne-RemoteSensing.zip | LiDAR data collected by the NEON Airborne Observation Platform (AOP) and processed at NEON including a canopy height model, digital elevation model and digital surface model for NEON's Harvard Forest and San Joaquin Experimental Range field sites. | +| Landstat 7 NDVI raster data | NEON-DS-Landsat-NDVI.zip | 2011 NDVI data product derived from Landsat 7 and processed by USGS cropped to NEON's Harvard Forest and San Joaquin Experimental Range field sites | + +[More information on this dataset](instructors/data.md) + +:::::::::::::::::::::::::::::::::::::::::::::::::: # Workshop Overview -| Lesson | Overview | -| ------- | ---------- | -| [Introduction to Geospatial Concepts](http://www.datacarpentry.org/organization-geospatial/) | Understand data structures and common storage and transfer formats for spatial data. | -| [Introduction to R for Geospatial Data](http://www.datacarpentry.org/r-intro-geospatial) | Import data into R, calculate summary statistics, and create publication-quality graphics. | -| [Introduction to Geospatial Raster and Vector Data with R](http://www.datacarpentry.org/r-raster-vector-geospatial) | Open, work with, and plot vector and raster-format spatial data in R. | +| Lesson | Overview | +| ---------------------------- | ------------------------------------------------------------------------------------------ | +| [Introduction to Geospatial Concepts](https://www.datacarpentry.org/organization-geospatial/) | Understand data structures and common storage and transfer formats for spatial data. | +| [Introduction to R for Geospatial Data](https://www.datacarpentry.org/r-intro-geospatial) | Import data into R, calculate summary statistics, and create publication-quality graphics. | +| [Introduction to Geospatial Raster and Vector Data with R](https://www.datacarpentry.org/r-raster-vector-geospatial) | Open, work with, and plot vector and raster-format spatial data in R. | + + diff --git a/instructors/data.md b/instructors/data.md new file mode 100644 index 0000000..4db8f9b --- /dev/null +++ b/instructors/data.md @@ -0,0 +1,79 @@ +--- +title: Workshop data +--- + +## NEON Spatio-temporal Teaching Data Subset + +Available on FigShare: +[NEON Spatio-temporal Teaching Data Subset](https://figshare.com/articles/Spatio_temporal_Series_Teaching_Data_Subsets/2009586). + +**CITATION:** Data Skills Teaching Data Subsets, NEON; Wasser, Leah; Jones, Megan A. (2016): +NEON Spatio-temporal Teaching Data Subset. figshare. Fileset. [https://doi.org/10.6084/m9.figshare.2009586.v10](https://doi.org/10.6084/m9.figshare.2009586.v10) + +The data and lessons in this workshop were originally developed through a hackathon funded by the [National Ecological Observatory Network (NEON)](https://www.neonscience.org/) - an NSF funded observatory in Boulder, Colorado - in collaboration with Data Carpentry, SESYNC and CYVERSE. NEON is collecting data for 30 years to help scientists understand +how our aquatic and terrestrial ecosystems are changing. The data used in these lessons cover two NEON field sites: + +- Harvard Forest (HARV) - Massachusetts, USA - [fieldsite description](https://www.neonscience.org/field-sites/field-sites-map/HARV) +- San Joaquin Experimental Range (SJER) - California, USA - [fieldsite description](https://www.neonscience.org/field-sites/field-sites-map/SJER) + +You can download all of the data used in this workshop by clicking +[this download link](https://ndownloader.figshare.com/articles/2009586/versions/10). +Clicking the download link will automatically download all of the files to your default download directory as a single compressed +(`.zip`) file. To expand this file, double click the folder icon in your file navigator application (for Macs, this is the Finder +application). + +These data files represent teaching version of the data, with sufficient complexity to teach many aspects of data analysis and +management, but with many complexities removed to allow students to focus on the core ideas and skills being taught. + +| Dataset | File name | Description | +| ---------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Site layout shapefiles | NEON-DS-Site-Layout-Files.zip | A set of shapefiles for the NEON's Harvard Forest field site and US and (some) state boundary layers. | +| Meteorological data | NEON-DS-Met-Time-Series.zip | Precipitation, temperature and other variables collected from a flux tower at the NEON Harvard Forest site | +| Airborne remote sensing data | NEON-DS-Airborne-RemoteSensing.zip | LiDAR data collected by the NEON Airborne Observation Platform (AOP) and processed at NEON including a canopy height model, digital elevation model and digital surface model for NEON's Harvard Forest and San Joaquin Experimental Range field sites. | +| Landstat 7 NDVI raster data | NEON-DS-Landsat-NDVI.zip | 2011 NDVI data product derived from Landsat 7 and processed by USGS cropped to NEON's Harvard Forest and San Joaquin Experimental Range field sites | + +### About the Site Layout Shapefiles + +These vector data provide information on the site characterization and infrastructure at the +[National Ecological Observatory Network's](https://www.neonscience.org/) +[Harvard Forest](https://www.neonscience.org/field-sites/field-sites-map/HARV) field site. +The Harvard Forest shapefiles are from the [Harvard Forest GIS \& Map](https://harvardforest.fas.harvard.edu/gis-maps/) archives. +US Country and State Boundary layers are from the [US Census Bureau](https://www.census.gov/geo/maps-data/data/tiger-cart-boundary.html). + +### About the Meteorological Data + +The data used in this lesson were collected at the [National Ecological Observatory Network's](https://www.neonscience.org/) +[Harvard Forest](https://www.neonscience.org/field-sites/field-sites-map/HARV). These data are proxy data for what will be available +for 30 years on the [NEON data portal](https://data.neonscience.org/home) for the Harvard Forest and other field sites located across +the United States. + +### About the Airborne Remote Sensing Data + +This data was collected at the [National Ecological Observatory Network's](https://www.neonscience.org/) +[Harvard Forest](https://www.neonscience.org/field-sites/field-sites-map/HARV) and +[San Joaquin Experimental Range](https://www.neonscience.org/field-sites/field-sites-map/SJER) field sites. + +Data are collected during peak greenness at each field site and are processed to provide useful data products to the community. The following NEON data products are used in these lessons: + +Data Created from Discrete Return (point clouds) Lidar Data: + +- DSM (Digital Surface Model; full mosaic) +- DTM (Digital Terrain Model; full mosaic) +- CHM (Canopy Height Model; full mosaic; Harvard Forest site only) + and +- RGB imagery (Harvard Forest site only) derived from the RGB Camera + +Additional information about airborne remote sensing data, including other data types for these and other sites can +be found on [NEON's Airborne Data](https://www.neonscience.org/data-collection/airborne-remote-sensing) page. + +### About the Landstat 7 NDVI Raster Data + +The imagery data used to create this raster teaching data subset were collected over the +[National Ecological Observatory Network's](https://www.neonscience.org/) +[Harvard Forest](https://www.neonscience.org/field-sites/field-sites-map/HARV) +and [San Joaquin Experimental Range](https://www.neonscience.org/field-sites/field-sites-map/SJER) field sites. +The imagery was created by the U.S. Geological Survey (USGS) using a +[multispectral scanner](https://eros.usgs.gov/#/Find_Data/Products_and_Data_Available/MSS) on a +[Landsat Satellite](https://landsat.usgs.gov/). The data files are Geographic Tagged Image-File Format (GeoTIFF). + + diff --git a/_extras/guide.md b/instructors/instructor-notes.md similarity index 70% rename from _extras/guide.md rename to instructors/instructor-notes.md index ba80927..2c1e129 100644 --- a/_extras/guide.md +++ b/instructors/instructor-notes.md @@ -1,18 +1,18 @@ --- -layout: page -title: "Instructor Notes" +title: Instructor Notes --- + ## Instructor notes ## Workshop Structure The following table provides the structure for the Geospatial Workshop and *estimated* timings for each lesson. -| Lesson | Length | Include if: | -|---|---|---| -| [Introduction to Geospatial Concepts](https://datacarpentry.org/organization-geospatial/) ([instructor notes](https://datacarpentry.org/organization-geospatial/guide/index.html)) | **75 minutes** - estimated, will not be accurate | Learners have no experience with geospatial data. | -| [Introduction to R for Geospatial Data](https://datacarpentry.org/r-intro-geospatial/) ([instructor notes](https://datacarpentry.org/r-intro-geospatial/guide/index.html)) | **4 hours** - estimated, will not be accurate | Learners have no experience with R. | -| [Introduction to Geospatial Raster and Vector Data with R](https://datacarpentry.org/r-raster-vector-geospatial/) ([instructor notes](https://datacarpentry.org/r-raster-vector-geospatial/guide/index.html)) | **7 hours** - estimated, will not be accurate | Always. | +| Lesson | Length | Include if: | +| ------ | ---------------------------------- | ------------------------------------------------- | +| [Introduction to Geospatial Concepts](https://datacarpentry.org/organization-geospatial/) ([instructor notes](https://datacarpentry.org/organization-geospatial/guide/index.html)) | **75 minutes** - estimated, will not be accurate | Learners have no experience with geospatial data. | +| [Introduction to R for Geospatial Data](https://datacarpentry.org/r-intro-geospatial/) ([instructor notes](https://datacarpentry.org/r-intro-geospatial/guide/index.html)) | **4 hours** - estimated, will not be accurate | Learners have no experience with R. | +| [Introduction to Geospatial Raster and Vector Data with R](https://datacarpentry.org/r-raster-vector-geospatial/) ([instructor notes](https://datacarpentry.org/r-raster-vector-geospatial/guide/index.html)) | **7 hours** - estimated, will not be accurate | Always. | If the workshop audience is experienced in both R and geospatial concepts, this workshop could theoretically be taught in one day. However, note that these timings are estimates and we are relying on feedback from pilot workshop Instructors to add accurate timing information. @@ -21,10 +21,13 @@ If the workshop audience is experienced in both R and geospatial concepts, this ## Technical tips and tricks #### Installation + - For the [Introduction to Geospatial Raster and Vector Data with R](https://datacarpentry.org/r-raster-vector-geospatial/) lesson, learners MUST download the data and install the software before the workshop. The data set is large (>150 Mb) and there are numerous software packages and R libraries to install. **Remind your learners multiple times!** - - Instructions to download data and set up a machine to analyze and process geospatial data: http://datacarpentry.org/geospatial-workshop/setup.html + - Instructions to download data and set up a machine to analyze and process geospatial data: [http://datacarpentry.org/geospatial-workshop/setup.html](https://datacarpentry.org/geospatial-workshop/setup.html) - You may find it helpful to host an "install party" prior to your workshop where you and your co-instructors help learners install the necessary software for the workshop. If this is unfeasible, leave time at the beginning of your workshop to ensure that learners have necessary geospatial software (GDAL, GEOS, PROJ.4) installed prior to teaching the course. [Instructors: add information about supporting learners with installation here] ## Common problems + + diff --git a/reference.md b/learners/reference.md similarity index 51% rename from reference.md rename to learners/reference.md index 9fd7c8a..456f940 100644 --- a/reference.md +++ b/learners/reference.md @@ -1,8 +1,9 @@ --- -layout: reference -root: . +title: 'Glossary' --- ## Glossary FIXME + + diff --git a/learners/setup.md b/learners/setup.md new file mode 100644 index 0000000..bf3f448 --- /dev/null +++ b/learners/setup.md @@ -0,0 +1,363 @@ +--- +title: Setup +--- + +# Overview + +This workshop is designed to be run on your local machine. First, you will need to download the data we use in the workshop. Then, you need to set up your machine to analyze and process geospatial data. We provide instructions below to either install all components manually (option A), or to use a Docker image that provides all the software and dependencies needed (option B). + +## Data + +You can download all of the data used in this workshop by clicking +[this download link](https://ndownloader.figshare.com/articles/2009586/versions/10). The file is 218.2 MB. + +Clicking the download link will automatically download all of the files to your default download directory as a single compressed +(`.zip`) file. To expand this file, double click the folder icon in your file navigator application (for Macs, this is the Finder +application). + +For a full description of the data used in this workshop see the [data page](instructors/data.md). + +## Option A: Local Installation + +### Software + +| Software | Install | Manual | Available for | Description | +| -------- | ------- | ------ | --------------------- | ------------------------------------------------------------- | +| [GDAL](https://www.gdal.org) | [Link](https://gdal.org/download.html) | [Link](https://gdal.org) | Linux, MacOS, Windows | Geospatial model for reading and writing a variety of formats | +| [GEOS](https://trac.osgeo.org/geos) | [Link](https://trac.osgeo.org/geos) | [Link](https://geos.osgeo.org/doxygen/) | Linux, MacOS, Windows | Geometry models and operations | +| [PROJ.4](https://proj4.org) | [Link](https://proj4.org/install.html) | [Link](https://proj4.org/index.html) | Linux, MacOS, Windows | Coordinate reference system transformations | +| [R](https://www.r-project.org) | [Link](https://cloud.r-project.org) | [Link](https://cloud.r-project.org) | Linux, MacOS, Windows | Software environment for statistical and scientific computing | +| [RStudio](https://www.rstudio.com) | [Link](https://www.rstudio.com/products/rstudio/download/#download) | | Linux, MacOS, Windows | GUI for R | +| [UDUNITS](https://www.unidata.ucar.edu/software/udunits/) | [Link](https://www.unidata.ucar.edu/downloads/udunits/index.jsp) | [Link](https://www.unidata.ucar.edu/software/udunits/#documentation) | Linux, MacOS, Windows | Unit conversions | + +We provide quick instructions below for installing the various software needed for this workshop. At points, they assume familiarity with the command line and with installation in general. As there are different operating systems and many different versions of operating systems and environments, these may not work on your computer. If an installation doesn't work for you, please refer to the installation instructions for that software listed in the table above. + +### GDAL, GEOS, and PROJ.4 + +The installation of the geospatial libraries GDAL, GEOS, and PROJ.4 varies significantly based on operating system. These are all dependencies for `sf`, the `R` package that we will be using for spatial data operations throughout this workshop. + +::::::::::::::: solution + +## Windows + +To install the geospatial libraries, install the latest version [RTools](https://cran.r-project.org/bin/windows/Rtools/) + +::::::::::::::::::::::::: + +::::::::::::::: solution + +## macOS - Install with Packages (Beginner) + +The simplest way to install these geospatial libraries is to install the latest version of [Kyng Chaos's pre-built package](https://www.kyngchaos.com/software/frameworks) for GDAL Complete. Be aware that several other libraries are also installed, including the UnixImageIO, SQLite3, and `NumPy`. + +After downloading the package in the link above, you will need to double-click the cardbord box icon to complete the installation. Depending on your security settings, you may get an error message about "unidentified developers". You can enable the installation by following [these instructions](https://kb.wisc.edu/page.php?id=25443) for installing programs from unidentified developers. + +::::::::::::::::::::::::: + +::::::::::::::: solution + +## macOS - Install with Homebrew (Advanced) + +Alternatively, participants who are comfortable with the command line can install the geospatial libraries individually using [homebrew](https://brew.sh): + +```bash +$ brew tap osgeo/osgeo4mac && brew tap --repair +$ brew install proj +$ brew install geos +$ brew install gdal2 +``` + +::::::::::::::::::::::::: + +::::::::::::::: solution + +## Linux + +Steps for installing the geospatial libraries will vary based on which form of Linux you are using. These instructions are adapted from the [`sf` package's `README`](https://github.com/r-spatial/sf). + +For **Ubuntu**: + +```bash +$ sudo add-apt-repository ppa:ubuntugis +$ sudo apt-get update +$ sudo apt-get install libgdal-dev libgeos-dev libproj-dev +``` + +For **Fedora**: + +```bash +$ sudo dnf install gdal-devel proj-devel geos-devel +``` + +For **Arch**: + +```bash +$ pacman -S gdal proj geos +``` + +For **Debian**: The [rocker geospatial](https://github.com/rocker-org/geospatial) Dockerfiles may be helpful. Ubuntu Dockerfiles are found [here](https://github.com/r-spatial/sf/tree/master/inst/docker). These may be helpful to get an idea of the commands needed to install the necessary dependencies. + +::::::::::::::::::::::::: + +### UDUNITS + +Linux users will have to install UDUNITS separately. Like the geospatial libraries discussed above, this is a dependency for the `R` package `sf`. Due to conflicts, it does not install properly on Linux machines when installed as part of the `sf` installation process. It is therefore necessary to install it using the command line ahead of time. + +::::::::::::::: solution + +## Linux + +Steps for installing the geospatial will vary based on which form of Linux you are using. These instructions are adapted from the [`sf` package's `README`](https://github.com/r-spatial/sf). + +For **Ubuntu**: + +```bash +$ sudo apt-get install libudunits2-dev +``` + +For **Fedora**: + +```bash +$ sudo dnf install udunits2-devel +``` + +For **Arch**: + +```bash +$ pacaur/yaourt/whatever -S udunits +``` + +For **Debian**: + +```bash +$ sudo apt-get install -y libudunits2-dev +``` + +::::::::::::::::::::::::: + +### R + +Participants who do not already have `R` installed should download and install it. + +::::::::::::::: solution + +## Windows + +To install `R`, Windows users should select "Download R for Windows" from RStudio and CRAN's [cloud download page](https://cloud.r-project.org), which will automatically detect a CRAN mirror for you to use. Select the `base` subdirectory after choosing the Windows download page. A `.exe` executable file containing the necessary components of base R can be downloaded by clicking on "Download R 3.x.x for Windows". + +::::::::::::::::::::::::: + +::::::::::::::: solution + +## macOS + +To install `R`, macOS users should select "Download R for (Mac) OS X" from RStudio and CRAN's [cloud download page](https://cloud.r-project.org), which will automatically detect a CRAN mirror for you to use. A `.pkg` file containing the necessary components of base R can be downloaded by clicking on the first available link (this will be the most recent), which will read `R-3.x.x.pkg`. + +::::::::::::::::::::::::: + +::::::::::::::: solution + +## Linux + +To install `R`, Linux users should select "Download R for Linux" from RStudio and CRAN's [cloud download page](https://cloud.r-project.org), which will automatically detect a CRAN mirror for you to use. Instructions for a number of different Linux operating systems are available. + +::::::::::::::::::::::::: + +### RStudio + +RStudio is a GUI for using `R` that is available for Windows, macOS, and various Linux operating systems. It can be downloaded [here](https://www.rstudio.com/products/rstudio/download/). You will need the **free** Desktop version for your computer. *In order to address issues with `ggplot2`, learners and instructors should run a recent version of RStudio (v1.2 or greater).* + +### R Packages + +The following `R` packages are used in the various geospatial lessons. + +- [`dplyr`](https://cran.r-project.org/package=dplyr) +- [`ggplot2`](https://cran.r-project.org/package=ggplot2) +- [`raster`](https://cran.r-project.org/package=raster) +- [`rgdal`](https://cran.r-project.org/package=rgdal) +- [`rasterVis`](https://cran.r-project.org/package=rasterVis) +- [`remotes`](https://cran.r-project.org/package=remotes) +- [`sf`](https://cran.r-project.org/package=sf) + +To install these packages in RStudio, do the following: +1\. Open RStudio by double-clicking the RStudio application icon. You should see +something like this: + +![](/fig/01-rstudio.png){alt='RStudio layout'} + +2\. Type the following into the console and hit enter. + +```r +install.packages(c("dplyr", "ggplot2", "raster", "rgdal", "rasterVis", "sf")) +``` + +You should see a status message starting with: + +```output +trying URL 'https://cran.rstudio.com/bin/macosx/el-capitan/contrib/3.5/dplyr_0.7.6.tgz' +Content type 'application/x-gzip' length 5686536 bytes (5.4 MB) +================================================== +downloaded 5.4 MB + +trying URL 'https://cran.rstudio.com/bin/macosx/el-capitan/contrib/3.5/ggplot2_3.0.0.tgz' +Content type 'application/x-gzip' length 3577658 bytes (3.4 MB) +================================================== +downloaded 3.4 MB +``` + +When the installation is complete, you will see a status message like: + +```output +The downloaded binary packages are in +/var/folders/7g/r8_n81y534z0vy5hxc6dx1t00000gn/T//RtmpJECKXM/downloaded_packages +``` + +You are now ready for the workshop! + +## Option B: Docker + +[Docker](https://www.docker.com) provides developers with a means for creating interactive [containers](https://docs.docker.com/glossary/?term=container) that contain pre-installed software. A selection of pre-installed software in Docker is called an [image](https://docs.docker.com/glossary/?term=image). An image can be downloaded and used to create a local container, allowing end-users to get software up and running quickly. This is particularly useful when a local installation of the software could be complex and time consuming. For `R` users, a Docker image can be used to create a virtual installation of `R` and RStudio that can be run through your web browser. + +This option involves downloading an Docker image that contains an installation of `R`, RStudio Server, all of the necessary dependencies listed above, and almost all of the `R` packages used in the geospatial lessons. You will need to install the appropriate version of Docker's Community Edition software and then download and use the `rocker/geospatial` Docker image to create a container that will allow you to use `R`, RStudio, and all the required GIS tools without installing any of them locally. + +Once up and running - you'll have full access to RStudio right from your browser: + +![](/fig/docker.png){alt='RStudio layout in browser showing similarity to desktop layout'} + +Please be aware that the `R` package `rasterVis` is not included in the `rocker/geospatial` Docker image. If your instructor teaches with this package then you will need to install this `R` package yourself. All other `R` packages will already be installed for you. + +::::::::::::::: solution + +## Downloading and Installing Docker Community Edition + +To get started with Docker, download the [Docker Community Edition](https://www.docker.com/community-edition) from [Docker's store](https://store.docker.com/search?type=edition&offering=community). Community editions are available for [Windows](https://store.docker.com/editions/community/docker-ce-desktop-windows), [macOS](https://store.docker.com/editions/community/docker-ce-desktop-mac), and Linux operating systems including [Debian](https://store.docker.com/editions/community/docker-ce-server-debian), [Fedora](https://store.docker.com/editions/community/docker-ce-server-fedora), and [Ubuntu](https://store.docker.com/editions/community/docker-ce-server-ubuntu). + +The download pages for each of these operating systems contain notes about some necessary system requirements and other pre-requisites. Once you download the installer and follow the on-screen prompts. + +Additional installation notes are available in Docker's documentation for each of these operating systems: [Windows](https://docs.docker.com/docker-for-windows/install/), [macOS](https://docs.docker.com/docker-for-mac/install/), [Debian](https://docs.docker.com/install/linux/docker-ce/debian/), [Fedora](https://docs.docker.com/install/linux/docker-ce/fedora/), and [Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/). + +::::::::::::::::::::::::: + +::::::::::::::: solution + +## Using the `rocker/geospatial` Docker Image via the Command Line + +#### Download and Set-up + +Once Docker is installed and up and running, you will need to open your computer's command line terminal. We'll use the terminal to download [`rocker/geospatial`](https://github.com/rocker-org/geospatial), a pre-made Docker image that contains an installation of `R`, RStudio Server, all of the necessary dependencies, and all but one of the `R` packages needed for this workshop. + +You need to have already installed Docker Community Edition (see instructions above) before proceeding. Once you have Docker downloaded and installed, make sure Docker is running and then enter the following command into the terminal to download the `rocker/geospatial` image: + +```bash +$ docker pull rocker/geospatial +``` + +Once the pull command is executed, the image needs to be run to become accessible as a container. In the following example, the image is named `rocker/geospatial` and the container is named `gis`. The [image](https://docs.docker.com/glossary/?term=image) contains the software you've downloaded, and the [container](https://docs.docker.com/glossary/?term=container) is the run-time instance of that image. New Docker users should need only one named container per image. + +When `docker run` is used, you can specify a folder on your computer to become accessible inside your RStudio Server instance. The following `docker run` command exposes Jane's `GitHub` directory to RStudio Server. Enter the file path where your workshop resources and data are stored: + +```bash +$ docker run -d -P --name gis -v /Users/jane/GitHub:/home/rstudio/GitHub -e PASSWORD=mypass rocker/geospatial +``` + +When she opens her RStudio instance below, she will see a `GitHub` folder in her file tab in the lower righthand corner of the screen. Windows and Linux users will have to adapt the file path above to follow the standards of their operating systems. More details are available on [rocker's Wiki](https://github.com/rocker-org/rocker/wiki/Sharing-files-with-host-machine). + +The last step before launching your container in a browser is to identify the port that your Docker container is running in: + +```bash +$ docker port gis +``` + +An output, for example, of `8787/tcp -> 0.0.0.0:32768` would indicate that you should point your browser to `http://localhost:32768/`. If prompted, enter `rstudio` for the username and the password provided in the `docker run` command above (`mypass` in the example above). + +#### Stopping a Container + +When you are done with a Docker session, make sure all of your files are saved locally on your computer **before closing your browser and Docker**. Once you have ensured all of your files are available (they should be saved at the file path designated in `docker run` above), you can stop your Docker container in the terminal: + +```bash +$ docker stop gis +``` + +#### Re-starting a Container + +Once a container has been named and created, you cannot create a container with the same name again using `docker run`. Instead, you can restart it: + +```bash +$ docker start gis +``` + +If you cannot remember the name of the container you created, you can use the following command to print a list of all named containers: + +```bash +$ docker ps -a +``` + +If you are returning to a session after stopping Docker itself, make sure Docker is running again before re-starting your container! + +::::::::::::::::::::::::: + +::::::::::::::: solution + +## Using the `rocker/geospatial` Docker Image via Kitematic + +#### Download and Install Kitematic + +[Kitematic](https://github.com/docker/kitematic) is the GUI, currently in beta, that Docker has built for accessing images and containers on Windows, macOS, and Ubuntu. You can download the appropriate installer files from Kitematic's [GitHub release page](https://github.com/docker/kitematic/releases/tag/v0.17.3). You need to have already installed Docker Community Edition (see instructions above) before installing Kitematic! + +#### Opening a Container with Kitematic + +Once you have installed Kitematic, make sure the Docker application is running and then open Kitematic. You should not need to create a login to use Kitematic. If prompted for login credentials, there is an option to skip that step. Use the search bar in the main window to find `rocker/geospatial` (pictured below) and click `Create` under that Docker repository. + +![](/fig/kitematicSearch.png) + +After downloading and installing the image, your container should start automatically. Before opening your browser, connect your Docker image to a local folder where you have your workshop resources stored by clicking on the `Settings` tab and then choosing `Volumes`. Click `Change` and then select the directory you would like to connect to. + +![](/fig/kitematicLocal.png) + +When you open RStudio instance below, you will see the contents of the connected folder inside the `kitematic` directory in the file tab located in the lower righthand corner of the screen. + +When you are ready, copy the `Access URL` from the `Home` tab: + +![](/fig/kitematicURL.png) + +Paste that url into your browser and, if prompted, enter `rstudio` for both the username and the password. + +#### Stopping and Restarting a Container + +When you are done with a Docker session, make sure all of your files are saved locally on your computer **before closing your browser and Docker**. Once you have ensured all of your files are available (they should be saved at the file path designated in `docker run` above), you can stop your Docker container by clicking on the `Stop` icon in Kitematic's toolbar. + +You can restart your container later by clicking the `Restart` button. + +::::::::::::::::::::::::: + +::::::::::::::: solution + +## Managing Docker Containers and Images + +To obtain a list of all of your current Docker containers: + +```bash +$ docker ps -a +``` + +To list all of the currently downloaded Docker images: + +```bash +$ docker images -a +``` + +These images can take up system resources, and if you'd like to remove them, you can use the `docker prune` command. To remove any Docker resources not affiliated with a container listed under `docker ps -a`: + +```bash +$ docker system prune +``` + +To remove **all** Docker resources, including currently named containers: + +```bash +$ docker system prune -a +``` + +::::::::::::::::::::::::: + + diff --git a/profiles/learner-profiles.md b/profiles/learner-profiles.md new file mode 100644 index 0000000..434e335 --- /dev/null +++ b/profiles/learner-profiles.md @@ -0,0 +1,5 @@ +--- +title: FIXME +--- + +This is a placeholder file. Please add content here. diff --git a/setup.md b/setup.md deleted file mode 100644 index ce38bb2..0000000 --- a/setup.md +++ /dev/null @@ -1,353 +0,0 @@ ---- -layout: page -title: Setup ---- - -# Overview - -This workshop is designed to be run on your local machine. First, you will need to download the data we use in the workshop. Then, you need to set up your machine to analyze and process geospatial data. We provide instructions below to either install all components manually (option A), or to use a Docker image that provides all the software and dependencies needed (option B). - -## Data - -You can download all of the data used in this workshop by clicking -[this download link](https://ndownloader.figshare.com/articles/2009586/versions/10). The file is 218.2 MB. - -Clicking the download link will automatically download all of the files to your default download directory as a single compressed -(`.zip`) file. To expand this file, double click the folder icon in your file navigator application (for Macs, this is the Finder -application). - -For a full description of the data used in this workshop see the [data page](data). - -## Option A: Local Installation - -### Software - -| Software | Install | Manual | Available for | Description | -| -------- | ------------ | ------ | ------------- | ----------- | -| [GDAL](http://www.gdal.org) | [Link](https://gdal.org/download.html) | [Link](https://gdal.org) | Linux, MacOS, Windows | Geospatial model for reading and writing a variety of formats | -| [GEOS](https://trac.osgeo.org/geos) | [Link](https://trac.osgeo.org/geos) | [Link](http://geos.osgeo.org/doxygen/) | Linux, MacOS, Windows | Geometry models and operations | -| [PROJ.4](http://proj4.org) | [Link](http://proj4.org/install.html)| [Link](http://proj4.org/index.html)| Linux, MacOS, Windows | Coordinate reference system transformations | -| [R](https://www.r-project.org) | [Link](https://cloud.r-project.org) | [Link](https://cloud.r-project.org) | Linux, MacOS, Windows | Software environment for statistical and scientific computing | -| [RStudio](https://www.rstudio.com) | [Link](https://www.rstudio.com/products/rstudio/download/#download) | | Linux, MacOS, Windows | GUI for R | -| [UDUNITS](https://www.unidata.ucar.edu/software/udunits/) | [Link](https://www.unidata.ucar.edu/downloads/udunits/index.jsp) | [Link](https://www.unidata.ucar.edu/software/udunits/#documentation) | Linux, MacOS, Windows | Unit conversions | - -We provide quick instructions below for installing the various software needed for this workshop. At points, they assume familiarity with the command line and with installation in general. As there are different operating systems and many different versions of operating systems and environments, these may not work on your computer. If an installation doesn't work for you, please refer to the installation instructions for that software listed in the table above. - - -### GDAL, GEOS, and PROJ.4 - -The installation of the geospatial libraries GDAL, GEOS, and PROJ.4 varies significantly based on operating system. These are all dependencies for `sf`, the `R` package that we will be using for spatial data operations throughout this workshop. - -> ## Windows -> ->To install the geospatial libraries, install the latest version [RTools](https://cran.r-project.org/bin/windows/Rtools/) -> -{: .solution} - -> ## macOS - Install with Packages (Beginner) -> -> The simplest way to install these geospatial libraries is to install the latest version of [Kyng Chaos's pre-built package](http://www.kyngchaos.com/software/frameworks) for GDAL Complete. Be aware that several other libraries are also installed, including the UnixImageIO, SQLite3, and `NumPy`. -> -> After downloading the package in the link above, you will need to double-click the cardbord box icon to complete the installation. Depending on your security settings, you may get an error message about "unidentified developers". You can enable the installation by following [these instructions](https://kb.wisc.edu/page.php?id=25443) for installing programs from unidentified developers. -> -{: .solution} - -> ## macOS - Install with Homebrew (Advanced) -> -> Alternatively, participants who are comfortable with the command line can install the geospatial libraries individually using [homebrew](https://brew.sh): -> ->~~~ ->$ brew tap osgeo/osgeo4mac && brew tap --repair ->$ brew install proj ->$ brew install geos ->$ brew install gdal2 ->~~~ ->{: .language-bash} -> -{: .solution} - -> ## Linux -> -> Steps for installing the geospatial libraries will vary based on which form of Linux you are using. These instructions are adapted from the [`sf` package's `README`](https://github.com/r-spatial/sf). -> -> For **Ubuntu**: -> ->~~~ ->$ sudo add-apt-repository ppa:ubuntugis ->$ sudo apt-get update ->$ sudo apt-get install libgdal-dev libgeos-dev libproj-dev ->~~~ ->{: .language-bash} -> -> For **Fedora**: -> ->~~~ ->$ sudo dnf install gdal-devel proj-devel geos-devel ->~~~ ->{: .language-bash} -> -> For **Arch**: -> ->~~~ ->$ pacman -S gdal proj geos ->~~~ ->{: .language-bash} -> -> For **Debian**: The [rocker geospatial](https://github.com/rocker-org/geospatial) Dockerfiles may be helpful. Ubuntu Dockerfiles are found [here](https://github.com/r-spatial/sf/tree/master/inst/docker). These may be helpful to get an idea of the commands needed to install the necessary dependencies. -> -{: .solution} - -### UDUNITS - -Linux users will have to install UDUNITS separately. Like the geospatial libraries discussed above, this is a dependency for the `R` package `sf`. Due to conflicts, it does not install properly on Linux machines when installed as part of the `sf` installation process. It is therefore necessary to install it using the command line ahead of time. - -> ## Linux -> -> Steps for installing the geospatial will vary based on which form of Linux you are using. These instructions are adapted from the [`sf` package's `README`](https://github.com/r-spatial/sf). -> -> For **Ubuntu**: -> ->~~~ ->$ sudo apt-get install libudunits2-dev ->~~~ ->{: .language-bash} -> -> For **Fedora**: -> ->~~~ ->$ sudo dnf install udunits2-devel ->~~~ ->{: .language-bash} -> -> For **Arch**: -> ->~~~ ->$ pacaur/yaourt/whatever -S udunits ->~~~ ->{: .language-bash} -> -> For **Debian**: -> ->~~~ ->$ sudo apt-get install -y libudunits2-dev ->~~~ ->{: .language-bash} -> -{: .solution} - - -### R - -Participants who do not already have `R` installed should download and install it. - -> ## Windows -> ->To install `R`, Windows users should select "Download R for Windows" from RStudio and CRAN's [cloud download page](https://cloud.r-project.org), which will automatically detect a CRAN mirror for you to use. Select the `base` subdirectory after choosing the Windows download page. A `.exe` executable file containing the necessary components of base R can be downloaded by clicking on "Download R 3.x.x for Windows". -> -{: .solution} - -> ## macOS -> ->To install `R`, macOS users should select "Download R for (Mac) OS X" from RStudio and CRAN's [cloud download page](https://cloud.r-project.org), which will automatically detect a CRAN mirror for you to use. A `.pkg` file containing the necessary components of base R can be downloaded by clicking on the first available link (this will be the most recent), which will read `R-3.x.x.pkg`. -> -{: .solution} - -> ## Linux -> ->To install `R`, Linux users should select "Download R for Linux" from RStudio and CRAN's [cloud download page](https://cloud.r-project.org), which will automatically detect a CRAN mirror for you to use. Instructions for a number of different Linux operating systems are available. -> -{: .solution} - -### RStudio -RStudio is a GUI for using `R` that is available for Windows, macOS, and various Linux operating systems. It can be downloaded [here](https://www.rstudio.com/products/rstudio/download/). You will need the **free** Desktop version for your computer. *In order to address issues with `ggplot2`, learners and instructors should run a recent version of RStudio (v1.2 or greater).* - -### R Packages - -The following `R` packages are used in the various geospatial lessons. - -* [`dplyr`](https://cran.r-project.org/package=dplyr) -* [`ggplot2`](https://cran.r-project.org/package=ggplot2) -* [`raster`](https://cran.r-project.org/package=raster) -* [`rgdal`](https://cran.r-project.org/package=rgdal) -* [`rasterVis`](https://cran.r-project.org/package=rasterVis) -* [`remotes`](https://cran.r-project.org/package=remotes) -* [`sf`](https://cran.r-project.org/package=sf) - -To install these packages in RStudio, do the following: -1\. Open RStudio by double-clicking the RStudio application icon. You should see -something like this: - -![RStudio layout](/fig/01-rstudio.png) - - -2\. Type the following into the console and hit enter. - -~~~ -install.packages(c("dplyr", "ggplot2", "raster", "rgdal", "rasterVis", "sf")) -~~~ -{: .language-r} - -You should see a status message starting with: - -~~~ -trying URL 'https://cran.rstudio.com/bin/macosx/el-capitan/contrib/3.5/dplyr_0.7.6.tgz' -Content type 'application/x-gzip' length 5686536 bytes (5.4 MB) -================================================== -downloaded 5.4 MB - -trying URL 'https://cran.rstudio.com/bin/macosx/el-capitan/contrib/3.5/ggplot2_3.0.0.tgz' -Content type 'application/x-gzip' length 3577658 bytes (3.4 MB) -================================================== -downloaded 3.4 MB -~~~ -{: .output} - -When the installation is complete, you will see a status message like: - -~~~ -The downloaded binary packages are in -/var/folders/7g/r8_n81y534z0vy5hxc6dx1t00000gn/T//RtmpJECKXM/downloaded_packages -~~~ -{: .output} - -You are now ready for the workshop! - -## Option B: Docker - -[Docker](https://www.docker.com) provides developers with a means for creating interactive [containers](https://docs.docker.com/glossary/?term=container) that contain pre-installed software. A selection of pre-installed software in Docker is called an [image](https://docs.docker.com/glossary/?term=image). An image can be downloaded and used to create a local container, allowing end-users to get software up and running quickly. This is particularly useful when a local installation of the software could be complex and time consuming. For `R` users, a Docker image can be used to create a virtual installation of `R` and RStudio that can be run through your web browser. - -This option involves downloading an Docker image that contains an installation of `R`, RStudio Server, all of the necessary dependencies listed above, and almost all of the `R` packages used in the geospatial lessons. You will need to install the appropriate version of Docker's Community Edition software and then download and use the `rocker/geospatial` Docker image to create a container that will allow you to use `R`, RStudio, and all the required GIS tools without installing any of them locally. - -Once up and running - you'll have full access to RStudio right from your browser: - -![RStudio layout in browser showing similarity to desktop layout](/fig/docker.png) - -Please be aware that the `R` package `rasterVis` is not included in the `rocker/geospatial` Docker image. If your instructor teaches with this package then you will need to install this `R` package yourself. All other `R` packages will already be installed for you. - -> ## Downloading and Installing Docker Community Edition -> -> To get started with Docker, download the [Docker Community Edition](https://www.docker.com/community-edition) from [Docker's store](https://store.docker.com/search?type=edition&offering=community). Community editions are available for [Windows](https://store.docker.com/editions/community/docker-ce-desktop-windows), [macOS](https://store.docker.com/editions/community/docker-ce-desktop-mac), and Linux operating systems including [Debian](https://store.docker.com/editions/community/docker-ce-server-debian), [Fedora](https://store.docker.com/editions/community/docker-ce-server-fedora), and [Ubuntu](https://store.docker.com/editions/community/docker-ce-server-ubuntu). -> -> The download pages for each of these operating systems contain notes about some necessary system requirements and other pre-requisites. Once you download the installer and follow the on-screen prompts. -> -> Additional installation notes are available in Docker's documentation for each of these operating systems: [Windows](https://docs.docker.com/docker-for-windows/install/), [macOS](https://docs.docker.com/docker-for-mac/install/), [Debian](https://docs.docker.com/install/linux/docker-ce/debian/), [Fedora](https://docs.docker.com/install/linux/docker-ce/fedora/), and [Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/). -> -{: .solution} - -> ## Using the `rocker/geospatial` Docker Image via the Command Line -> -> #### Download and Set-up -> Once Docker is installed and up and running, you will need to open your computer's command line terminal. We'll use the terminal to download [`rocker/geospatial`](https://github.com/rocker-org/geospatial), a pre-made Docker image that contains an installation of `R`, RStudio Server, all of the necessary dependencies, and all but one of the `R` packages needed for this workshop. -> -> You need to have already installed Docker Community Edition (see instructions above) before proceeding. Once you have Docker downloaded and installed, make sure Docker is running and then enter the following command into the terminal to download the `rocker/geospatial` image: -> ->~~~ ->$ docker pull rocker/geospatial ->~~~ ->{: .language-bash} -> -> Once the pull command is executed, the image needs to be run to become accessible as a container. In the following example, the image is named `rocker/geospatial` and the container is named `gis`. The [image](https://docs.docker.com/glossary/?term=image) contains the software you've downloaded, and the [container](https://docs.docker.com/glossary/?term=container) is the run-time instance of that image. New Docker users should need only one named container per image. -> -> When `docker run` is used, you can specify a folder on your computer to become accessible inside your RStudio Server instance. The following `docker run` command exposes Jane's `GitHub` directory to RStudio Server. Enter the file path where your workshop resources and data are stored: -> ->~~~ ->$ docker run -d -P --name gis -v /Users/jane/GitHub:/home/rstudio/GitHub -e PASSWORD=mypass rocker/geospatial ->~~~ ->{: .language-bash} -> -> When she opens her RStudio instance below, she will see a `GitHub` folder in her file tab in the lower righthand corner of the screen. Windows and Linux users will have to adapt the file path above to follow the standards of their operating systems. More details are available on [rocker's Wiki](https://github.com/rocker-org/rocker/wiki/Sharing-files-with-host-machine). -> -> The last step before launching your container in a browser is to identify the port that your Docker container is running in: -> ->~~~ ->$ docker port gis ->~~~ ->{: .language-bash} -> -> An output, for example, of `8787/tcp -> 0.0.0.0:32768` would indicate that you should point your browser to `http://localhost:32768/`. If prompted, enter `rstudio` for the username and the password provided in the `docker run` command above (`mypass` in the example above). -> -> #### Stopping a Container -> When you are done with a Docker session, make sure all of your files are saved locally on your computer **before closing your browser and Docker**. Once you have ensured all of your files are available (they should be saved at the file path designated in `docker run` above), you can stop your Docker container in the terminal: -> ->~~~ ->$ docker stop gis ->~~~ ->{: .language-bash} -> -> #### Re-starting a Container -> Once a container has been named and created, you cannot create a container with the same name again using `docker run`. Instead, you can restart it: -> ->~~~ ->$ docker start gis ->~~~ ->{: .language-bash} -> -> If you cannot remember the name of the container you created, you can use the following command to print a list of all named containers: -> ->~~~ ->$ docker ps -a ->~~~ ->{: .language-bash} -> -> If you are returning to a session after stopping Docker itself, make sure Docker is running again before re-starting your container! -> -{: .solution} - -> ## Using the `rocker/geospatial` Docker Image via Kitematic -> #### Download and Install Kitematic -> [Kitematic](https://github.com/docker/kitematic) is the GUI, currently in beta, that Docker has built for accessing images and containers on Windows, macOS, and Ubuntu. You can download the appropriate installer files from Kitematic's [GitHub release page](https://github.com/docker/kitematic/releases/tag/v0.17.3). You need to have already installed Docker Community Edition (see instructions above) before installing Kitematic! -> -> #### Opening a Container with Kitematic -> Once you have installed Kitematic, make sure the Docker application is running and then open Kitematic. You should not need to create a login to use Kitematic. If prompted for login credentials, there is an option to skip that step. Use the search bar in the main window to find `rocker/geospatial` (pictured below) and click `Create` under that Docker repository. -> -> ![](/fig/kitematicSearch.png) -> -> After downloading and installing the image, your container should start automatically. Before opening your browser, connect your Docker image to a local folder where you have your workshop resources stored by clicking on the `Settings` tab and then choosing `Volumes`. Click `Change` and then select the directory you would like to connect to. -> -> ![](/fig/kitematicLocal.png) -> -> When you open RStudio instance below, you will see the contents of the connected folder inside the `kitematic` directory in the file tab located in the lower righthand corner of the screen. -> -> When you are ready, copy the `Access URL` from the `Home` tab: -> -> ![](/fig/kitematicURL.png) -> -> Paste that url into your browser and, if prompted, enter `rstudio` for both the username and the password. -> -> #### Stopping and Restarting a Container -> When you are done with a Docker session, make sure all of your files are saved locally on your computer **before closing your browser and Docker**. Once you have ensured all of your files are available (they should be saved at the file path designated in `docker run` above), you can stop your Docker container by clicking on the `Stop` icon in Kitematic's toolbar. -> -> You can restart your container later by clicking the `Restart` button. -> -{: .solution} - -> ## Managing Docker Containers and Images -> To obtain a list of all of your current Docker containers: -> ->~~~ ->$ docker ps -a ->~~~ ->{: .language-bash} -> -> To list all of the currently downloaded Docker images: -> ->~~~ ->$ docker images -a ->~~~ ->{: .language-bash} -> -> These images can take up system resources, and if you'd like to remove them, you can use the `docker prune` command. To remove any Docker resources not affiliated with a container listed under `docker ps -a`: -> ->~~~ ->$ docker system prune ->~~~ ->{: .language-bash} -> -> To remove **all** Docker resources, including currently named containers: -> ->~~~ ->$ docker system prune -a ->~~~ ->{: .language-bash} -> -{: .solution} diff --git a/site/README.md b/site/README.md new file mode 100644 index 0000000..42997e3 --- /dev/null +++ b/site/README.md @@ -0,0 +1,2 @@ +This directory contains rendered lesson materials. Please do not edit files +here.