source commit: 3a30cbc

Author: actions-user

01-getting-started.md

@@ -0,0 +1,84 @@
+---
+title: Introduction
+teaching: 10
+exercises: 0
+---
+
+::::::::::::::::::::::::::::::::::::::: objectives
+
+- Describe OpenRefine’s uses and applications.
+- Differentiate data cleaning from data organization.
+- Experiment with OpenRefine’s user interface.
+- Locate helpful resources to learn more about OpenRefine.
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+:::::::::::::::::::::::::::::::::::::::: questions
+
+- How is OpenRefine useful?
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+## Lesson
+
+## Motivations for the OpenRefine Lesson
+
+- Data is often very messy, and this tool saves a lot of time on cleaning
+  headaches.
+
+- Data cleaning steps often need repeating with multiple files. It is important to know what you did to your data. This makes it possible for you to repeat these steps again with similarly structured data. OpenRefine is
+  perfect for speeding up repetitive tasks by replaying previous actions on
+  multiple datasets.
+
+- Additionally, journals, granting agencies, and other institutions are requiring documentation of the
+  steps you took when working with your data. With OpenRefine, you can capture
+  all actions applied to your raw data and share them with your publication as
+  supplemental material.
+
+- Any operation that changes the data in OpenRefine can be reversed or
+  undone.
+
+- Some concepts such as clustering algorithms are quite complex, but with OpenRefine
+  we can introduce them, use them, and show their power.
+  
+  > **Note:** You must export your modified dataset to a new file: OpenRefine does not save over the original source file. All changes are stored in the OpenRefine project.
+
+## Before we get started
+
+The following setup is necessary before we can get started (see the [instructions here](../learners/setup.md).)
+
+## What is OpenRefine?
+
+- OpenRefine is a Java program that runs on your machine (not in the cloud): it is a desktop application that uses your web browser as a graphical interface. No internet connection is needed, and none of the data or commands you enter in OpenRefine are sent to a remote server.
+- OpenRefine does not modify your original dataset. All actions can be reversed in OpenRefine and you can capture all the actions applied to your data and share this documentation with your publication as supplemental material.
+- OpenRefine saves as you go. You can return to the project at any time to pick up where you left off or export your data to a new file.
+- OpenRefine can be used to standardise and clean data across your file.
+
+### It can also help you
+
+- Get an overview of a data set
+- Resolve inconsistencies in a data set
+- Help you split data up into more granular parts
+- Match local data up to other data sets
+- Enhance a data set with data from other sources
+- Save a set of data cleaning steps to replay on multiple files
+
+OpenRefine is a powerful, free, and open source tool with a large growing community of practice. More help can be found at [https://openrefine.org](https://openrefine.org).
+
+### Features
+
+- Open source ([source on GitHub](https://github.com/OpenRefine/OpenRefine)).
+- A large growing community, from novice to expert, ready to help.
+
+### More Information on OpenRefine
+
+You can find out a lot more about OpenRefine at the official user manual [docs.openrefine.org](https://docs.openrefine.org/). There is a [user forum](https://forum.openrefine.org) that can answer a lot of beginner questions and problems. [Recipes](https://github.com/OpenRefine/OpenRefine/wiki/Recipes), scripts, projects, and extensions are available to add functionality to OpenRefine. These can be copied into your OpenRefine instance to run on your dataset.
+
+:::::::::::::::::::::::::::::::::::::::: keypoints
+
+- OpenRefine is a powerful, free and open source tool that can be used for data cleaning.
+- OpenRefine will automatically track any steps you take in working with your data.
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+

02-importing-data.md

@@ -0,0 +1,90 @@
+---
+title: Importing Data to OpenRefine
+teaching: 10
+exercises: 0
+---
+
+::::::::::::::::::::::::::::::::::::::: objectives
+
+- Create a new OpenRefine project from a CSV file
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+:::::::::::::::::::::::::::::::::::::::: questions
+
+- How can we import our data into OpenRefine?
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+## Importing data
+
+:::::::::::::::::::::::::::::::::::::::::  callout
+
+## What kinds of data files can I import?
+
+There are several options for getting your data set into OpenRefine. You can import files in a variety of formats including:
+
+- Comma-separated values (CSV) or tab-separated values (TSV)
+- Text files
+- Fixed-width columns
+- JSON
+- XML
+- OpenDocument spreadsheet (ODS)
+- Excel spreadsheet (XLS or XLSX)
+- RDF data (JSON-LD, N3, N-Triples, Turtle, RDF/XML)
+- Wikitext
+
+See the [Create a project by importing data](https://docs.openrefine.org/manual/starting#create-a-project-by-importing-data) page in the OpenRefine manual for more information.
+
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+## Create your first OpenRefine project (using provided data)
+
+Start OpenRefine, which will open in your browser (at the address [http://127.0.0.0:3333](https://127.0.0.0:3333)). Once OpenRefine is launched in your
+browser, the left margin has options to:
+
+- `Create Project`
+- `Open Project`
+- `Import Project`
+- `Language Settings`
+
+1. Click `Create Project` from the left margin and select then `This Computer` (because you're uploading data from your computer).
+
+2. Click `Choose Files` and browse to where you stored the file `Portal_rodents_19772002_simplified.csv`. Select the
+  file and click `Open`, or just double-click on the filename.
+  
+  ![](fig/or372-create-project.png){alt='Menu to create a new project'}
+
+3. Click `Next>>` under the browse button to upload the data into OpenRefine.
+
+4. On the next screen, OpenRefine will present you with a preview of your data. You can check here for obvious errors, if, for example, your file was tab-delimited rather than comma-delimited, the preview would look strange (and you could correct it by choosing the correct separator and clicking the `Update Preview` button on the right. If you selected the wrong file, click `<<Start Over` at the top left.
+
+5. In the middle of the page, will be a set of options (`Character encoding`, etc.). Make sure the tick box next to `Trim leading & trailing whitespace from strings` is not ticked. (We're going to need the leading whitespace in one of our examples.)
+  
+  ![](fig/or372-data-import.png){alt='Menu to import data'}
+
+6. If all looks well, click `Create Project>>` in the top right. You will be presented with a view onto your data. This is OpenRefine!
+
+The columns are all imported as text, even the columns with numbers. We will see how to format the numeric columns in the next episode.
+
+:::::::::::::::::::::::::::::::::::::::::  callout
+
+## OpenRefine does not modify your original dataset
+
+Once your data is imported into a project - OpenRefine leaves your raw data intact and works on a copy which it creates
+inside the newly created project. All the data transformation and cleaning steps you apply will be performed on this copy
+and you can undo any changes too.
+
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+:::::::::::::::::::::::::::::::::::::::: keypoints
+
+- Use the Create Project option to import data
+- You can control how data imports using options on the import screen
+- Several file types may be imported into OpenRefine
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+

03-exploring-data.md

@@ -0,0 +1,168 @@
+---
+title: Exploring Data with OpenRefine
+teaching: 15
+exercises: 20
+---
+
+::::::::::::::::::::::::::::::::::::::: objectives
+
+- Learn about different types of facets and how they can be used to summarise data of different data types
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+:::::::::::::::::::::::::::::::::::::::: questions
+
+- How can we summarise our data?
+- How can we find errors in our data?
+- How can we edit data to fix errors?
+- How can we convert column data from one data type to another?
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+## Exploring data with facets
+
+Facets are one of the most useful features of OpenRefine. Data faceting is a process of exploring data by applying multiple filters to investigate its composition. It also allows you to identify a subset of data that you wish to change in bulk.
+
+A `facet` groups all the like values that appear in a column, and allows you to filter the data by those values. It also allows you to edit values across many records at the same time.
+
+### Exploring text columns
+
+One type of facet is called a 'Text facet'. This groups all the identical text values in a column and lists each value with the number of records it appears in. The facet information always appears in the left hand panel in the OpenRefine interface.
+
+Here we will use faceting to look for potential errors in data entry in the `scientificName` column.
+
+1. Scroll over to the `scientificName` column.
+
+2. Click the down arrow and choose `Facet` > `Text facet`.
+  
+  ![](fig/or362-facet-menu.png){alt='Facet menu of a column'}
+
+3. In the left panel, you'll now see a box containing every unique value in the `scientificName` column
+  along with a number representing how many times that value occurs in the column.
+  
+  ![](fig/or362-faceted-scientificname.png){alt='Faceting results on the column scientificName'}
+
+4. Try sorting this facet by name and by count. Do you notice any problems with the data? What are they?
+
+5. Hover the mouse over one of the names in the `facet` list. You should see that you have an `edit` function available.
+
+6. You could use this to fix an error immediately, and OpenRefine will ask whether you want to make the same correction to every value it finds like that one. But OpenRefine offers even better ways to find and fix these errors, which we'll use instead. We'll learn about these when we talk about clustering.
+
+:::::::::::::::::::::::::::::::::::::::::  callout
+
+## Facets and large datasets
+
+Facets are intended to group together common values and OpenRefine limits the number of values allowed in a single facet to ensure the software does not perform slowly or run out of memory. If you create a facet where there are many unique values (for example, a facet on a 'book title' column in a data set that has one row per book) the facet created will be very large and may either slow down the application, or OpenRefine will not create the facet.
+
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+:::::::::::::::::::::::::::::::::::::::  challenge
+
+## Exercise
+
+1. Using faceting, find out how many years are represented in the census.
+2. Which years have the most and least observations?
+3. Is the column formatted as Number, Date, or Text?
+
+:::::::::::::::  solution
+
+## Solution
+
+1. For the column `yr` do `Facet` > `Text facet`. A box will appear in the left panel showing that there are 16 unique entries in this column.
+2. After creating a facet, click `Sort by count` in the facet box. The year with the most observations is 1978. The least is 1993.
+3. By default, the column `yr` is formatted as Text.
+
+:::::::::::::::::::::::::
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+### Exploring numeric columns
+
+When a table is imported into OpenRefine, all columns are treated as having text values. We can transform columns to other data types (e.g. number or date) using the `Edit cells` > `Common transforms` feature. Here we will experiment changing columns to numbers and see what additional capabilities that grants us.
+
+#### Numeric facet
+
+Sometimes there are non-number values or blanks in a column which may represent errors in data entry and we want to find them. We can do that with a `Numeric facet`.
+
+Create a `numeric facet` for the column `yr`. The facet will be empty because OpenRefine sees all the values as text.
+
+To transform cells in the `yr` column to numbers, click the down arrow for that column, then `Edit cells` > `Common transforms…` > `To number`. You will notice the `yr` values change from left-justified to right-justified, and black to green color.
+
+:::::::::::::::::::::::::::::::::::::::  challenge
+
+## Exercise
+
+The dataset included other numeric columns that we will explore in this exercise:
+
+- `period` - Unique number assigned to each survey period
+- `plot` - Plot number animal was caught on, from 1 to 24
+- `recordID` - Unique record ID number to facilitate quick reference to particular entry
+
+Transform the columns `period`, `plot`, and `recordID` from text to numbers.
+
+1. How does changing the format change the faceting display for the `yr` column?
+2. Can all columns be transformed to numbers?
+
+:::::::::::::::  solution
+
+## Solution
+
+Displaying a `Numeric facet` of `yr` shows a histogram of the number of
+entries per year. Notice that the data is shown as a number, not a date. If you instead transform the column to a date, the program will assume all entries are on January 1st of the year.
+
+Only observations that include only numerals (0-9) can be transformed to numbers. If you apply a number transformation to a column that doesn't meet this criteria, and then click the Undo / Redo tab, you will see a step that starts with Text transform on 0 cells. This means that the data in that column was not transformed.
+
+:::::::::::::::::::::::::
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+The next exercise will explore what happens when a numeric column contains values that are not numbers.
+
+:::::::::::::::::::::::::::::::::::::::  challenge
+
+## Exercise
+
+1. For a column you transformed to numbers, edit one or two cells, replacing the numbers with text (such as `abc`) or blank (no number or text).
+2. Use the pulldown menu to apply a numeric facet to the column you edited. The facet will appear in the left panel.
+3. Notice that there are several checkboxes in this facet: `Numeric`, `Non-numeric`, `Blank`, and `Error`. Below these are counts of the number of cells in each category. You should see checks for `Non-numeric` and `Blank` if you changed some values.
+4. Experiment with checking or unchecking these boxes to select subsets of your data.
+  
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+When done examining the numeric data, remove this facet by clicking the `x` in the upper left corner of its panel. Note that this does not undo the edits you made to the cells in this column.
+
+#### Examine a pair of numeric columns using scatterplots
+
+Now that we have multiple columns representing numbers, we can see how they relate to one another using the scatterplot facet. Select a numeric column, for example `recordID`, and use the pulldown menu to > `Facet` > `Scatterplot facet`. A new window called `Scatterplot Matrix` will appear. There are squares for each pair of numeric columns organized in an upper right triangle. Each square has little dots for the cell values from each row.
+
+![](fig/or372-scatterplots.png){alt='Scatterplots between numeric columns'}
+
+Click the image of the scatterplot between `recordID` and `yr` to select this one for the facet.
+
+:::::::::::::::::::::::::::::::::::::::  challenge
+
+## Exercise
+
+Click in the scatterplot facet in the lower left margin and drag to highlight a rectangle. How does this change the data rows displayed?
+
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+:::::::::::::::::::::::::::::::::::::::::  callout
+
+## More Details on Faceting
+
+Full documentation on faceting can be found at [Exploring facets: Faceting](https://docs.openrefine.org/manual/facets)
+
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+:::::::::::::::::::::::::::::::::::::::: keypoints
+
+- Faceting can identify errors or outliers in data
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+