part2/Part 2 - Exercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2CtO7uNHe4ZE"
   },
   "source": [
    "# Part 2 - Exercises - Data Transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "editable": true,
    "executionInfo": {
     "elapsed": 8834,
     "status": "ok",
     "timestamp": 1717156830871,
     "user": {
      "displayName": "Jon Mease",
      "userId": "02098018605502091895"
     },
     "user_tz": 240
    },
    "id": "F1QcNjRcxgUj",
    "outputId": "49d9ddef-2dfa-425c-8882-f439ca03e464",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import altair as alt\n",
    "import pandas as pd\n",
    "from vega_datasets import data\n",
    "print(\"The installed Vega-Altair version is \" + alt.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lfVp1EevXyZ4"
   },
   "source": [
    "## Diamonds Exercise\n",
    "Let's explore the diamonds dataset from the Seaborn sample datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "executionInfo": {
     "elapsed": 576,
     "status": "ok",
     "timestamp": 1717156832691,
     "user": {
      "displayName": "Jon Mease",
      "userId": "02098018605502091895"
     },
     "user_tz": 240
    },
    "id": "kJLby6lEYFVW",
    "outputId": "50d48bd1-fd3a-413f-8b04-4a8fb62d2506"
   },
   "outputs": [],
   "source": [
    "diamonds_url = \"../resources/datasets/diamonds.csv\"\n",
    "diamonds = pd.read_csv(diamonds_url)\n",
    "diamonds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "brgLlJ9HYI7E"
   },
   "source": [
    "Here are descriptions of the dataset columns:\n",
    "\n",
    "|Variable|Description|Values|\n",
    "|--- |--- |--- |\n",
    "|carat|weight of the diamond|0.2-5.01|\n",
    "|cut|quality of the cut|Fair, Good, Very Good, Premium, Ideal|\n",
    "|color|diamond color|J (worst) to D (best)|\n",
    "|clarity|measurement of how clear the diamond is|I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best)|\n",
    "|depth|total depth percentage|43-79|\n",
    "|table|width of top of diamond relative to widest point|43-95|\n",
    "|price|price in US dollars|\\$326-\\$18,823|\n",
    "|x|length in mm|0-10.74|\n",
    "|y|width in mm|0-58.9|\n",
    "|z|depth in mm|0-31.8|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yEyNNZiWYYZ9"
   },
   "source": [
    "### Simple scatter\n",
    "\n",
    "Create a simple scatterplot with `price` on the y-axis and `carat` on the x-axis. Use a [circle mark](https://altair-viz.github.io/user_guide/marks/circle.html) and lower the `size` and `opacity` properties to reduce overplotting.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qJRDcwkQew0g"
   },
   "source": [
    "> **Note:** Because this dataset has more than 5,000 rows, you will get a `MaxRowsError` by default. Follow the instructions in the error message to enable the \"vegafusion\" data transformer to address this error, which will optimize the generated Vega specification by removing unused columns and evaluating data transformations in the Python kernel. This raises the row limit to 100k, and the limit is enforced after any data aggregations have been applied.\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Image)</summary>\n",
    "  <img src=\"../resources/images/part2/diamonds1.png\">\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "\n",
    "  ```python\n",
    "alt.data_transformers.enable(\"vegafusion\")\n",
    "alt.Chart(diamonds).mark_circle(opacity=0.2, size=10).encode(\n",
    "    alt.X(\"carat\"),\n",
    "    alt.Y(\"price\"),\n",
    ")\n",
    "  ```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z0azmrs6hzcf"
   },
   "source": [
    "### Simple histogram\n",
    "Create a histogram of the `carat` column. This will require enabling binning on the `x` encoding, and using `count()` as the y encoding.\n",
    "\n",
    "Click \"Open in Vega Editor\" from the chart's dropdown menu and notice how the dataset included in the spec is already binned and aggregated (by VegaFusion).\n",
    "\n",
    "Use `chart.transformed_data()` to extract the binned and aggregated data as a pandas DataFrame.\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Image)</summary>\n",
    "  <img src=\"../resources/images/part2/diamonds2.png\">\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "\n",
    "  ```python\n",
    "alt.data_transformers.enable(\"vegafusion\")\n",
    "chart = alt.Chart(diamonds).mark_bar().encode(\n",
    "    alt.X(\"carat\").bin(),\n",
    "    alt.Y(\"count()\"),\n",
    ")\n",
    "chart.show()\n",
    "chart.transformed_data()\n",
    "  ```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ahWMG8jrixKM"
   },
   "source": [
    "What happens if you only use `count()` as the `y` encoding without binning `x`?\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "  The bars are centered on each unique value of `carat`. The bars have a fixed width, so they may overlap each other. When binning is enabled, Vega-Altair automatically sets the bar width to match the bin intervals.\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4irULMfAo8T_"
   },
   "source": [
    "### Aggregated Bar Chart\n",
    "\n",
    "Create a vertical bar chart showing the average `price` for each `cut` category. As before, use `chart.transformed_data()` to extract the binned and aggregated data as a pandas DataFrame.\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Image)</summary>\n",
    "  <img src=\"../resources/images/part2/diamonds3.png\">\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "\n",
    "  ```python\n",
    "chart = alt.Chart(diamonds).mark_bar().encode(\n",
    "    alt.X('cut'),\n",
    "    alt.Y('average(price)')\n",
    ")\n",
    "chart.show()\n",
    "chart.transformed_data()\n",
    "  ```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GbBs5rG-s2P_"
   },
   "source": [
    "### Heatmap\n",
    "\n",
    "Create a heatmap with `cut` as the x encoding, `color` and the y-encoding, and `average(price)` as the heatmap color.\n",
    "\n",
    "Then configure the scales so that the best cut and quality is in the lower right corner, and the worst cut and quality is in the upper left corner.\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Image)</summary>\n",
    "  <img src=\"../resources/images/part2/diamonds4.png\">\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "\n",
    "  ```python\n",
    "alt.Chart(diamonds).mark_rect().encode(\n",
    "    alt.X('cut').scale(domain=[\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]),\n",
    "    alt.Y('color').sort(\"descending\"),\n",
    "    alt.Color('average(price)')\n",
    ")\n",
    "  ```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtered Heatmap\n",
    "Repeat the Heatmap from the previous example, but this time use a filter transform to filter to diamonds larger than 1.5 carats. What do you notice about the price distribution of these diamonds?\n",
    "\n",
    "For more information on the filter transform, see the [Vega-Altair documentation](https://altair-viz.github.io/user_guide/transform/filter.html).\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Image)</summary>\n",
    "  <img src=\"../resources/images/part2/diamonds5.png\">\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "\n",
    "  ```python\n",
    "alt.Chart(diamonds).mark_rect().transform_filter(\"datum.carat > 1.5\").encode(\n",
    "    alt.X('cut').scale(domain=[\"Fair\", \"Good\", \"Very Good\", \"Premium\", \"Ideal\"]),\n",
    "    alt.Y('color').sort(\"descending\"),\n",
    "    alt.Color('average(price)')\n",
    ")\n",
    "  ```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ureI59UJAK2i"
   },
   "source": [
    "## Spotify Exercise\n",
    "\n",
    "We'll use the Spotify dataset (already seen in the Part 1 Exercises) to explore using time-based binning.\n",
    "\n",
    "We'll load the dataset, convert the \"Release Date\" column to a pandas `datetime`, and filter it to include songs released in 2018 and later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 214,
     "status": "ok",
     "timestamp": 1717156831080,
     "user": {
      "displayName": "Jon Mease",
      "userId": "02098018605502091895"
     },
     "user_tz": 240
    },
    "id": "NvJghWpxB4KU"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "spotify_url = \"../resources/datasets/spotify.csv\"\n",
    "df = pd.read_csv(spotify_url, na_values=\" \")\n",
    "df['Release Date'] = pd.to_datetime(df['Release Date'], format=\"mixed\")\n",
    "df = df[df['Release Date'] >= \"2018\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time-binned Histogram (Yearly)\n",
    "\n",
    "Create a histogram of songs releases per year.\n",
    "\n",
    "See the Vega-Altair [TimeUnit documentaion page](https://altair-viz.github.io/user_guide/transform/timeunit.html#timeunit-within-encoding) for reference.\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Image)</summary>\n",
    "  <img src=\"../resources/images/part2/spotify1.png\">\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "\n",
    "  ```python\n",
    "alt.Chart(df, width=500).mark_bar().encode(\n",
    "    alt.X(\"year(Release Date)\"),\n",
    "    alt.Y(\"count()\")\n",
    ")\n",
    "  ```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time-binned Histogram (Monthly)\n",
    "\n",
    "Create a histogram of song releases per month.\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Image)</summary>\n",
    "  <img src=\"../resources/images/part2/spotify2.png\">\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "\n",
    "  ```python\n",
    "alt.Chart(df, width=500).mark_bar().encode(\n",
    "    alt.X(\"yearmonth(Release Date)\"),\n",
    "    alt.Y(\"count()\")\n",
    ")\n",
    "  ```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's the difference between the `month` and `yearmonth` timeunits?\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "    \n",
    "The `month` time unit will group together songs released in January across every year, whereas the `yearmonth` will only group together songs released in January of the same year.\n",
    "    \n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Histogram of Top Artists\n",
    "\n",
    "Here we will work with a subset of the Spotify dataset, corresponding only to songs by the Top 5 most frequently occurring artists in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_top = df[df[\"Artist\"].isin(df[\"Artist\"].value_counts()[:5].index)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the following corresponds to many bars stacked on top of each other.  (So the heights of these bars are the total stream counts for the artists, combined.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(df_top).mark_bar().encode(\n",
    "    alt.X(\"Artist\"),\n",
    "    alt.Y(\"Streams\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change the histogram so that instead, each column contains a single bar, corresponding to the maximum stream count across all that artist's songs.\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "\n",
    "  ```python\n",
    "alt.Chart(df_top).mark_bar().encode(\n",
    "    alt.X(\"Artist\"),\n",
    "    alt.Y(\"max(Streams)\")\n",
    ")\n",
    "  ```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example bar chart indicating frequencies of songs with a specified chord."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(df_top).mark_bar().encode(\n",
    "    alt.X(\"Chord\"),\n",
    "    alt.Y(\"count()\"),\n",
    "    alt.Color(\"Artist\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the previous chart as a template, make the following changes.  (We recommend preserving the above template, and making a new chart specification below.)\n",
    "* We want the bars to be arranged by artist, not by chord.  (So a single bar should contain only information about a single artist.)\n",
    "* Colors should be based on chord.  Choose an appropriate color scheme.\n",
    "* Rescale the values so that all bars have the same height.  (**Hint**.  Browse the bar chart section of the [example gallery](https://altair-viz.github.io/gallery/index.html) for model chart.  The solution is not meant to be complicated, just a few words.)\n",
    "* Choose an appropriate axis title, since it no longer represents counts.\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Image)</summary>\n",
    "  <img src=\"../resources/images/part2/spotify3.png\">\n",
    "</details>\n",
    "\n",
    "<details>\n",
    "  <summary>(Show Answer)</summary>\n",
    "    \n",
    "  ```python\n",
    "alt.Chart(df_top).mark_bar().encode(\n",
    "    alt.X(\"Artist\"),\n",
    "    alt.Y(\"count()\").stack(\"normalize\").title(\"Percent of songs\"),\n",
    "    alt.Color(\"Chord\").scale(scheme=\"purpleorange\")\n",
    ")\n",
    "  ```\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keep exploring!\n",
    "Check out all of the transformation types that Vega-Altair supports in [the documentation](https://altair-viz.github.io/user_guide/transform/index.html).  Pick one we haven't discussed yet and apply it to the spotify dataset. For example, use the density transform on song tempo.\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}