loading-spark-koalas-on-colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "loading-spark-on-colab.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmodzg2X148u",
        "colab_type": "text"
      },
      "source": [
        "# Setup\n",
        "\n",
        "This is a quick piece of code showing how to fire [Apache Spark](https://spark.apache.org/) and then [Koalas](https://koalas.readthedocs.io/en/latest/index.html) on Google Colab. Koalas \"makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark\" (word of the creators).\n",
        "\n",
        "First, we install the required packages and set environment variables:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cuoVDPSB2IAw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!apt-get install openjdk-8-jdk-headless -qq > /dev/null\n",
        "!wget -q https://archive.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz\n",
        "!tar xf spark-2.4.4-bin-hadoop2.7.tgz\n",
        "!pip install -q findspark\n",
        "!pip install koalas\n",
        "\n",
        "import os\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n",
        "os.environ[\"SPARK_HOME\"] = \"/content/spark-2.4.4-bin-hadoop2.7\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MljcOoQe2Qvk",
        "colab_type": "text"
      },
      "source": [
        "# Initializing\n",
        "\n",
        "Next, we start a Spark session. This step is needed in order to interact with the Spark API:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PnZyInNSKvrD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from pyspark.sql.session import SparkSession\n",
        "\n",
        "spark = SparkSession.builder\\\n",
        "    .master('local[*]')\\\n",
        "    .getOrCreate()"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D5vyFs3ecMkU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import databricks.koalas as ks"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Gm8wBgheE8x",
        "colab_type": "text"
      },
      "source": [
        "Now, we show and example of how to move data around the different frameworks:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bG9YJ-_3cRdD",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "30e99aa0-2f4a-4ae5-cead-c14c93de0d0e"
      },
      "source": [
        "# A simple pandas DataFrame\n",
        "df = pd.DataFrame({\n",
        "    'numbers': [1,2,3],\n",
        "    'strings': ['a','b','c']\n",
        "})\n",
        "type(df)"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pandas.core.frame.DataFrame"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "54FtoCQCcumE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "089a019c-beb2-43f0-9bff-ba2611214f5b"
      },
      "source": [
        "# A Spark DataFrame/RDD, now ready for big data analysis\n",
        "sdf = spark.createDataFrame(df)\n",
        "type(sdf)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pyspark.sql.dataframe.DataFrame"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "InK8dd9gc5KO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "a53a6772-8857-4015-8401-c16e3c799ca5"
      },
      "source": [
        "# A Koalas DataFrame! Now you can use most of the pandas' idiom to work with big data.\n",
        "kdf = sdf.to_koalas()\n",
        "type(kdf)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "databricks.koalas.frame.DataFrame"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    }
  ]
}