Big_Data_Como_instalar_o_PySpark_no_Google_Colab.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/eduardoml93/PySpark-no-Google-Colab/blob/main/Big_Data_Como_instalar_o_PySpark_no_Google_Colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DTruZS5TPEvK"
      },
      "source": [
        "# Big Data: Como instalar o PySpark no Google Colab\n",
        "\n",
        "Como instalar o PySpark no Google Colab é uma dúvida comum entre aqueles que estão migrando seus projetos de Data Science para ambientes na nuvem.\n",
        "\n",
        "O termo Big Data está cada vez mais presente, e mesmo projetos pessoais podem assumir uma grande dimensionalidade devido à quantidade de dados disponíveis.\n",
        "\n",
        "Para analisar grandes volumes de dados, Big Data, com velocidade, o Apache Spark é uma ferramenta muito utilizada, dada a sua capacidade de processamento de dados e computação paralela.\n",
        "\n",
        "O Spark foi pensado para ser acessível, oferecendo diversas APIs e frameworks em Python, Scala, SQL e diversas outras linguagens.\n",
        "\n",
        "## PySpark no Google Colab\n",
        "\n",
        "PySpark é a interface alto nível que permite você conseguir acessar e usar o Spark por meio da linguagem Python. Usando o PySpark, você consegue escrever todo o seu código usando apenas o nosso estilo Python de escrever código.\n",
        "\n",
        "Já o Google Colab é uma ferramenta incrível, poderosa e gratuita – com suporte de GPU inclusive. Uma vez que roda 100% na nuvem, você não tem a necessidade de instalar qualquer coisa na sua própria máquina.\n",
        "\n",
        "No entanto, apesar da maioria das bibliotecas de Data Science estarem previamente instaladas no Colab, o mesmo não acontece com o PySpark. Para conseguir usar o PySpark é necessário alguns passos intermediários, que não são triviais para aqueles que estão começando.\n",
        "\n",
        "Dessa maneira, preparei um tutorial simples e direto ensinando a instalar as dependências e a biblioteca.\n",
        "\n",
        "## Instalando o PySpark no Google Colab\n",
        "\n",
        "Instalar o PySpark não é um processo direto como de praxe em Python. Não basta usar um pip install apenas. Na verdade, antes de tudo é necessário instalar dependências como o Java 8, Apache Spark 2.3.2 junto com o Hadoop 2.7."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cabkOXVRPYgq"
      },
      "source": [
        "A próxima etapa é configurar as variáveis de ambiente, pois isso habilita o ambiente do Colab a identificar corretamente onde as dependências estão rodando.\n",
        "\n",
        "Para conseguir “manipular” o terminal e interagir como ele, você pode usar a biblioteca os."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "L9hiSVSb0eiM"
      },
      "outputs": [],
      "source": [
        "# Instalando o Java\n",
        "!apt-get install openjdk-8-jdk-headless -qq > /dev/null"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "execution_count": 1,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "1-1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "4G-U50vu0e8w"
      },
      "outputs": [],
      "source": [
        "# Fazendo download\n",
        "!wget -q https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop2.7.tgz\n",
        "\n",
        "# Descompactando os arquivos\n",
        "!tar xf spark-3.1.2-bin-hadoop2.7.tgz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "RkzIGgBh0hcc"
      },
      "outputs": [],
      "source": [
        "# Importando a biblioteca os\n",
        "import os\n",
        "\n",
        "# Definindo a variável de ambiente do Java\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n",
        "\n",
        "# Definindo a variável de ambiente do Spark\n",
        "os.environ[\"SPARK_HOME\"] = \"/content/spark-3.1.2-bin-hadoop2.7\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "bDlmAcGZ0kem"
      },
      "outputs": [],
      "source": [
        "# instalando a findspark\n",
        "!pip install -q findspark"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "Jtf65JVy0zbv"
      },
      "outputs": [],
      "source": [
        "# Importando a findspark\n",
        "import findspark\n",
        "\n",
        "# Iniciando o findspark\n",
        "findspark.init()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 219
        },
        "id": "4bWJOLB501jY",
        "outputId": "13773459-dc2e-44cf-ca09-cdb347cc1a35"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "            <div>\n",
              "                <p><b>SparkSession - in-memory</b></p>\n",
              "                \n",
              "        <div>\n",
              "            <p><b>SparkContext</b></p>\n",
              "\n",
              "            <p><a href=\"http://3a3ada6fe91e:4040\">Spark UI</a></p>\n",
              "\n",
              "            <dl>\n",
              "              <dt>Version</dt>\n",
              "                <dd><code>v3.1.2</code></dd>\n",
              "              <dt>Master</dt>\n",
              "                <dd><code>local[*]</code></dd>\n",
              "              <dt>AppName</dt>\n",
              "                <dd><code>pyspark-shell</code></dd>\n",
              "            </dl>\n",
              "        </div>\n",
              "        \n",
              "            </div>\n",
              "        "
            ],
            "text/plain": [
              "<pyspark.sql.session.SparkSession at 0x7fdba49cd760>"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# importando o pacote necessário para iniciar uma seção Spark\n",
        "from pyspark.sql import SparkSession\n",
        "\n",
        "# iniciando o spark context\n",
        "sc = SparkSession.builder.master('local[*]').getOrCreate()\n",
        "\n",
        "# Verificando se a sessão foi criada\n",
        "sc"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Klur5aiRPdqx"
      },
      "source": [
        "Com tudo pronto, vamos rodar uma sessão local para testar se a instalação funcionou corretamente."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gpJqWggpPKXO",
        "outputId": "2dfc9517-691a-42d9-9ff2-d8f4da073827"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2023-01-05 02:52:00--  https://raw.githubusercontent.com/jonates/opendata/master/receita_federal/receita_federal_arrecadacao_por_UF_2020.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 6216 (6.1K) [text/plain]\n",
            "Saving to: ‘receita_federal_arrecadacao_por_UF_2020.csv.1’\n",
            "\n",
            "\r          receita_f   0%[                    ]       0  --.-KB/s               \rreceita_federal_arr 100%[===================>]   6.07K  --.-KB/s    in 0s      \n",
            "\n",
            "2023-01-05 02:52:00 (57.4 MB/s) - ‘receita_federal_arrecadacao_por_UF_2020.csv.1’ saved [6216/6216]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "\n",
        "\n",
        "# download do http para arquivo local\n",
        "# Fazendo download do arquivo\n",
        "!wget --verbose --show-progress --no-check-certificate https://raw.githubusercontent.com/jonates/opendata/master/receita_federal/receita_federal_arrecadacao_por_UF_2020.csv\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "4PixxXbq17-4"
      },
      "outputs": [],
      "source": [
        "# carregando um conjunto de dados que baixamos da internet\n",
        "receitafederal = sc.read.csv(\n",
        "    path = \"/content/receita_federal_arrecadacao_por_UF_2020.csv\", \n",
        "    inferSchema = True, \n",
        "    header = True,\n",
        "    sep = ';', \n",
        "    encoding = \"UTF-8\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sVzZ3xfn1-0B",
        "outputId": "1a3b7eaa-5729-4794-a480-d06efce7b832"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "pyspark.sql.dataframe.DataFrame"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Verificando o tipo de objeto criado\n",
        "type(receitafederal)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ml001ckR2BRz",
        "outputId": "572ef97c-7a69-4d1e-b741-d366b1781527"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "+---+------------+----+------------------------+------------------------+-------------+---------------------------+-------------+--------------+-------------------------------+-------------------------------+-------------------------+--------------+-----------------------------+--------------+-----------------+------------------------------------------------------+-----------------------------+\n",
            "| uf|      regiao| ano|imposto_sobre_importacao|imposto_sobre_exportacao|    ipi_total|imposto_sobre_a_renda_total|         irpf|          irpj|imposto_s_renda_retido_na_fonte|imposto_s_operacoes_financeiras|imposto_territorial_rural|        cofins|contribuicao_para_o_pis_pasep|          csll|cide_combustiveis|cpsss_contrib_p_o_plano_de_segurid_social_serv_publico|outras_receitas_administradas|\n",
            "+---+------------+----+------------------------+------------------------+-------------+---------------------------+-------------+--------------+-------------------------------+-------------------------------+-------------------------+--------------+-----------------------------+--------------+-----------------+------------------------------------------------------+-----------------------------+\n",
            "| AC|       Norte|2020|               985174,77|                   36,13|   4240199,15|               300199853,59|  62072028,88|  113486699,53|                   124641125,18|                     1897985,08|               1487344,08|  144057300,57|                  76934968,18|   68168473,35|                0|                                          104836396,45|                  14062545,18|\n",
            "| AL|    Nordeste|2020|             16616714,74|                19893,75|  72250365,28|              1033019967,29| 247567845,14|  340709902,24|                   444742219,91|                     3702912,85|               4421759,01|  729105725,54|                 249770494,48|  229004663,27|         132127,2|                                          338365145,08|                  48113882,82|\n",
            "| AM|       Norte|2020|            762232235,91|      -0,470000000000002| 169272676,24|              4468506183,61| 257090145,96| 2695103942,73|                  1516312094,92|                    44421129,59|               2348075,49| 4269533703,39|                1149824781,99| 1983534312,19|       14978664,1|                                          341440307,24|                 272633042,17|\n",
            "| AP|       Norte|2020|               309705,72|                   12,13|   2684901,14|               236338731,37|  52232105,57|  112525771,56|                    71580854,24|                     2864198,87|                797787,75|  106365959,63|                  62458965,04|   56187326,93|                0|                                          309464338,76|                   4004358,08|\n",
            "| BA|    Nordeste|2020|           1030620380,98|                30131,08|1583343870,65|               5997973282,7|  982824367,8| 2670174389,56|                  2344974525,34|                    25438592,62|               55878608,8| 5555103826,17|                1686822120,06| 1767716643,83|       2022454,51|                                          917452592,46|                   9234434,79|\n",
            "| CE|    Nordeste|2020|            712512834,54|                 2225,73| 652218015,22|              5315214147,03| 601900486,16| 2444764790,35|                  2268548870,52|                    18347619,28|               3743560,25| 3855584306,24|                1108484440,14| 1586572162,78|      40927710,14|                                          700664874,87|                  87941017,52|\n",
            "| DF|Centro-Oeste|2020|             38676559,11|                 2898,39|  81049965,17|             56077120890,97|1035328309,44| 7073534944,19|                 47968257637,34|                  2706081789,12|               2674418,58| 8459868980,28|                5217740369,54| 3574488387,95|           4779,6|                                           18682026689|                6724953354,73|\n",
            "| ES|     Sudeste|2020|           1903559527,65|                29138,38|2760357766,98|               4405048796,3|  532877760,4| 2388039971,85|                  1484131064,05|                    69631222,96|               9201413,05| 4357367700,69|                1238402181,87| 1236501444,16|       4969057,37|                                          423898629,98|                  25602174,44|\n",
            "| GO|Centro-Oeste|2020|            385559156,68|                36012,74| 424591468,17|              5342509382,11|1095425137,87| 2538282615,96|                  1708801628,28|                    50697839,86|             203141980,89| 3158060035,52|                1171321965,71|  1298410494,9|         58191,55|                                           602951176,5|                 162638156,15|\n",
            "| MA|    Nordeste|2020|             51723529,27|               -65446,29|  59786823,14|              1737367153,68| 515033922,64|  647479963,47|                   574853267,57|                     2523741,03|              12589302,25| 2335129747,52|                 712180155,03|  389379529,55|      90598250,59|                                          389881993,92|                  34193386,91|\n",
            "| MG|     Sudeste|2020|           1560969146,37|                17186,05|7450712670,68|             24596928656,02|3606118114,78| 13115582723,6|                  7875227817,64|                   564815589,35|             183172444,97|11925227612,16|                3878472117,53| 6337326709,79|          7359,21|                                         2369216972,77|                  57418517,38|\n",
            "| MS|Centro-Oeste|2020|               1731995,1|                12705,32|  86269901,57|              2340505130,09| 566646793,97| 1031285097,22|                    742573238,9|                    31569632,31|             297484227,44| 1226575545,39|                 488601312,05|  527512179,78|        458401,31|                                          331842783,42|                  25128935,11|\n",
            "| MT|Centro-Oeste|2020|             43440232,07|                 9563,46| 124278133,21|              3406941803,98| 656757408,43| 1666450758,32|                  1083733637,23|                    87145045,51|             260716716,88| 1822661624,44|                 697187912,34|  993536919,54|       2340329,87|                                          351822234,29|                  52954971,81|\n",
            "| PA|       Norte|2020|             98912193,42|                24666,11| 177838811,43|              2917164475,88| 407483520,99| 1364974772,35|                  1144706182,54|                   110031378,18|              30840910,54| 1907101405,68|                 712679134,57|  814984015,39|      28903601,29|                                          630461056,25|                  23479657,12|\n",
            "| PB|    Nordeste|2020|              75455623,5|               -10532,26|  94180485,04|              1678428897,46| 355210341,23|  550589116,38|                   772629439,85|                     8179192,34|               4485567,73|  876737394,87|                 320732511,48|  377234716,55|            17,08|                                           721192971,4|                  18996900,12|\n",
            "| PE|    Nordeste|2020|           1149002785,85|                71410,72| 1405505008,1|              5042629144,11| 809444049,26| 2128202623,49|                  2104982471,36|                    31874905,31|               7582875,08| 5526242288,74|                1576655430,66| 1426586017,85|      64100903,48|                                         1085021182,79|                 206953243,51|\n",
            "| PI|    Nordeste|2020|               328172,13|                 1558,31|  33654426,64|              1062146506,88| 235233389,02|  451289960,09|                   375623157,77|                     2862676,33|              19632280,83|  511120361,46|                 210171838,23|   278500292,1|                0|                                          315936497,84|                  16195984,82|\n",
            "| PR|         Sul|2020|           3964526727,39|                54601,49|4243592969,64|             17606943216,98|3070982622,46| 8636121208,99|                  5899839385,53|                   418626409,71|             190291734,39|13800656694,15|                3714652395,95| 4265783037,66|      90634199,62|                                         1061530802,22|                 422263624,47|\n",
            "| RJ|     Sudeste|2020|           2772035812,15|                 5102,54|2988947657,17|              71462739179,1|5396899630,21|33023309844,26|                 33042529704,63|                  1241887034,59|              13650520,13|31775281577,45|                 9212986524,4|16896065806,46|    1048318687,99|                                         3773448013,47|                 6169915876,1|\n",
            "| RN|    Nordeste|2020|             13279408,52|                  233,05|  52643852,25|              1651943412,64| 304628827,55|  631736550,23|                   715578034,86|                     7804679,88|               2857411,45|  927910133,31|                 356272954,17|  382564725,99|           633,99|                                          572728894,35|                  31903990,99|\n",
            "+---+------------+----+------------------------+------------------------+-------------+---------------------------+-------------+--------------+-------------------------------+-------------------------------+-------------------------+--------------+-----------------------------+--------------+-----------------+------------------------------------------------------+-----------------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Espiando o dataset\n",
        "receitafederal.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AB8ug_xC480G",
        "outputId": "011698cb-b98f-460f-fa67-df15e2a4e772"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "(28, 18)\n"
          ]
        }
      ],
      "source": [
        "#linhas e colunas\n",
        "print((receitafederal.count(), len(receitafederal.columns)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "249dnkKa2GL5",
        "outputId": "f78481cf-cf20-4ebd-ebaa-d513cc19d9e5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- uf: string (nullable = true)\n",
            " |-- regiao: string (nullable = true)\n",
            " |-- ano: integer (nullable = true)\n",
            " |-- imposto_sobre_importacao: string (nullable = true)\n",
            " |-- imposto_sobre_exportacao: string (nullable = true)\n",
            " |-- ipi_total: string (nullable = true)\n",
            " |-- imposto_sobre_a_renda_total: string (nullable = true)\n",
            " |-- irpf: string (nullable = true)\n",
            " |-- irpj: string (nullable = true)\n",
            " |-- imposto_s_renda_retido_na_fonte: string (nullable = true)\n",
            " |-- imposto_s_operacoes_financeiras: string (nullable = true)\n",
            " |-- imposto_territorial_rural: string (nullable = true)\n",
            " |-- cofins: string (nullable = true)\n",
            " |-- contribuicao_para_o_pis_pasep: string (nullable = true)\n",
            " |-- csll: string (nullable = true)\n",
            " |-- cide_combustiveis: string (nullable = true)\n",
            " |-- cpsss_contrib_p_o_plano_de_segurid_social_serv_publico: string (nullable = true)\n",
            " |-- outras_receitas_administradas: string (nullable = true)\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Verificando o schema() deste sparkdataframe\n",
        "receitafederal.printSchema()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "jYrWz6lF2b0b"
      },
      "outputs": [],
      "source": [
        "# importando os métodos com funções para transformações de variáveis\n",
        "from pyspark.sql.functions import *"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jtj3GaUW2erG",
        "outputId": "04d77d96-5764-4ce1-d28e-9f9fb7a6f340"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "root\n",
            " |-- irpf: float (nullable = true)\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Transformando o atributo irpf em numerica\n",
        "receitafederal = receitafederal.withColumn(\n",
        "    colName = 'irpf', \n",
        "    col = regexp_replace('irpf',',','.').cast('float')\n",
        "    )\n",
        "\n",
        "# Inspecionando o resultado\n",
        "receitafederal.select('irpf').printSchema()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qVTjShMb2k6R",
        "outputId": "79641593-2fa5-4b73-d088-a92fc2550dd3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "+------------+--------------+\n",
            "|      regiao|     sum(irpf)|\n",
            "+------------+--------------+\n",
            "|Centro-Oeste| 3.354157696E9|\n",
            "|    Nordeste| 4.303029696E9|\n",
            "|       Norte| 1.404179308E9|\n",
            "|     Sudeste|2.496098528E10|\n",
            "|         Sul| 7.380957184E9|\n",
            "|       Total|4.140331008E10|\n",
            "+------------+--------------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Verificando o total do irpf por Região do Brasil\n",
        "receitafederal.groupBy('regiao').sum('irpf').orderBy('regiao').show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KTgH_nmk3fUH",
        "outputId": "0ed2ab74-9b29-4b6a-847a-469759c2ef9e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "28"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "receitafederal.count()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Sv7vgRRp3iLz",
        "outputId": "09e9e63a-2aaf-4603-b2be-4b1f38db9b22"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "+---+------+----+------------------------+------------------------+------------+---------------------------+------------+-------------+-------------------------------+-------------------------------+-------------------------+-------------+-----------------------------+-------------+-----------------+------------------------------------------------------+-----------------------------+\n",
            "| uf|regiao| ano|imposto_sobre_importacao|imposto_sobre_exportacao|   ipi_total|imposto_sobre_a_renda_total|        irpf|         irpj|imposto_s_renda_retido_na_fonte|imposto_s_operacoes_financeiras|imposto_territorial_rural|       cofins|contribuicao_para_o_pis_pasep|         csll|cide_combustiveis|cpsss_contrib_p_o_plano_de_segurid_social_serv_publico|outras_receitas_administradas|\n",
            "+---+------+----+------------------------+------------------------+------------+---------------------------+------------+-------------+-------------------------------+-------------------------------+-------------------------+-------------+-----------------------------+-------------+-----------------+------------------------------------------------------+-----------------------------+\n",
            "| AC| Norte|2020|               985174,77|                   36,13|  4240199,15|               300199853,59| 6.2072028E7| 113486699,53|                   124641125,18|                     1897985,08|               1487344,08| 144057300,57|                  76934968,18|  68168473,35|                0|                                          104836396,45|                  14062545,18|\n",
            "| AM| Norte|2020|            762232235,91|      -0,470000000000002|169272676,24|              4468506183,61|2.57090144E8|2695103942,73|                  1516312094,92|                    44421129,59|               2348075,49|4269533703,39|                1149824781,99|1983534312,19|       14978664,1|                                          341440307,24|                 272633042,17|\n",
            "| AP| Norte|2020|               309705,72|                   12,13|  2684901,14|               236338731,37| 5.2232104E7| 112525771,56|                    71580854,24|                     2864198,87|                797787,75| 106365959,63|                  62458965,04|  56187326,93|                0|                                          309464338,76|                   4004358,08|\n",
            "| PA| Norte|2020|             98912193,42|                24666,11|177838811,43|              2917164475,88| 4.0748352E8|1364974772,35|                  1144706182,54|                   110031378,18|              30840910,54|1907101405,68|                 712679134,57| 814984015,39|      28903601,29|                                          630461056,25|                  23479657,12|\n",
            "| RO| Norte|2020|               591043,93|                  475,91| 71418691,26|               884239765,76|2.00061712E8|    378899086|                    305278964,8|                    35211582,71|              10935235,68| 421537338,79|                 188190433,38| 230838315,45|                0|                                          336764426,83|                  13214541,73|\n",
            "| RR| Norte|2020|               124822,63|                  518,06|    844259,6|               290206138,05| 5.9049432E7|  122499930,2|                   108656775,59|                      236780,91|               2207509,14| 124477438,83|                  80780659,81|  72391313,04|                0|                                          184449438,95|                   4571969,78|\n",
            "| TO| Norte|2020|                49179,69|                 4108,64| 20380294,27|               809924133,07|3.66190368E8| 258035182,57|                   185698584,05|                     1932185,55|              33815426,64| 346525036,29|                 164303706,06| 160580142,93|         16870,88|                                          150708466,76|                  11524516,07|\n",
            "+---+------+----+------------------------+------------------------+------------+---------------------------+------------+-------------+-------------------------------+-------------------------------+-------------------------+-------------+-----------------------------+-------------+-----------------+------------------------------------------------------+-----------------------------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "receitafederal.filter(receitafederal.regiao == 'Norte').show(10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gMtSrCVj4pqe",
        "outputId": "c7aa6f95-962c-4ce3-a194-1f620a778f8a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "+---+-------+----+------------------------+------------------------+--------------+---------------------------+-------------+--------------+-------------------------------+-------------------------------+-------------------------+--------------+-----------------------------+--------------+-----------------+------------------------------------------------------+-----------------------------+\n",
            "| uf| regiao| ano|imposto_sobre_importacao|imposto_sobre_exportacao|     ipi_total|imposto_sobre_a_renda_total|         irpf|          irpj|imposto_s_renda_retido_na_fonte|imposto_s_operacoes_financeiras|imposto_territorial_rural|        cofins|contribuicao_para_o_pis_pasep|          csll|cide_combustiveis|cpsss_contrib_p_o_plano_de_segurid_social_serv_publico|outras_receitas_administradas|\n",
            "+---+-------+----+------------------------+------------------------+--------------+---------------------------+-------------+--------------+-------------------------------+-------------------------------+-------------------------+--------------+-----------------------------+--------------+-----------------+------------------------------------------------------+-----------------------------+\n",
            "| SP|Sudeste|2020|          21652765043,51|             21521909,64|23948731180,87|            188713634882,57|1.54250895E10|74437932737,85|                 98850612389,77|                 15553293028,64|             349800091,06|95234143515,96|               23437246529,32|37102275970,48|      97833902,56|                                         2139815221,07|                6274441857,51|\n",
            "+---+-------+----+------------------------+------------------------+--------------+---------------------------+-------------+--------------+-------------------------------+-------------------------------+-------------------------+--------------+-----------------------------+--------------+-----------------+------------------------------------------------------+-----------------------------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "receitafederal.filter(receitafederal.uf == 'SP').show(10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1syvfJjP4zVm"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yj5eswa9ZzWS"
      },
      "source": [
        "## Big Data e Python\n",
        "\n",
        "A biblioteca PySpark permite você criar seu servidor Apache Spark, trabalhar com grandes volumes de dados e até mesmo fazer streaming em tempo real.\n",
        "\n",
        "Na minha opinião, o Spark é o melhor framework para trabalhar com Big Data. Tenha certeza que o PySpark vai te ajudar muito ao criar uma interface Python que permita a comunicação entre seu projeto e o servidor.\n",
        "\n",
        "Neste artigo, o meu objetivo foi unicamente apresentar a biblioteca, além de ensinar como você pode instalá-la em um ambiente de nuvem gratuito, o Google Colab. Aproveite e comece a usar hoje mesmo 🙂"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.0"
    },
    "vscode": {
      "interpreter": {
        "hash": "5f93d240d485154fd43f86cec9739ae95568bc27aa8b35854060c2e9b48b407b"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}