7-cow-bucket-index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "44288d4b-1602-44ad-9881-f0e510a30d76",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Apache Hudi Core Conceptions (7) - COW: Bucket Index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d6261e3-fa15-410d-8472-29dd02f86b10",
   "metadata": {},
   "source": [
    "## 1. Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "57b9edaf-90a7-4b7c-b072-bc8c50c2e045",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:55:21.435904Z",
     "iopub.status.busy": "2023-03-07T05:55:21.435564Z",
     "iopub.status.idle": "2023-03-07T05:55:21.452472Z",
     "shell.execute_reply": "2023-03-07T05:55:21.451816Z",
     "shell.execute_reply.started": "2023-03-07T05:55:21.435878Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Current session configs: <tt>{'conf': {'spark.jars': 'hdfs:///tmp/hudi-spark-bundle.jar', 'spark.serializer': 'org.apache.spark.serializer.KryoSerializer', 'spark.sql.extensions': 'org.apache.spark.sql.hudi.HoodieSparkSessionExtension', 'spark.sql.catalog.spark_catalog': 'org.apache.spark.sql.hudi.catalog.HoodieCatalog'}, 'proxyUser': 'user_bluishglc', 'kind': 'spark'}</tt><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "No active sessions."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%configure -f\n",
    "{\n",
    "    \"conf\" : {\n",
    "        \"spark.jars\":\"hdfs:///tmp/hudi-spark-bundle.jar\",            \n",
    "        \"spark.serializer\":\"org.apache.spark.serializer.KryoSerializer\",\n",
    "        \"spark.sql.extensions\":\"org.apache.spark.sql.hudi.HoodieSparkSessionExtension\",\n",
    "        \"spark.sql.catalog.spark_catalog\":\"org.apache.spark.sql.hudi.catalog.HoodieCatalog\"\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "35143e14-7b24-450e-b029-c6ae0829e98f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:55:21.454105Z",
     "iopub.status.busy": "2023-03-07T05:55:21.453768Z",
     "iopub.status.idle": "2023-03-07T05:55:25.823349Z",
     "shell.execute_reply": "2023-03-07T05:55:25.822466Z",
     "shell.execute_reply.started": "2023-03-07T05:55:21.454070Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--   1 emr-notebook hdfsadmingroup   61421977 2023-03-07 05:55 /tmp/hudi-spark-bundle.jar\n",
      "/home/emr-notebook/hudi-stat.sh\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "# deploy hudi bundle jar\n",
    "hdfs dfs -copyFromLocal -f /usr/lib/hudi/hudi-spark-bundle.jar /tmp/hudi-spark-bundle.jar\n",
    "hdfs dfs -ls /tmp/hudi-spark-bundle.jar\n",
    "# deploy hudi-stat.sh - a utility shell script \n",
    "wget https://github.com/bluishglc/hudi-core-conceptions/releases/download/v1.0/hudi-stat.sh -O ~/hudi-stat.sh &>/dev/null\n",
    "chmod a+x ~/hudi-stat.sh\n",
    "ls ~/hudi-stat.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6bb44084-8afe-47fe-9454-b5a496275808",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:55:25.825347Z",
     "iopub.status.busy": "2023-03-07T05:55:25.825096Z",
     "iopub.status.idle": "2023-03-07T05:55:25.831174Z",
     "shell.execute_reply": "2023-03-07T05:55:25.830253Z",
     "shell.execute_reply.started": "2023-03-07T05:55:25.825323Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "table {float:left}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    "table {float:left}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2b219f4-941e-4be6-bbb1-2f0bd082d8b0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-25T02:57:45.037101Z",
     "iopub.status.busy": "2023-02-25T02:57:45.036681Z",
     "iopub.status.idle": "2023-02-25T02:58:07.353493Z",
     "shell.execute_reply": "2023-02-25T02:58:07.352596Z",
     "shell.execute_reply.started": "2023-02-25T02:57:45.037060Z"
    },
    "tags": []
   },
   "source": [
    "## 2. COW + BUCKET INDEX (SIMPLE HASHING)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49a4a7a6-2ee3-4e1d-a219-74e73baa8e0e",
   "metadata": {
    "tags": []
   },
   "source": [
    "Note：for SIMPLE BUCKET INDEX， there is no limit to the size of files (<=120MB), in other words, The `hoodie.parquet.max.file.size because` and `hoodie.parquet.small.file.limit` do NOT work under SIMPLE BUCKET INDEX, because the buckets number is fixed, so file groups number is fixed, it is NOT allowed to split any new file groups, otherwise, the mapping ralationship between record key and fileId will be broken."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "782f59e3-4e87-4ef2-a8e9-c379f2c1fbdf",
   "metadata": {},
   "source": [
    "KEY|VALUE\n",
    ":---|:---\n",
    "hoodie.index.type|BUCKET\n",
    "hoodie.copyonwrite.record.size.estimate|219"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93aa27d6-4726-4e0f-9e11-aa753a7f8124",
   "metadata": {},
   "source": [
    "### 2.1. Create Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fda7649c-b670-4c0f-a9b0-ee6252addaf1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:55:25.832748Z",
     "iopub.status.busy": "2023-03-07T05:55:25.832524Z",
     "iopub.status.idle": "2023-03-07T05:55:55.386258Z",
     "shell.execute_reply": "2023-03-07T05:55:55.385522Z",
     "shell.execute_reply.started": "2023-03-07T05:55:25.832726Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting Spark application\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tbody><tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>User</th><th>Current session?</th></tr><tr><td>15</td><td>application_1678096020253_0035</td><td>spark</td><td>idle</td><td><a target=\"_blank\" href=\"http://ip-10-0-129-209.ec2.internal:20888/proxy/application_1678096020253_0035/\" class=\"emr-proxy-link j-6V65AY8X5VHR application_1678096020253_0035\" emr-resource=\"j-6V65AY8X5VHR\n",
       "\" application-id=\"application_1678096020253_0035\">Link</a></td><td><a target=\"_blank\" href=\"http://ip-10-0-132-108.ec2.internal:8042/node/containerlogs/container_1678096020253_0035_01_000001/livy\">Link</a></td><td>None</td><td>✔</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparkSession available as 'spark'.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eaf809b83ea3493e98ef9a84f0952bb5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(HTML(value='Type:'), Button(description='Table', layout=Layout(width='70px'), st…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4310e2106de3419189d2c3e65abc5595",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "set TABLE_NAME=reviews_cow_bucket_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "26bcfe85-15a5-479a-a960-293243683d47",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:55:55.387904Z",
     "iopub.status.busy": "2023-03-07T05:55:55.387456Z",
     "iopub.status.idle": "2023-03-07T05:55:55.392597Z",
     "shell.execute_reply": "2023-03-07T05:55:55.391847Z",
     "shell.execute_reply.started": "2023-03-07T05:55:55.387871Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: TABLE_NAME=reviews_cow_bucket_1\n"
     ]
    }
   ],
   "source": [
    "%env TABLE_NAME=reviews_cow_bucket_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f80715f0-6220-4639-92df-14b3ad6f2448",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:55:55.393701Z",
     "iopub.status.busy": "2023-03-07T05:55:55.393481Z",
     "iopub.status.idle": "2023-03-07T05:55:56.374396Z",
     "shell.execute_reply": "2023-03-07T05:55:56.373478Z",
     "shell.execute_reply.started": "2023-03-07T05:55:55.393680Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37eb6e92205340deb94da10dc897ad95",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(HTML(value='Type:'), Button(description='Table', layout=Layout(width='70px'), st…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5432bdca1e2a452cb83f88be2b7597ba",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "set TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bucket_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fbd9aa12-bff5-4335-9aa8-48dc616ab10f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:55:56.377739Z",
     "iopub.status.busy": "2023-03-07T05:55:56.377208Z",
     "iopub.status.idle": "2023-03-07T05:55:56.383103Z",
     "shell.execute_reply": "2023-03-07T05:55:56.382277Z",
     "shell.execute_reply.started": "2023-03-07T05:55:56.377702Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bucket_1\n"
     ]
    }
   ],
   "source": [
    "%env TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bucket_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e115d65b-4315-4761-b59a-ed2021729319",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:55:56.385335Z",
     "iopub.status.busy": "2023-03-07T05:55:56.384714Z",
     "iopub.status.idle": "2023-03-07T05:56:00.349668Z",
     "shell.execute_reply": "2023-03-07T05:56:00.348907Z",
     "shell.execute_reply.started": "2023-03-07T05:55:56.385298Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reviews_cow_bucket_1\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "echo $(basename $TABLE_PATH)\n",
    "aws s3 rm $TABLE_PATH --recursive &>/dev/null\n",
    "rm -rf ~/${TABLE_NAME}\n",
    "sleep 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "67cff72d-f5e9-49cd-8b1e-d5047e9658b5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:56:00.351536Z",
     "iopub.status.busy": "2023-03-07T05:56:00.351164Z",
     "iopub.status.idle": "2023-03-07T05:56:03.720555Z",
     "shell.execute_reply": "2023-03-07T05:56:03.719598Z",
     "shell.execute_reply.started": "2023-03-07T05:56:00.351497Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6f6bc54070584c51b224fbd4e6d164c7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7083a0f9bd94f0c9e45981f37538091",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "drop table if exists ${TABLE_NAME}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "44efbff8-90f9-4468-b029-abea6ab50f68",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:56:03.722163Z",
     "iopub.status.busy": "2023-03-07T05:56:03.721915Z",
     "iopub.status.idle": "2023-03-07T05:56:07.067293Z",
     "shell.execute_reply": "2023-03-07T05:56:07.066417Z",
     "shell.execute_reply.started": "2023-03-07T05:56:03.722140Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6ce92bf82e5b43eeabf8173abea08aee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d4bf8da71114bb79143d199165e5a6d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "create table if not exists ${TABLE_NAME}(\n",
    "    review_id string, \n",
    "    star_rating long, \n",
    "    review_body string, \n",
    "    review_date date, \n",
    "    year long,\n",
    "    timestamp long,\n",
    "    parity int\n",
    ")\n",
    "using hudi\n",
    "location '${TABLE_PATH}'\n",
    "partitioned by (parity)\n",
    "options ( \n",
    "    type = 'cow',  \n",
    "    primaryKey = 'review_id', \n",
    "    preCombineField = 'timestamp',\n",
    "    hoodie.index.type = 'BUCKET',\n",
    "    hoodie.bucket.index.num.buckets = '2'\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04d33594-ef3a-4d56-a9a0-1ccb34dbd6c0",
   "metadata": {},
   "source": [
    "### 2.2. Batch 1 - Insert ( 104 MB / FileGroup )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e74101b9-7d41-4c4a-9b60-cf262c5930bf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:56:07.069007Z",
     "iopub.status.busy": "2023-03-07T05:56:07.068664Z",
     "iopub.status.idle": "2023-03-07T05:57:04.559153Z",
     "shell.execute_reply": "2023-03-07T05:57:04.558432Z",
     "shell.execute_reply.started": "2023-03-07T05:56:07.068971Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e4de71fb3424f7f8e107da6f6ba75eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "566966de4b6a49188ffb9fe8ee6a52a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "insert into \n",
    "    ${TABLE_NAME}\n",
    "select \n",
    "    review_id, \n",
    "    star_rating, \n",
    "    review_body, \n",
    "    review_date, \n",
    "    year,\n",
    "    unix_timestamp(current_timestamp()) as timestamp,\n",
    "    mod(crc32(review_id), 2) as parity\n",
    "from\n",
    "    reviews\n",
    "where\n",
    "    year = 2007"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "62e25150-86db-49f0-b3ca-651ee7ecbb08",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:57:04.560344Z",
     "iopub.status.busy": "2023-03-07T05:57:04.560110Z",
     "iopub.status.idle": "2023-03-07T05:57:21.893900Z",
     "shell.execute_reply": "2023-03-07T05:57:21.893142Z",
     "shell.execute_reply.started": "2023-03-07T05:57:04.560322Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[ TIMELINE ]\n",
      "\n",
      "╔═════╤═══════════════════╤════════╤═══════════╤═════════════╤═════════════╤═════════════╗\n",
      "║ No. │ Instant           │ Action │ State     │ Requested   │ Inflight    │ Completed   ║\n",
      "║     │                   │        │           │ Time        │ Time        │ Time        ║\n",
      "╠═════╪═══════════════════╪════════╪═══════════╪═════════════╪═════════════╪═════════════╣\n",
      "║ 0   │ 20230307055608090 │ commit │ COMPLETED │ 03-07 05:56 │ 03-07 05:56 │ 03-07 05:56 ║\n",
      "╚═════╧═══════════════════╧════════╧═══════════╧═════════════╧═════════════╧═════════════╝\n",
      "\n",
      "[ COMMITS ]\n",
      "\n",
      "╔═══════════════════╤═════════════════════╤═══════════════════╤═════════════════════╤══════════════════════════╤═══════════════════════╤══════════════════════════════╤══════════════╗\n",
      "║ CommitTime        │ Total Bytes Written │ Total Files Added │ Total Files Updated │ Total Partitions Written │ Total Records Written │ Total Update Records Written │ Total Errors ║\n",
      "╠═══════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣\n",
      "║ 20230307055608090 │ 364.3 MB            │ 4                 │ 0                   │ 2                        │ 2194714               │ 0                            │ 0            ║\n",
      "╚═══════════════════╧═════════════════════╧═══════════════════╧═════════════════════╧══════════════════════════╧═══════════════════════╧══════════════════════════════╧══════════════╝\n",
      "\n",
      "[ STORAGE ]\n",
      "\n",
      "/home/emr-notebook/reviews_cow_bucket_1\n",
      "├── [182M 05:57:21]  parity=0\n",
      "│   ├── [ 91M 05:56:52]  00000000-d7a3-4e5d-9bf5-b077f5c38bfe-0_0-23-421_20230307055608090.parquet\n",
      "│   └── [ 91M 05:56:52]  00000001-e09c-4037-b0c7-831fdf0ef2d9-0_1-23-422_20230307055608090.parquet\n",
      "└── [182M 05:57:21]  parity=1\n",
      "    ├── [ 91M 05:56:52]  00000000-f59b-4e14-b815-d83167fda637-0_2-23-423_20230307055608090.parquet\n",
      "    └── [ 91M 05:56:52]  00000001-d82f-4a2b-aa36-baad95d27a3d-0_3-23-424_20230307055608090.parquet\n",
      "\n",
      " 364M used in 2 directories, 4 files\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "~/hudi-stat.sh $TABLE_PATH timeline commits storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f18eaf86-0739-4366-b5e6-37c877041ab0",
   "metadata": {},
   "source": [
    "The 1st batch insert has exceeded `hoodie.parquet.small.file.limit` (100MB), let's see what will hadppend next."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da41e23-5df1-44a6-a20c-05c66b63a78f",
   "metadata": {},
   "source": [
    "### 2.3. Batch 2 - Insert ( 129 MB / FileGroup )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "83330cd5-8057-4003-b196-6a781e8ed411",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:57:21.895411Z",
     "iopub.status.busy": "2023-03-07T05:57:21.895150Z",
     "iopub.status.idle": "2023-03-07T05:58:27.408822Z",
     "shell.execute_reply": "2023-03-07T05:58:27.408143Z",
     "shell.execute_reply.started": "2023-03-07T05:57:21.895378Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1c8f17a9d6ea49f4a6725fc918957667",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "78715e178d6641ec962713a94763879e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "insert into \n",
    "    ${TABLE_NAME}\n",
    "select \n",
    "    review_id, \n",
    "    star_rating, \n",
    "    review_body, \n",
    "    review_date, \n",
    "    year,\n",
    "    unix_timestamp(current_timestamp()) as timestamp,\n",
    "    mod(crc32(review_id), 2) as parity\n",
    "from\n",
    "    reviews\n",
    "where\n",
    "    year = 2008"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1053316f-8411-4097-882b-56ce928b1f72",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-07T05:58:27.409982Z",
     "iopub.status.busy": "2023-03-07T05:58:27.409754Z",
     "iopub.status.idle": "2023-03-07T05:58:46.258821Z",
     "shell.execute_reply": "2023-03-07T05:58:46.257653Z",
     "shell.execute_reply.started": "2023-03-07T05:58:27.409960Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[ TIMELINE ]\n",
      "\n",
      "╔═════╤═══════════════════╤════════╤═══════════╤═════════════╤═════════════╤═════════════╗\n",
      "║ No. │ Instant           │ Action │ State     │ Requested   │ Inflight    │ Completed   ║\n",
      "║     │                   │        │           │ Time        │ Time        │ Time        ║\n",
      "╠═════╪═══════════════════╪════════╪═══════════╪═════════════╪═════════════╪═════════════╣\n",
      "║ 0   │ 20230307055608090 │ commit │ COMPLETED │ 03-07 05:56 │ 03-07 05:56 │ 03-07 05:56 ║\n",
      "╟─────┼───────────────────┼────────┼───────────┼─────────────┼─────────────┼─────────────╢\n",
      "║ 1   │ 20230307055723252 │ commit │ COMPLETED │ 03-07 05:57 │ 03-07 05:57 │ 03-07 05:58 ║\n",
      "╚═════╧═══════════════════╧════════╧═══════════╧═════════════╧═════════════╧═════════════╝\n",
      "\n",
      "[ COMMITS ]\n",
      "\n",
      "╔═══════════════════╤═════════════════════╤═══════════════════╤═════════════════════╤══════════════════════════╤═══════════════════════╤══════════════════════════════╤══════════════╗\n",
      "║ CommitTime        │ Total Bytes Written │ Total Files Added │ Total Files Updated │ Total Partitions Written │ Total Records Written │ Total Update Records Written │ Total Errors ║\n",
      "╠═══════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣\n",
      "║ 20230307055723252 │ 773.6 MB            │ 0                 │ 4                   │ 2                        │ 4683044               │ 0                            │ 0            ║\n",
      "╟───────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢\n",
      "║ 20230307055608090 │ 364.3 MB            │ 4                 │ 0                   │ 2                        │ 2194714               │ 0                            │ 0            ║\n",
      "╚═══════════════════╧═════════════════════╧═══════════════════╧═════════════════════╧══════════════════════════╧═══════════════════════╧══════════════════════════════╧══════════════╝\n",
      "\n",
      "[ STORAGE ]\n",
      "\n",
      "/home/emr-notebook/reviews_cow_bucket_1\n",
      "├── [569M 05:58:46]  parity=0\n",
      "│   ├── [ 91M 05:56:52]  00000000-d7a3-4e5d-9bf5-b077f5c38bfe-0_0-23-421_20230307055608090.parquet\n",
      "│   ├── [194M 05:58:16]  00000000-d7a3-4e5d-9bf5-b077f5c38bfe-0_0-56-872_20230307055723252.parquet\n",
      "│   ├── [ 91M 05:56:52]  00000001-e09c-4037-b0c7-831fdf0ef2d9-0_1-23-422_20230307055608090.parquet\n",
      "│   └── [193M 05:58:16]  00000001-e09c-4037-b0c7-831fdf0ef2d9-0_1-56-873_20230307055723252.parquet\n",
      "└── [569M 05:58:46]  parity=1\n",
      "    ├── [ 91M 05:56:52]  00000000-f59b-4e14-b815-d83167fda637-0_2-23-423_20230307055608090.parquet\n",
      "    ├── [193M 05:58:16]  00000000-f59b-4e14-b815-d83167fda637-0_2-56-874_20230307055723252.parquet\n",
      "    ├── [ 91M 05:56:52]  00000001-d82f-4a2b-aa36-baad95d27a3d-0_3-23-424_20230307055608090.parquet\n",
      "    └── [194M 05:58:16]  00000001-d82f-4a2b-aa36-baad95d27a3d-0_3-56-875_20230307055723252.parquet\n",
      "\n",
      " 1.1G used in 2 directories, 8 files\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "~/hudi-stat.sh $TABLE_PATH timeline commits storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6243b919-05a0-4385-a056-129bc6395b37",
   "metadata": {
    "tags": []
   },
   "source": [
    "Evidence: latest version of parquet file is over `hoodie.parquet.small.file.limit` ( 120 MB ), no new file group is splited.\n",
    "Conclusion: the default file sizing rule [100MB, 120MB] is inapplicable "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Spark",
   "language": "scala",
   "name": "sparkkernel"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".sc",
   "mimetype": "text/x-scala",
   "name": "scala",
   "pygments_lexer": "scala"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}