Merge pull request #91 from databricks-industry-solutions/post-forecast-analysis

added post evaluation model selection notebook

Author: ryuta-yoshimatsu

README.md

@@ -12,6 +12,7 @@ Get started now!
 
 ## What's New
 
+- Feb 2025: Added a post evaluation notebook that shows how to run fine-grained model selection after running MMF. Try the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py).
 - Jan 2025: [TimesFM](https://github.com/google-research/timesfm) is available for univariate and covariate forecasting. Try the notebooks: [univariate](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/foundation_daily.py) and [covariate](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/external_regressors/foundation_external_regressors_daily.py).
 - Jan 2025: [Chronos Bolt](https://github.com/amazon-science/chronos-forecasting) models are available for univariate forecasting. Try the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/foundation_daily.py).
 - Jan 2025: [Moirai MoE](https://github.com/SalesforceAIResearch/uni2ts) models are available for univariate forecasting. Try the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/foundation_daily.py).
@@ -25,7 +26,7 @@ To run this solution on a public [M4](https://www.kaggle.com/datasets/yogesh94/m
 
 Local models are used to model individual time series. They could be advantageous over other types of model for their capabilities to tailor fit to individual series, offer greater interpretability, and require lower data requirements. We support models from [statsforecast](https://github.com/Nixtla/statsforecast), [r fable](https://cran.r-project.org/web/packages/fable/vignettes/fable.html) and [sktime](https://www.sktime.net/en/stable/). Covariates (i.e. exogenous regressors) are currently only supported for some models from statsforecast. 
 
-To get started, attach the [examples/daily/local_univariate_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/local_univariate_daily.py) notebook to a cluster running [DBR 15.4LTS for ML](https://docs.databricks.com/en/release-notes/runtime/15.4lts-ml.html) or later versions. The cluster can be either a single-node or multi-node CPU cluster. Make sure to set the following [Spark configurations](https://spark.apache.org/docs/latest/configuration.html) on the cluster before you start using MMF: ```spark.sql.execution.arrow.enabled true``` and ```spark.sql.adaptive.enabled false``` (more detailed explanation to follow). 
+To get started, attach the [examples/daily/local_univariate_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/local_univariate_daily.py) notebook to a cluster running [DBR 15.4LTS for ML](https://docs.databricks.com/en/release-notes/runtime/15.4lts-ml.html) or later versions. The cluster can be either a single-node or multi-node CPU cluster. Make sure to set the following [Spark configurations](https://spark.apache.org/docs/latest/configuration.html) on the cluster before you start using MMF: ```spark.sql.execution.arrow.enabled true``` and ```spark.sql.adaptive.enabled false``` (more detailed explanation can be found [here](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/local_univariate_daily.py)). 
 
 In this notebook, we will apply 20+ models to 100 time series. You can specify the models to use in a list:
 
@@ -110,7 +111,7 @@ run_forecast(
 
 To modify the model hyperparameters, change the values in [mmf_sa/models/models_conf.yaml](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/mmf_sa/models/models_conf.yaml) or overwrite these values in [mmf_sa/forecasting_conf.yaml](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/mmf_sa/forecasting_conf.yaml). 
 
-MMF is fully integrated with MLflow, so once the training kicks off, the experiments will be visible in the MLflow Tracking UI with the corresponding metrics and parameters (note that we do not log all local models in MLFlow, but we store the binaries in the tables ```evaluation_output``` and ```scoring_output```). The metric you see in the MLflow Tracking UI is a simple mean over backtesting trials over all time series.
+MMF is fully integrated with MLflow, so once the training kicks off, the experiments will be visible in the MLflow Tracking UI with the corresponding metrics and parameters (note that we do not log all local models in MLFlow, but we store the binaries in the tables ```evaluation_output``` and ```scoring_output```). The metric you see in the MLflow Tracking UI is a simple mean over backtesting trials over all time series. Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
 
 We encourage you to read through [examples/daily/local_univariate_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/local_univariate_daily.py) notebook to better understand how local models can be applied to your time series using MMF. An example notebook for forecasting with exogenous regressors can be found in [examples/external_regressors/local_univariate_external_regressors_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/external_regressors/local_univariate_external_regressors_daily.py).
 
@@ -189,7 +190,7 @@ To modify the model hyperparameters or reset the range of the hyperparameter sea
 
 MMF is fully integrated with MLflow and so once the training kicks off, the experiments will be visible in the MLflow Tracking UI with the corresponding metrics and parameters. Once the training is complete the models will be logged to MLFlow and registered to Unity Catalog. 
 
-We encourage you to read through [examples/daily/global_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/global_daily.py) notebook to better understand how global models can be applied to your time series using MMF. An example notebook for forecasting with exogenous regressors can be found in [examples/external_regressors/global_external_regressors_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/external_regressors/global_external_regressors_daily.py).
+We encourage you to read through [examples/daily/global_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/global_daily.py) notebook to better understand how global models can be applied to your time series using MMF. An example notebook for forecasting with exogenous regressors can be found in [examples/external_regressors/global_external_regressors_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/external_regressors/global_external_regressors_daily.py). Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
 
 ### Foundation Models
 
@@ -238,7 +239,7 @@ To modify the model hyperparameters, change the values in [mmf_sa/models/models_
 
 MMF is fully integrated with MLflow and so once the training kicks off, the experiments will be visible in the MLflow Tracking UI with the corresponding metrics and parameters. During the evaluation, the models are logged and registered to Unity Catalog.
 
-We encourage you to read through [examples/daily/foundation_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/foundation_daily.py) notebook to better understand how foundation models can be applied to your time series using MMF. An example notebook for forecasting with exogenous regressors can be found in [examples/external_regressors/foundation_external_regressors_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/external_regressors/foundation_external_regressors_daily.py).
+We encourage you to read through [examples/daily/foundation_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/daily/foundation_daily.py) notebook to better understand how foundation models can be applied to your time series using MMF. An example notebook for forecasting with exogenous regressors can be found in [examples/external_regressors/foundation_external_regressors_daily.py](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/external_regressors/foundation_external_regressors_daily.py). Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
 
 #### Using Time Series Foundation Models on Databricks
 

examples/daily/foundation_daily.py

@@ -200,6 +200,11 @@ def transform_group(df):
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
+
+# COMMAND ----------
+
 # MAGIC %md ### Delete Tables
 # MAGIC Let's clean up the tables.
 

examples/daily/global_daily.py

@@ -193,6 +193,11 @@ def transform_group(df):
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
+
+# COMMAND ----------
+
 # MAGIC %md ### Delete Tables
 # MAGIC Let's clean up the tables.
 

examples/daily/local_univariate_daily.py

@@ -231,6 +231,11 @@ def transform_group(df):
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
+
+# COMMAND ----------
+
 # MAGIC %md ### Delete Tables
 # MAGIC Let's clean up the tables.
 

examples/external_regressors/foundation_external_regressors_daily.py

@@ -152,6 +152,11 @@
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
+
+# COMMAND ----------
+
 # MAGIC %md ### Delete Tables
 # MAGIC Let's clean up the tables.
 

examples/external_regressors/global_external_regressors_daily.py

@@ -159,6 +159,11 @@
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
+
+# COMMAND ----------
+
 # MAGIC %md ### Delete Tables
 # MAGIC Let's clean up the tables.
 

examples/external_regressors/local_univariate_external_regressors_daily.py

@@ -190,6 +190,11 @@
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
+
+# COMMAND ----------
+
 # MAGIC %md ### Delete Tables
 # MAGIC Let's clean up the tables.
 

examples/hourly/foundation_hourly.py

@@ -180,6 +180,11 @@ def transform_group(df):
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
+
+# COMMAND ----------
+
 # MAGIC %md ### Delete Tables
 # MAGIC Let's clean up the tables.
 

examples/hourly/global_hourly.py

@@ -174,6 +174,11 @@ def transform_group(df):
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
+
+# COMMAND ----------
+
 # MAGIC %md ### Delete Tables
 # MAGIC Let's clean up the tables.
 

examples/hourly/local_univariate_hourly.py

@@ -201,6 +201,11 @@ def transform_group(df):
 
 # COMMAND ----------
 
+# MAGIC %md
+# MAGIC Refer to the [notebook](https://github.com/databricks-industry-solutions/many-model-forecasting/blob/main/examples/post-evaluation-analysis.py) for guidance on performing fine-grained model selection after running `run_forecast`.
+
+# COMMAND ----------
+
 # MAGIC %md ### Delete Tables
 # MAGIC Let's clean up the tables.