From 01f7ee3ee88b7005b552d143daf28b36f7603d06 Mon Sep 17 00:00:00 2001 From: Lopa10ko Date: Fri, 22 Mar 2024 20:08:41 +0300 Subject: [PATCH 1/2] feat: add new m4 benchmark results --- docs/source/benchmarks/forecasting.rst | 51 ++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/docs/source/benchmarks/forecasting.rst b/docs/source/benchmarks/forecasting.rst index d83a2880ae..969084b06f 100644 --- a/docs/source/benchmarks/forecasting.rst +++ b/docs/source/benchmarks/forecasting.rst @@ -55,3 +55,54 @@ Timeout for Fedot and other frameworks was set by 2 minutes on each series. For Additionally you can examine papers about Fedot performance on different time series forecasting tasks `[1] `__ , `[2] `__, `[3] `__, `[4] `__, `[5] `__, `[6] `__, `[7] `__. + + +More M4 benchmarking +~~~~~~~~~~~~~~~~~~~~ + +This benchmark is based on a unified benchmarking interface provided by the `pytsbe framework `__ (a tool for benchmarking automated time-series forecasting algorithms). +The `pytsbe` tool uses `subsample `__ from `M4 competition `__ (sample contains 998 series with daily, weekly, monthly, quarterly, yearly intervals). +The forecasting horizons for each series type are: 6 for yearly series, 8 for quarterly series, 18 for monthly series, 13 for weekly series, and 14 for daily series. +The estimation metric used is Symmetric Mean Absolute Percentage Error (SMAPE). + + +-------------+----------+--------+--------+--------+-----------+--------+---------+ + | Library | Quantile | Intervals | + + + +--------+--------+--------+-----------+--------+---------+ + | | | Daily | Weekly | Montly | Quarterly | Yearly | Overall | + +=============+==========+========+========+========+===========+========+=========+ + | LagLlama | 10 | 1.457 | 3.258 | 5.303 | 5.713 | 11.665 | 2.64 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 50 | 4.513 | 11.167 | 18.534 | 20.027 | 33.141 | 13.036 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 90 | 13.123 | 28.268 | 62.091 | 48.793 | 73.565 | 48.056 | + +-------------+----------+--------+--------+--------+-----------+--------+---------+ + | NBEATS | 10 | 0.732 | 1.021 | 1.173 | 1.818 | 3.038 | 1.036 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 50 | 1.948 | 4.384 | 7.628 | 8.193 | 12.648 | 4.643 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 90 | 4.57 | 19.665 | 38.343 | 49.764 | 36.045 | 28.567 | + +-------------+----------+--------+--------+--------+-----------+--------+---------+ + | TimeGPT | 10 | 1.687 | 1.272 | 1.134 | 2.459 | 4.179 | 1.536 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 50 | 5.586 | 7.17 | 6.235 | 7.058 | 8.982 | 6.565 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 90 | 15.716 | 23.337 | 35.786 | 28.056 | 32.902 | 26.387 | + +-------------+----------+--------+--------+--------+-----------+--------+---------+ + | autogluon | 10 | 0.93 | 0.744 | 1.26 | 2.159 | 2.624 | 1.131 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 50 | 2.37 | 5.96 | 7.402 | 6.168 | 7.598 | 4.704 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 90 | 6.189 | 20.888 | 33.51 | 24.909 | 40.516 | 25.026 | + +-------------+----------+--------+--------+--------+-----------+--------+---------+ + | Fedot | 10 | 0.977 | 0.677 | 1.534 | 2.01 | 2.28 | 1.126 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 50 | 2.159 | 4.638 | 7.468 | 7.868 | 9.59 | 4.659 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 90 | 5.645 | 19.341 | 40.824 | 32.774 | 43.666 | 28.732 | + +-------------+----------+--------+--------+--------+-----------+--------+---------+ + | repeat_last | 10 | 0.795 | 1.059 | 1.477 | 2.534 | 4.242 | 1.146 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 50 | 2.008 | 5.365 | 7.796 | 7.379 | 9.066 | 5.158 | + + +----------+--------+--------+--------+-----------+--------+---------+ + | | 90 | 4.66 | 22.38 | 37.294 | 27.215 | 33.074 | 25.79 | + +-------------+----------+--------+--------+--------+-----------+--------+---------+ \ No newline at end of file From 80f172f322c395d875baa6bf04e2c6857003eeb6 Mon Sep 17 00:00:00 2001 From: Lopa10ko Date: Fri, 5 Apr 2024 21:32:47 +0300 Subject: [PATCH 2/2] chore: update FEDOT metrics on m4 --- docs/source/benchmarks/forecasting.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/benchmarks/forecasting.rst b/docs/source/benchmarks/forecasting.rst index 969084b06f..a14947a553 100644 --- a/docs/source/benchmarks/forecasting.rst +++ b/docs/source/benchmarks/forecasting.rst @@ -94,11 +94,11 @@ The estimation metric used is Symmetric Mean Absolute Percentage Error (SMAPE). + +----------+--------+--------+--------+-----------+--------+---------+ | | 90 | 6.189 | 20.888 | 33.51 | 24.909 | 40.516 | 25.026 | +-------------+----------+--------+--------+--------+-----------+--------+---------+ - | Fedot | 10 | 0.977 | 0.677 | 1.534 | 2.01 | 2.28 | 1.126 | + | Fedot | 10 | 0.97 | 0.733 | 1.342 | 1.771 | 2.892 | 1.064 | + +----------+--------+--------+--------+-----------+--------+---------+ - | | 50 | 2.159 | 4.638 | 7.468 | 7.868 | 9.59 | 4.659 | + | | 50 | 2.326 | 4.95 | 7.123 | 6.786 | 8.682 | 4.655 | + +----------+--------+--------+--------+-----------+--------+---------+ - | | 90 | 5.645 | 19.341 | 40.824 | 32.774 | 43.666 | 28.732 | + | | 90 | 5.398 | 19.131 | 43.519 | 36.36 | 41.147 | 30.29 | +-------------+----------+--------+--------+--------+-----------+--------+---------+ | repeat_last | 10 | 0.795 | 1.059 | 1.477 | 2.534 | 4.242 | 1.146 | + +----------+--------+--------+--------+-----------+--------+---------+