|
| 1 | +<!-- |
| 2 | +Licensed to the Apache Software Foundation (ASF) under one |
| 3 | +or more contributor license agreements. See the NOTICE file |
| 4 | +distributed with this work for additional information |
| 5 | +regarding copyright ownership. The ASF licenses this file |
| 6 | +to you under the Apache License, Version 2.0 (the |
| 7 | +"License"); you may not use this file except in compliance |
| 8 | +with the License. You may obtain a copy of the License at |
| 9 | +
|
| 10 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 11 | +
|
| 12 | +Unless required by applicable law or agreed to in writing, |
| 13 | +software distributed under the License is distributed on an |
| 14 | +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 15 | +KIND, either express or implied. See the License for the |
| 16 | +specific language governing permissions and limitations |
| 17 | +under the License. |
| 18 | +--> |
| 19 | + |
| 20 | +# DataFusion Comet 0.12.0 Changelog |
| 21 | + |
| 22 | +This release consists of 105 commits from 13 contributors. See credits at the end of this changelog for more information. |
| 23 | + |
| 24 | +**Fixed bugs:** |
| 25 | + |
| 26 | +- fix: Fix `None.get` in `stringDecode` when `bin` child cannot be converted [#2606](https://github.com/apache/datafusion-comet/pull/2606) (cfmcgrady) |
| 27 | +- fix: Update FuzzDataGenerator to produce dictionary-encoded string arrays & fix bugs that this exposes [#2635](https://github.com/apache/datafusion-comet/pull/2635) (andygrove) |
| 28 | +- fix: Fallback to Spark for lpad/rpad for unsupported arguments & fix negative length handling [#2630](https://github.com/apache/datafusion-comet/pull/2630) (andygrove) |
| 29 | +- fix: Mark SortOrder with floating-point as incompatible [#2650](https://github.com/apache/datafusion-comet/pull/2650) (andygrove) |
| 30 | +- fix: Fall back to Spark for `trunc` / `date_trunc` functions when format string is unsupported, or is not a literal value [#2634](https://github.com/apache/datafusion-comet/pull/2634) (andygrove) |
| 31 | +- fix: [native_datafusion] only pass single partition of PartitionedFiles into DataSourceExec [#2675](https://github.com/apache/datafusion-comet/pull/2675) (mbutrovich) |
| 32 | +- fix: Fix subcommands options in fuzz-testing [#2684](https://github.com/apache/datafusion-comet/pull/2684) (manuzhang) |
| 33 | +- fix: Do not replace SMJ with HJ for `LeftSemi` [#2687](https://github.com/apache/datafusion-comet/pull/2687) (comphead) |
| 34 | +- fix: Apply spotless on Iceberg 1.8.1 diff [iceberg] [#2700](https://github.com/apache/datafusion-comet/pull/2700) (hsiang-c) |
| 35 | +- fix: Fix generate-user-guide-reference-docs failure when mvn command is not executed at root [#2691](https://github.com/apache/datafusion-comet/pull/2691) (manuzhang) |
| 36 | +- fix: Fix missing SortOrder fallback reason in range partitioning [#2716](https://github.com/apache/datafusion-comet/pull/2716) (andygrove) |
| 37 | +- fix: CometLiteral class cast exception with arrays [#2718](https://github.com/apache/datafusion-comet/pull/2718) (andygrove) |
| 38 | +- fix: NormalizeNaNAndZero::children() returns child's child [#2732](https://github.com/apache/datafusion-comet/pull/2732) (mbutrovich) |
| 39 | +- fix: checkSparkMaybeThrows should compare Spark and Comet results in success case [#2728](https://github.com/apache/datafusion-comet/pull/2728) (andygrove) |
| 40 | +- fix: Mark `WindowsExec` as incompatible [#2748](https://github.com/apache/datafusion-comet/pull/2748) (andygrove) |
| 41 | +- fix: Add strict floating point mode and fallback to Spark for min/max/sort on floating point inputs when enabled [#2747](https://github.com/apache/datafusion-comet/pull/2747) (andygrove) |
| 42 | +- fix: Implement producedAttributes for CometWindowExec [#2789](https://github.com/apache/datafusion-comet/pull/2789) (rahulbabarwal89) |
| 43 | +- fix: Pass all Comet configs to native plan [#2801](https://github.com/apache/datafusion-comet/pull/2801) (andygrove) |
| 44 | + |
| 45 | +**Implemented enhancements:** |
| 46 | + |
| 47 | +- feat: Add option to write benchmark results to file [#2640](https://github.com/apache/datafusion-comet/pull/2640) (andygrove) |
| 48 | +- feat: Implement metrics for iceberg compat [#2615](https://github.com/apache/datafusion-comet/pull/2615) (EmilyMatt) |
| 49 | +- feat: Define function signatures in CometFuzz [#2614](https://github.com/apache/datafusion-comet/pull/2614) (andygrove) |
| 50 | +- feat: cherry-pick UUID conversion logic from #2528 [#2648](https://github.com/apache/datafusion-comet/pull/2648) (mbutrovich) |
| 51 | +- feat: support `concat` for strings [#2604](https://github.com/apache/datafusion-comet/pull/2604) (comphead) |
| 52 | +- feat: Add support for `abs` [#2689](https://github.com/apache/datafusion-comet/pull/2689) (andygrove) |
| 53 | +- feat: Support variadic function in CometFuzz [#2682](https://github.com/apache/datafusion-comet/pull/2682) (manuzhang) |
| 54 | +- feat: CometExecRule refactor: Unify CometNativeExec creation with Serde in CometOperatorSerde trait [#2768](https://github.com/apache/datafusion-comet/pull/2768) (andygrove) |
| 55 | +- feat: support cot [#2755](https://github.com/apache/datafusion-comet/pull/2755) (psvri) |
| 56 | +- feat: Add bash script to build and run fuzz testing [#2686](https://github.com/apache/datafusion-comet/pull/2686) (manuzhang) |
| 57 | +- feat: Add getSupportLevel to CometAggregateExpressionSerde trait [#2777](https://github.com/apache/datafusion-comet/pull/2777) (andygrove) |
| 58 | +- feat: Add CI check to ensure generated docs are in sync with code [#2779](https://github.com/apache/datafusion-comet/pull/2779) (andygrove) |
| 59 | +- feat: Add prettier enforcement [#2783](https://github.com/apache/datafusion-comet/pull/2783) (andygrove) |
| 60 | +- feat: hyperbolic trig functions [#2784](https://github.com/apache/datafusion-comet/pull/2784) (psvri) |
| 61 | +- feat: [iceberg] Native scan by serializing FileScanTasks to iceberg-rust [#2528](https://github.com/apache/datafusion-comet/pull/2528) (mbutrovich) |
| 62 | + |
| 63 | +**Documentation updates:** |
| 64 | + |
| 65 | +- docs: Add changelog for 0.11.0 release [#2585](https://github.com/apache/datafusion-comet/pull/2585) (mbutrovich) |
| 66 | +- docs: Improve documentation layout [#2587](https://github.com/apache/datafusion-comet/pull/2587) (andygrove) |
| 67 | +- docs: Publish 0.11.0 user guide [#2589](https://github.com/apache/datafusion-comet/pull/2589) (andygrove) |
| 68 | +- docs: Put Comet logo in top nav bar, respect light/dark mode [#2591](https://github.com/apache/datafusion-comet/pull/2591) (andygrove) |
| 69 | +- docs: Improve main landing page [#2593](https://github.com/apache/datafusion-comet/pull/2593) (andygrove) |
| 70 | +- docs: Improve site navigation [#2597](https://github.com/apache/datafusion-comet/pull/2597) (andygrove) |
| 71 | +- docs: Update benchmark results [#2596](https://github.com/apache/datafusion-comet/pull/2596) (andygrove) |
| 72 | +- docs: Upgrade pydata-sphinx-theme to 0.16.1 [#2602](https://github.com/apache/datafusion-comet/pull/2602) (andygrove) |
| 73 | +- docs: Fix redirect [#2603](https://github.com/apache/datafusion-comet/pull/2603) (andygrove) |
| 74 | +- docs: Fix broken image link [#2613](https://github.com/apache/datafusion-comet/pull/2613) (andygrove) |
| 75 | +- docs: Add FFI docs to contributor guide [#2668](https://github.com/apache/datafusion-comet/pull/2668) (andygrove) |
| 76 | +- docs: Various documentation updates [#2674](https://github.com/apache/datafusion-comet/pull/2674) (andygrove) |
| 77 | +- docs: Add supported SortOrder expressions and fix a typo [#2694](https://github.com/apache/datafusion-comet/pull/2694) (andygrove) |
| 78 | +- docs: Minor docs update for running Spark SQL tests [#2712](https://github.com/apache/datafusion-comet/pull/2712) (andygrove) |
| 79 | +- docs: Update contributor guide for adding a new expression [#2704](https://github.com/apache/datafusion-comet/pull/2704) (andygrove) |
| 80 | +- docs: Documentation updates for `LocalTableScan` and `WindowExec` [#2742](https://github.com/apache/datafusion-comet/pull/2742) (andygrove) |
| 81 | +- docs: Typo fix [#2752](https://github.com/apache/datafusion-comet/pull/2752) (wForget) |
| 82 | +- docs: Categorize some configs as `testing` and add notes about known time zone issues [#2740](https://github.com/apache/datafusion-comet/pull/2740) (andygrove) |
| 83 | +- docs: Run prettier on all markdown files [#2782](https://github.com/apache/datafusion-comet/pull/2782) (andygrove) |
| 84 | +- docs: Ignore prettier formatting for generated tables [#2790](https://github.com/apache/datafusion-comet/pull/2790) (andygrove) |
| 85 | +- docs: Add new section to contributor guide, explaining how to add a new operator [#2758](https://github.com/apache/datafusion-comet/pull/2758) (andygrove) |
| 86 | + |
| 87 | +**Other:** |
| 88 | + |
| 89 | +- chore: Start 0.12.0 development [#2584](https://github.com/apache/datafusion-comet/pull/2584) (mbutrovich) |
| 90 | +- chore: Bump Spark from 3.5.6 to 3.5.7 [#2574](https://github.com/apache/datafusion-comet/pull/2574) (cfmcgrady) |
| 91 | +- chore(deps): bump parquet from 56.0.0 to 56.2.0 in /native [#2608](https://github.com/apache/datafusion-comet/pull/2608) (dependabot[bot]) |
| 92 | +- chore(deps): bump tikv-jemallocator from 0.6.0 to 0.6.1 in /native [#2609](https://github.com/apache/datafusion-comet/pull/2609) (dependabot[bot]) |
| 93 | +- chore(deps): bump tikv-jemalloc-ctl from 0.6.0 to 0.6.1 in /native [#2610](https://github.com/apache/datafusion-comet/pull/2610) (dependabot[bot]) |
| 94 | +- tests: FuzzDataGenerator instead of Parquet-specific generator [#2616](https://github.com/apache/datafusion-comet/pull/2616) (mbutrovich) |
| 95 | +- chore: Simplify on-heap memory configuration [#2599](https://github.com/apache/datafusion-comet/pull/2599) (andygrove) |
| 96 | +- Feat: Add sha1 function impl [#2471](https://github.com/apache/datafusion-comet/pull/2471) (kazantsev-maksim) |
| 97 | +- chore: Refactor Parquet/DataFrame fuzz data generators [#2629](https://github.com/apache/datafusion-comet/pull/2629) (andygrove) |
| 98 | +- chore: Remove needless from_raw calls [#2638](https://github.com/apache/datafusion-comet/pull/2638) (EmilyMatt) |
| 99 | +- chore: support DataFusion 50.3.0 [#2605](https://github.com/apache/datafusion-comet/pull/2605) (comphead) |
| 100 | +- chore(deps): bump actions/upload-artifact from 4 to 5 [#2654](https://github.com/apache/datafusion-comet/pull/2654) (dependabot[bot]) |
| 101 | +- chore(deps): bump cc from 1.2.42 to 1.2.43 in /native [#2653](https://github.com/apache/datafusion-comet/pull/2653) (dependabot[bot]) |
| 102 | +- chore(deps): bump actions/download-artifact from 5 to 6 [#2652](https://github.com/apache/datafusion-comet/pull/2652) (dependabot[bot]) |
| 103 | +- chore: extract comparison into separate tool [#2632](https://github.com/apache/datafusion-comet/pull/2632) (comphead) |
| 104 | +- chore: Various improvements to `checkSparkAnswer*` methods in `CometTestBase` [#2656](https://github.com/apache/datafusion-comet/pull/2656) (andygrove) |
| 105 | +- chore: Remove code for unpacking dictionaries prior to FilterExec [#2659](https://github.com/apache/datafusion-comet/pull/2659) (andygrove) |
| 106 | +- chore: display schema for datasets being compared [#2665](https://github.com/apache/datafusion-comet/pull/2665) (comphead) |
| 107 | +- chore: Remove `CopyExec` [#2663](https://github.com/apache/datafusion-comet/pull/2663) (andygrove) |
| 108 | +- chore: Add extended explain plans to stability suite [#2669](https://github.com/apache/datafusion-comet/pull/2669) (andygrove) |
| 109 | +- chore(deps): bump aws-config from 1.8.8 to 1.8.10 in /native [#2677](https://github.com/apache/datafusion-comet/pull/2677) (dependabot[bot]) |
| 110 | +- chore(deps): bump cc from 1.2.43 to 1.2.44 in /native [#2678](https://github.com/apache/datafusion-comet/pull/2678) (dependabot[bot]) |
| 111 | +- chore: `tpcbench` output `explain` just once and formatted [#2679](https://github.com/apache/datafusion-comet/pull/2679) (comphead) |
| 112 | +- chore: Add tolerance for `ComparisonTool` [#2699](https://github.com/apache/datafusion-comet/pull/2699) (comphead) |
| 113 | +- chore: Expand test coverage for `CometWindowsExec` [#2711](https://github.com/apache/datafusion-comet/pull/2711) (comphead) |
| 114 | +- chore: generate Float/Double NaN [#2695](https://github.com/apache/datafusion-comet/pull/2695) (hsiang-c) |
| 115 | +- minor: Combine two CI workflows for Spark SQL tests [#2727](https://github.com/apache/datafusion-comet/pull/2727) (andygrove) |
| 116 | +- chore: Improve framework for specifying that configs can be set with env vars [#2722](https://github.com/apache/datafusion-comet/pull/2722) (andygrove) |
| 117 | +- chore: Rename `COMET_EXPLAIN_VERBOSE_ENABLED` to `COMET_EXTENDED_EXPLAIN_FORMAT` and change default [#2644](https://github.com/apache/datafusion-comet/pull/2644) (andygrove) |
| 118 | +- chore: Fallback to Spark for windows functions [#2726](https://github.com/apache/datafusion-comet/pull/2726) (comphead) |
| 119 | +- chore: Refactor operator serde - part 1 [#2738](https://github.com/apache/datafusion-comet/pull/2738) (andygrove) |
| 120 | +- Feat: Add CometLocalTableScanExec operator [#2735](https://github.com/apache/datafusion-comet/pull/2735) (kazantsev-maksim) |
| 121 | +- chore(deps): bump cc from 1.2.44 to 1.2.45 in /native [#2750](https://github.com/apache/datafusion-comet/pull/2750) (dependabot[bot]) |
| 122 | +- chore(deps): bump aws-credential-types from 1.2.8 to 1.2.9 in /native [#2751](https://github.com/apache/datafusion-comet/pull/2751) (dependabot[bot]) |
| 123 | +- chore: Operator serde refactor part 2 [#2741](https://github.com/apache/datafusion-comet/pull/2741) (andygrove) |
| 124 | +- chore: Fallback to Spark for `array_reverse` for `array<binary>` [#2759](https://github.com/apache/datafusion-comet/pull/2759) (comphead) |
| 125 | +- chore: [iceberg] test iceberg 1.10.0 [#2709](https://github.com/apache/datafusion-comet/pull/2709) (manuzhang) |
| 126 | +- chore: Add `docs/comet-*` to rat exclude list [#2762](https://github.com/apache/datafusion-comet/pull/2762) (manuzhang) |
| 127 | +- Chore: Refactor static invoke exprs [#2671](https://github.com/apache/datafusion-comet/pull/2671) (kazantsev-maksim) |
| 128 | +- minor: Small refactor for consistent serde for hash aggregate [#2764](https://github.com/apache/datafusion-comet/pull/2764) (andygrove) |
| 129 | +- minor: Move `operator2Proto` to `CometExecRule` [#2767](https://github.com/apache/datafusion-comet/pull/2767) (andygrove) |
| 130 | +- chore: various refactoring changes for iceberg [iceberg] [#2680](https://github.com/apache/datafusion-comet/pull/2680) (parthchandra) |
| 131 | +- chore: Refactor CometExecRule handling of sink operators [#2771](https://github.com/apache/datafusion-comet/pull/2771) (andygrove) |
| 132 | +- minor: Refactor to move window-specific code from `QueryPlanSerde` to `CometWindowExec` [#2780](https://github.com/apache/datafusion-comet/pull/2780) (andygrove) |
| 133 | +- chore: Remove many references to `COMET_EXPR_ALLOW_INCOMPATIBLE` [#2775](https://github.com/apache/datafusion-comet/pull/2775) (andygrove) |
| 134 | +- chore: Remove COMET_EXPR_ALLOW_INCOMPATIBLE config [#2786](https://github.com/apache/datafusion-comet/pull/2786) (andygrove) |
| 135 | +- chore: check `missingInput` for Comet plan nodes [#2795](https://github.com/apache/datafusion-comet/pull/2795) (comphead) |
| 136 | +- chore: Finish refactoring expression serde out of `QueryPlanSerde` [#2791](https://github.com/apache/datafusion-comet/pull/2791) (andygrove) |
| 137 | +- chore: Update docs to fix CI after #2784 [#2799](https://github.com/apache/datafusion-comet/pull/2799) (mbutrovich) |
| 138 | +- chore: Update q79 golden plan for Spark 4.0 after #2795 [#2800](https://github.com/apache/datafusion-comet/pull/2800) (mbutrovich) |
| 139 | +- Fix: Fix null handling in CometVector implementations [#2643](https://github.com/apache/datafusion-comet/pull/2643) (cfmcgrady) |
| 140 | + |
| 141 | +## Credits |
| 142 | + |
| 143 | +Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. |
| 144 | + |
| 145 | +``` |
| 146 | + 54 Andy Grove |
| 147 | + 11 Oleks V |
| 148 | + 10 dependabot[bot] |
| 149 | + 9 Matt Butrovich |
| 150 | + 6 Manu Zhang |
| 151 | + 3 Fu Chen |
| 152 | + 3 Kazantsev Maksim |
| 153 | + 2 Emily Matheys |
| 154 | + 2 Vrishabh |
| 155 | + 2 hsiang-c |
| 156 | + 1 Parth Chandra |
| 157 | + 1 Zhen Wang |
| 158 | + 1 rahulbabarwal89 |
| 159 | +``` |
| 160 | + |
| 161 | +Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. |
0 commit comments