Release-0.7.0 roadmap

[zhoujinsong]

This issue will track the roadmap of Amoro 0.7.0

Feature Development

• [Feature]: Support automatically generating and managing Tags and Branchs for Table #1354
• [Feature]: Support killing the running optimizing process #1862
• [Feature]: Support detail page for Optimizing Process #2114
• [Feature]: Support Spark-based optimizer #1812
• [Feature]: Add metric collection for ams #1528
• [Feature]: Support managing tables with multiple formats under one Catalog #1061
• [Feature]: Make local terminal cores configurable #2492
• [Feature]: Support reading encrypted iceberg data files #2401
• [Feature]: Add Tests for helm chart templates #2778
• [Feature]: Add basic authentication to REST API #2667

Improvement

• [Improvement]: Improve major optimizing: compact segment files to target size #2330
• [Improvement]: Put the database username and passowrd into kubernetes Secrets #2291
• [Improvement]: Avoid releasing external optimizer instances through ams #2315
• [Improvement]: Mixed-format Table stats for Trino engine need to be calculated individually #2246
• [Improvement]: Add cache for loading table when get table meta in unkeyed table #2296
• [Improvement]: Add planning optimizing status for tables #2290
• [Improvement]: Support computed columns for mixed-format tables in Flink DDL #1457
• [Improvement]: The Files page supports filtering by partition name and sorting by dictionary value. #2316
• [Improvement]: Enhance state ctrl in the terminal page #2363
• [Improvement]: Support log out of the ams dashboard #2359
• [Improvement]: Support parallelized planning in one optimizer group #1951
• [Improvement]: Skip cleaning up dangling delete files for Iceberg V1 table #2222
• [Improvement]: Configure iceberg.worker.num-threads in the configuration file #2386
• [Improvement]: Needs to sort the external Iceberg table list  #1716
• [HotFix] Add table filter for catalog properties #2310
• [Improvement]: Show the format version of Iceberg Table #2260
• [Improvement]: Make the maximum input file size for per optimize thread configurable #2385
• [Improvement]: The Files page supports filtering by partition name and sorting by dictionary value. #2316
• [Improvement]: Improve major optimizing: compact segment files to target size #2330
• [Improvement]: Document on using Kyuubi as a terminal backend in a Kerberos environment #2428
• [Improvement]: Use partition filter to speed up optimizing plan #2417
• [Mixed][Improvement]:Support for serialization and deserialization thread-safe in lookup JOIN #2497
• [Improvement][Flink]: Support customizing Arctic source parallelism #1035
• [Improvement][AMS]: Add more configurable properties to the ams server #2333
• [Improvement]: Dispose the current tasks after task deserialization error #2521
• [Improvement]: Paimon snapshot display on the page should be arranged in reverse order according to the snapshot time of the table. #2606
• [Improvement][Flink] Add support for setting/altering mixed-format table properties with Flink SQL  #2
• [Improvement]: Helm adds Kyuubi (as a terminal backend) and PostgreSQL related conf #2306
• [Improvement]: Validate hive-site.xml when uploading file for Hive-catalog  #2520
• [Improvement]: Optimize the speed of searching for tables in the tables navigation bar #2215
• [Improvement]: Use CachingCatalog to reduce the time cost of IcebergCatalogWrapper#loadTable #1794
• [Improvement]: Enable returnUrl parameter when redirecting to login page #2637
• [Improvement]: Display more details in the page of Optimizing->Optimizers #2698
• [Improvement]: Improve the version info about the project #2813
• [Improvement]: AMS should use G1 GC by default #2830
• [Improvement]: Support different simple authorization user name for different catalogs #2918
• Move admin username and password to secrets #2948

Bug Tracking

• [Bug]: After disabed self-optimizing, the table status is not updated to idle #2250
• [Bug]: org.apache.iceberg.exceptions.ValidationException: Missing required files to delete #2454
• [Bug]: Periodic snapshots expiring tasks will be scheduled redundantly #2453
• [Bug]: When using Flink api to write data into mix-iceberg table, it will update the table metadata to incorrect value  #2525
• [Bug]: partition values is null ​​will not trigger Optimizing #2542
• [Bug]: When expiring historic data,the latest snapshot generated by amoro should not be used #2555
• [Bug]: The process of deleting expired snapshots was interrupted by an exception #2579
• [Bug]: Loading Mixed-Iceberg format tables of internal catalog failed in Flink #2587
• [Bug]: Failed to remove orphan files after dropping a partition field #2617
• [Bug]:'Last Commit Time' for Paimon partition table in File pages has wrong zone  #2607
• [Bug]: If an exception occurs in externalCatalog.listTables during the exploreExternalCatalog process, exploreExternalCatalog will fail this time. #2638
• [Bug]: Failed to view snapshot details  #2614
• [Bug]: Hive table name may be too long for AMS table identifiers #2649
• [Bug]: Skip the deleted direcotories when cleaning orphan files #2619
• [Bug]: Optimizing process stuck after a few failures #2623
• [Feature]: Support Flink optmizer run in session cluster #2652
• [Bug]: Lack of database.type mandatory check in config, leading to NPE in existing MySQL config check #2733
• [Bug]: Iceberg temporary snapshot tables should be excluded from data cleanups #2488
• [Bug]: Filformat ORC do not support roll new file for mixed_format #2560

Document Improvement