Community roadmap 2023

[gianm]

It would be cool to have a publicly published roadmap of things the community is working on, so I thought we should do one for 2023. The year isn't even half over, so it seems like a good time 🙂.

I'm starting this list off with some things that are on my radar. I hope everyone reading this sees some exciting features on the list! Please feel free to post here if you've got any feedback, questions, or comments about items that are or aren't here.

Btw, note that items being on this list doesn't mean they will definitely ship in 2023. It is rather a statement that there is interest in working on them, and some likelihood they will get done. So, it's a good way to get a sense for what kinds of things are receiving active development interest and activity.

Query language

• Window functions (Starting on Window Functions #13458, Support Framing for Window Aggregations #13514, Window planning: use collation traits, improve subquery logic. #13902, Add window-focused tests from Drill #13773)
• Distributed shuffling JOIN (Sort-merge join and hash shuffles for MSQ. #13506)
• Array columns (nested columns + arrays = array columns! #13803)
• Array UNNEST (Unnest functionality for Druid #13268)
• Standardize on SQL-compatible null handling (SQL-compatible null handling and strict booleans by default #14154)
• ORDER BY non-__time in native Scan query
• UNION ALL anywhere

Query engine

• Detect unavailable segments (Detect segment unavailability for queries in broker  #13967)
• Larger, self-adjusting subquery limit (Limit the subquery results by memory usage #13952)
• High-cardinality GROUP BY performance enhancements
• Interactive API for multi-stage query (Adding Interactive API's for MSQ engine #14416)
• Export multi-stage query task results to object storage (Adding Ability for MSQ to write select results to durable storage. #14527)
• Query directly from object storage (Adding Interactive API's for MSQ engine #14416)
• Update to Calcite 1.34 or later (Upgrade calcite in druid #13532)
• EXPLAIN ANALYZE

Data management

• Batch DELETE DML (DELETE syntax for MSQ #14145)
• Batch UPDATE, UPSERT DML
• DDL (CREATE / ALTER / DROP TABLE)
• Schema auto-discovery
• Compaction concurrent with streaming ingestion
• Apache Spark readers/writers (Spark Direct Readers and Writers for Druid. #10920)
• Apache Iceberg integration (Proposal: Druid extension to read and ingest Iceberg data files #13923)
• Continuous auto-compaction (Continuous automatic compaction #13852)
• Metadata catalog (Druid Catalog Proposal #12546)
• Validation of load rules, partitioning, etc
• Broadcast tables (alternative to lookups)
• Reset to arbitrary Kafka offsets (Reset offsets supervisor API #14772)
• Ingesting data from multiple kafka topics (Add support to read from multiple kafka topics in same supervisor #14424)
• Improved kill tasks (Speed up kill tasks by deleting segments in batch #14131, Enhance the logic used by the Coordinator when determining if an unused segment can be killed (permanently deleted) #12526)

Operations

• Java 17 full support (Java 17 compatibility #12838)
• Java 21 experimental support
• Overlord scalability to larger numbers of tasks (Use separate executor to handle task updates in TaskQueue #14533)
• Coordinator scalability to larger numbers of segments (Enable smartSegmentLoading on the Coordinator #13197)
• K8s task runner (Support for middle manager less druid, tasks launch as k8s jobs #13156 and follow-ups)
• SysMonitor on ARM (Added new SysMonitor based on Oshi library #14359)

For Druid developers

• Standardized extension API
• Improve build pipeline and testing system

Ecosystem

Projects that are not part of Apache Druid proper, but happening in the surrounding ecosystem.

• Druid Kubernetes operator: ingestion controller (WIP Ingestion Controller datainfrahq/druid-operator#22), Historical autoscaling (Druid components autoscaling best practices datainfrahq/druid-operator#40).

[gianm]

@a2l007 shall I add #13923? I think a lot of people would be into this.

[suneet-s]

I think something under Operations for better kill tasks to help operators save costs would be nice to add. #14131 makes them faster and #12526 would make it safer for operators to use.

Under data management - I see there is already a note about concurrent compaction. Related to that I'd like to add #13852 Both seem to be in service of making compaction faster and more robust.

[maytasm]

+1 on #13923
and would also like to add #10920 to the roadmap

[gianm]

I think something under Operations for better kill tasks to help operators save costs would be nice to add. #14131 makes them faster and #12526 would make it safer for operators to use.

With regard to #12526 — @capistrant would you be able to pick this back up this year? Or if not, @suneet-s do you think you or someone else would be able to take it over the finish line? If so, I'd love to add it.

Under data management - I see there is already a note about concurrent compaction. Related to that I'd like to add #13852 Both seem to be in service of making compaction faster and more robust.

@suneet-s added as "Continuous auto-compaction"! 🚀

+1 on #13923

Same, hopefully @a2l007 can comment on it.

and would also like to add #10920 to the roadmap

@maytasm I would like to add it too 🙂 although it seems stuck. Do you know if @JulianJaffePinterest or you would be able to pick it back up? I know it's gone through the wringer during review, so if it gets picked back up I will set some time aside to personally stay with you til it gets in. That should help.

[AdheipSingh]

Kubernetes - Control Plane For Apache Druid

• We believe that Kubernetes can serve as a control plane for any application, including those with data and stateful sets. While Helm charts are useful for configuration management, they don't maintain the state of the application. That's why we've plan to build a control plane based on kubernetes operator pattern that acts as a bridge between your application's requirements and Kubernetes infrastructure.

Here are few pointers to be worked upon :

• Building Ingestion Spec controller WIP
• Scaling Historicals based on usage
• Rest a lot of work is needed to build up documentation, which is being worked upon :)

Looking forward for collaboration and feedback

[a2l007]

@gianm

@a2l007 shall I add #13923? I think a lot of people would be into this.

Yes please, I'll try to find sometime soon to put up a PR.

[maytasm]

@gianm

@maytasm I would like to add it too 🙂 although it seems stuck. Do you know if @JulianJaffePinterest or you would be able to pick it back up? I know it's gone through the wringer during review, so if it gets picked back up I will set some time aside to personally stay with you til it gets in. That should help.

I will pick it up and get it over the line in q3 of this year

[suneet-s]

With regard to #12526 — @capistrant would you be able to pick this back up this year? Or if not, @suneet-s do you think you or someone else would be able to take it over the finish line? If so, I'd love to add it.

I plan on picking it up this year if others don't have cycles.

[gianm]

Kubernetes - Control Plane For Apache Druid

@AdheipSingh Very cool! I added some of this stuff to a new "Ecosystem" section.

shall I add #13923? I think a lot of people would be into this.
Yes please, I'll try to find sometime soon to put up a PR.

@a2l007 Excellent, I added this to the "Data management" section.

and would also like to add #10920 to the roadmap
I will pick it up and get it over the line in q3 of this year

@maytasm Thanks! I added this to the "Data management" section.

With regard to #12526 ...
I plan on picking it up this year if others don't have cycles.

@suneet-s Thanks! I added this to the "Data management" section.

[JulianJaffePinterest]

@maytasm I should have some time in the next month or so to put up my Spark 3.3 update to #10920. There are a few pieces I need to pull out and patch over so if you already have a version in progress I'll defer to you.

[abhishekagarwal87]

I will also recommend adding #14424 to this roadmap. :)