Roadmap Discussion Thread (2025)

[marcklingen]

Hi all!

We have exciting plans for 2025 and would love to hear your thoughts in this thread!

✨ What would you like to see added, fixed, or improved in 2025? ✨

Thank you for growing and shaping the project with us!
@marcklingen, @maxdeichmann, @clemra, @marliessophie, @jannikmaierhoefer, @Steffen911

---

Current roadmap (langfuse.com/roadmap)

🚧 In progress

• Various improvements, including:
  • Use "log levels" to simplify large agent traces, filter by them in UI
  • Typesafe sdk methods for all API routes
  • Commit messages on prompts
  • Duplicate prompts to edit name
  • User screen that's more helpful to summarize metrics
  • Evaluation runs on historical data
  • Eval execution states across Langfuse Tracing and Datasets
  • Use JSON path to extract structured data from traces for llm-as-judge evals
  • Save table filters as views
  • "Peek view" for tables, preview trace without switching the page
  • New observation table
  • ...

🔮 Planned

• Agent Observability: Improve native support for agentic applications
  • Visualizations of agent executions, e.g. Graph (#2509)
  • Filtering for tool calls used within an execution
  • Opinionated agent evaluations
• Tracing Enhancements
  • Environments: Native support for separating different environments within a project (#5005)
  • Realtime LLM: Bespoke traces for datastreams of realtime APIs, especially multi-modal (#4757)
  • Open Telemetry: Add support for OpenTelemetry-based instrumentations by adding an OpenTelemetry exporter/collector (#2509)
• Prompt Management
  • Diff view of prompt changes (#1105)
  • Custom RBAC for prompt change management (#4480)
  • LLM-assisted prompt engineering
  • Prompt composability to increase reuse of common prompts (#1660)
  • Folders (#4874)
  • Native support for tool calls (#2624)
  • Placeholders for chat messages (#2210)
• Evaluation
  • Non-LLM evaluators: classifiers, custom code, regex, etc.
  • Evals on repeated spans/observations within a trace
  • Comparisons of different evaluation metrics
  • Session-level scores (#2728)
• Query 100s of millions of traces:
  • Full-text search: Search across the inputs/outputs of traces, sessions, and datasets (#939)
  • Self-serve dashboards: Create and share custom dashboards based on metrics extracted from traces (#1011)
• Data Platform
  • Webhooks to subscribe to changes within your Langfuse project (#1033)
  • Batched exports to own storage buckets for downstream use and archival
  • Rule-based routing between Langfuse product features
• Other
  • UI/UX: improvements of all core product features
  • Onboarding: simplify getting started with Langfuse when not reading the docs
  • Data retention: Project-level data retention policies (#2268)

[ignas-rss]

Would be great to have a way to batch delete old traces - maybe in the api, the current approach is driving me nuts.
Also a way to get latest trace by parameters, such as username, would be great or at least sort it by date/time when iterating as getting the last trace for user is challenging.

[marcklingen]

Would be great to have a way to batch delete old traces - maybe in the api, the current approach is driving me nuts.

Shipping deletions is a high priority, tracking this here: #5056

Also a way to get latest trace by parameters, such as username, would be great or at least sort it by date/time when iterating as getting the last trace for user is challenging.

The ui and api support filtering by userId and sorting them. Can you explain what does not work for you?

[dwi-afelio]

Webhooks, full-text search, and classifiers (e.g., adding tags if the input or output contains specific content) would be great additions. However, what would truly help me is the ability to remove an evaluator (LLM as a judge).
Thank you for creating such a wonderful tool!

[marcklingen]

Webhooks

Thanks for your +1 on this, tracking this here: https://github.com/orgs/langfuse/discussions/1033

full-text search

This is on the mid-term roadmap and there are some ways to build this in a performant way now that we have moved to clickhouse, great that you want this, tracking this here: https://github.com/orgs/langfuse/discussions/939

and classifiers (e.g., adding tags if the input or output contains specific content) would be great additions

Follow-up question: Do you want LLM-as-a-judge to be able to create categorical scores in order to classify?

However, what would truly help me is the ability to remove an evaluator (LLM as a judge).

Great call, added this issue in order to fix this: #5057

[dwi-afelio]

And maybe fetch traces without a tag (as it's already possible for traces with a tag)

[marcklingen]

Thanks for the feedback. Are you suggesting "SELECT * FROM traces WHERE tags IS NULL"? Do you need this for the API or the UI?

[andreieuganox]

I know it can be off-topic for you, but human-in-the-loop would be a great addition. You already have trace input-output, so it means that it is halfway there.... Just saying it can be a good one;)

[marcklingen]

Interesting! Like being able to stop an execution and then continuing it via a webhook when a review has happened in an annotation/review queue? Would love to learn more, could be a simple extension once we have added webhooks

[leo-paz]

it would be great if traces we're pushed through websockets or polled, i find myself having to refresh 10+ times trying to see my trace as fast as possible

[marcklingen]

live traces have been something @maxdeichmann wanted to add for a long time already, we might finally get to this

[michalprubatalkie]

How about uploading datasets from csv / json files + dataset versioning + metrics aggregation by dataset's items metadata

[marcklingen]

@marliessophie just added dataset uploads in #4656

Versioning is tricky as datasets change frequently in some cases and has been discussed multiple times already. Do you have a good suggestion on the expected behavior here?

metrics aggregation by dataset's items metadata seems to be a good candidate for langfuse.com/ideas as I'd love to understand this in more detail

[yangcheng]

would be great to have self-host option with playground, even with less features

[nialldevlin1]

Would be great to filter on metadata rather than just on tags.

The current workaround is to supply the metadata values as a list of tags and then filter on them. For example, with the metadata {'product_id': 'foo', 'application_id': 'bar'} the tags would be ['foo', 'bar']. This isn't perfect though, as it isn't necessarily clear when filtering on tags in the UI which values correspond to which fields. It's also possible that I could have a clash between a product name and an application name (i.e. I could have one trace's metadata {'product_id': 'foo', 'application_id': 'bar'} and another as {'product_id': 'baz', 'application_id': 'foo'}) which would lead to incorrect usage tracking.

I'd also really appreciate the ability to save graphs on the dashboard. Currently you can only apply a filter to the whole dashboard, but having the ability to save graphs that have different filters applied would enable better comparison and improve UX.

[marcklingen]

Thanks for sharing! You can filter based on metadata on the trace table, does that help already?

I'd also really appreciate the ability to save graphs on the dashboard. Currently you can only apply a filter to the whole dashboard, but having the ability to save graphs that have different filters applied would enable better comparison and improve UX.

Love this, this will be part of https://github.com/orgs/langfuse/discussions/1011

[nialldevlin1]

Thanks for sharing! You can filter based on metadata on the trace table, does that help already?

Not quite - ideally we'd be able to filter on metadata on the dashboard as well, as this will give filtered aggregates like cost etc.

For example, imagine I have a metadata field feature_id and I want to check which features are the most expensive. To current workaround is to use tags, but this has some edge cases where this doesn't work.

Does that make sense? Let me know if there's an alternative way to gather this information that I'm not yet considering!

Love this, this will be part of https://github.com/orgs/langfuse/discussions/1011

Great, looking forward to seeing it!

[gyliu513]

FYI, we have two issues for agent observability

• AI Agent Semantic Convetion
• AI Agent framework Semantic Convetion

[marcklingen]

thanks @gyliu513! OTel is high up on the roadmap, we plan to add a v1 collector soon

[marcklingen]

Thanks so much for all your feedback already!

I was quite busy (in a good way) to read and respond to lots of direct email responses on this topic and will now get to the ideas here. Let's keep this thread open, especially to address high-level priorities

[michael-lo-221]

RE: Self Hosting Langfuse + Clickhouse

v3 will move Observability data to Clickhouse. I've been implementing Langfuse at my work so we try out different Observability products. Clickhouse isn't in our tech stack so we will be stuck with v2.9.

Will there be any option for self hosting users to stick with postgres?

[marcklingen]

To unblock many features and scalability constraints we needed to adopt an OLAP db for core tracing in langfuse. Clickhouse is a popular OSS choice. We have written more on this here: https://langfuse.com/changelog/2024-12-09-Langfuse-v3-stable-release#reasoning

We aim to make self-hosting simpler, especially on low-scale scenarios. Expect more documentation and guidance from us on how to do this.

There won't be a postgres-only option of Langfuse as this is unfeasible for us to maintain at the product velocity of the project as most features require some level of query-optimization for the specific db in order to work well at scale.

[ngocht]

Prompt Management on Organization Level

It would be nice to have a functionality to "promote" Prompts from a dev project to a prod project. At the moment the prompts need to be duplicated and there is no good rollout strategy for CI/CD.

[marcklingen]

I understand this problem well and we hope to resolve it via native support for environments: https://github.com/orgs/langfuse/discussions/5005

[anthony-liner]

Plans for 2025 look exciting! 😄

It would be great to have support for batch exporting large volumes of traces/observations to external data warehouses (e.g., BigQuery). ClickHouse OLAP performs poorly when retrieving raw input/output logs from thousands of rows.
#1219 #2131

[marcklingen]

Thanks for sharing! We consider adding batched exports to blob storage for downstream use in DWHs. Our experience using Clickhouse has been fairly good (that's why we selected it as the core db) and especially batches of rows that are within the same timeframe can be accessed way better than expected.

[msalafia]

It would be great to visualize metadata on a trace detail in the UI.

[marcklingen]

How would you like to visualize the metadata?

[msalafia]

I just would like to see them on the trace detail, i don't have any particular idea. As for as i know, the only way in the UI to visualize them is on traces, but its seems bugged since the field metadata is not scrollable.