Concept for Callbacks for Events within Spring AI Framework

[henningSaulCM]

Hi,

I'm relatively knew to Spring AI (and AI in general). I'm currently looking into re-implementing a demo application (built using some other AI framework) using Spring AI. That other AI framework has some concept and mechanism for "Callbacks" to respond to events within the framework, e.g. calling function/tool, function/tool returned with result, etc. The demo application returns those "events" to the frontend to provide some feedback to the user for the current request, e.g.

8:[{"type":"events","data":{"title":"Using tool: 'local_index_query_engine' with inputs: 'query: How can I create an Article?'"}}]
8:[{"type":"events","data":{"title":"Retrieving context for query: 'How can I create an Article?'"}}]
8:[{"type":"events","data":{"title":"Retrieved 2 sources to use as context for the query"}}]
8:[{"type":"tools","data":{"toolCall":{"id":"call_5M54tvFLzvK0X3t6ArqV9L0c","name":"local_index_query_engine","input":{"query":"How can I create an Article?"}},"toolOutput":{"output":"To create an Article, you can follow these steps....","isError":false}}}]
...

Is there a similar concept in Spring AI (existing or planned) or can you think of alternative approaches to expose the internal steps of the framework to the end user/in a client request?

Your feedback is much appreciated!
--Henning

[tobins]

Hello!

Would Function Calling work for what you are interested in doing?

https://docs.spring.io/spring-ai/reference/api/functions.html

With Spring AI, you can give the AI model functions to invoke to complete a request.

[henningSaulCM]

Thanks for chiming in, @tobins.

My question is to some extent within the context of Function Calling (which we already do/use).

To be concrete, I was looking for an equivalent (or an alterative approach) to https://docs.llamaindex.ai/en/stable/module_guides/observability/callbacks/root/.
This allows one to easily get callbacks/notifications for steps/actions within the Framework.

The example (typescript) app I'm looking at uses the above mechanism to notify the frontend about events, e.g. when a tool is called during the execution.

  callbackManager.on("llm-tool-call", (event) => {
    const { name, input } = event.detail.payload.toolCall;
    const inputString = Object.entries(input)
      .map(([key, value]) => `${key}: ${value}`)
      .join(", ");
    appendEventData(
      stream,
      `Using tool: '${name}' with inputs: '${inputString}'`,
    );
  });

  callbackManager.on("llm-tool-result", (event) => {
    const { toolCall, toolResult } = event.detail.payload;
    appendToolData(stream, toolCall, toolResult);
  });

[dami-px]

@henningSaulCM do you have any updates on this? I am trying to find a solution to the same issue where I would like to give a response to the user while the agent is "Thinking" or calling tools

[henningSaulCM]

Hi @dami-px,

we ended up with the following custom solution:

1. Wrapping the actual tool invocation (in our case in an abstract BiFunction base) to capture input parameters and the response and add them to the toolContext.
2. A custom Advisor to add this information from the toolContext to the AssistantMessage's metadata (after the tool call(s))
3. Custom code in our (streaming) chat controller that sends Message Annotations for this metadata to the client/frontend
4. The frontend visualizes these tool calls/message annotations

So we don't have user feedback while a tool is being called, but after a tool has been called (and an AssistantMessage is returned).
In our case we're just showing all tool calls at the end of the chat request as some kind of "debug" information.

I hope this helps. Ideally, Spring AI can provide some support in this area in future releases.

[dami-px]

I really appreciate your response. Hopefully there's some support for this in the future.