Extract information on tool calls

[christophscheuch]

Absolutely love how easy ellmer makes the tool calling! I'm trying to figure out how to get more information on which tools are called if I pass multiple tools. Essentially, I want to keep track whether the model calls the right tools with the right parameters depending on the question. Maybe it makes sense to have a helper function for this in the package?

My quick-and-dirty approach based on the example from the docs:

library(ellmer)
library(dplyr)

chat <- chat_openai(model = "gpt-4o")

#' Gets the current time in the given time zone.
#'
#' @param tz The time zone to get the current time in.
#' @return The current time in the given time zone.
get_current_time <- function(tz = "UTC") {
  format(Sys.time(), tz = tz, usetz = TRUE)
}

chat$register_tool(tool(
  get_current_time,
  "Gets the current time in the given time zone.",
  tz = type_string(
    "The time zone to get the current time in. Defaults to `\"UTC\"`.",
    required = FALSE
  )
))

chat$chat("How long ago exactly was the moment Neil Armstrong touched down on the moon?")

extract_tool_calls <- function(chat_turns) {
  all_tool_calls <- list()
  for (j in seq_along(chat_turns)) {
    turn <- chat_turns[[j]]
    if (length(turn@contents) > 0) {
      for (content in turn@contents) {
        if (inherits(content, "ellmer::ContentToolRequest")) {
          tool_call <- list(
            id = content@id,
            name = content@name,
            arguments = content@arguments,
            role = turn@role
          )
          all_tool_calls[[j]] <- tool_call
        } else if (inherits(content, "ellmer::ContentToolResult")) {
          tool_result <- list(
            id = content@id,
            value = content@value,
            error = content@error,
            role = turn@role
          )
          all_tool_calls[[j]] <- tool_result
        }
      }
    }
  }
  all_tool_calls |> 
    bind_rows()
}

extract_tool_calls(chat$get_turns()) 

Most of the time this returns the following rows:

# A tibble: 2 × 5
  id                            name             arguments    role      value                  
  <chr>                         <chr>            <named list> <chr>     <chr>                  
1 call_apWd2L3gHLvA7AHaxlB5nNWm get_current_time <chr [1]>    assistant NA                     
2 call_apWd2L3gHLvA7AHaxlB5nNWm NA               <NULL>       user      2025-02-05 20:40:09 UTC

On a few occasions there are more rows because the assistant forgot to provide a timezone, so it sent another call 🙈