"API call didn't return a message" error repeating very frequently in v5.5, with openai.

[HG2407]

Describe the bug
A clear and concise description of what the bug is.
The issue is "API call didn't return a message", error is repeating very frequently with letta v5.5. I am using openai's gpt 4o. The problem is that the inner monologue is displayed everytime on the ui but there is no actual user response given. I have been able to debug the issue. It is because in the normal response, when everything is correct, the message.choices contains tool calls for send message. But whenever this issue arises the message.choices doesn't contain any function call. It should have been fixed according to a previous closed bug. But it is still there and is very frustating. I am attaching the logs as well.

correct normal response:
original response choices in unpack_all_inner_thoughts_from_kwargs: {
    "id": "message-fd4674bd-0943-4372-a1bd-55f84da2175d",
    "choices": [
        {
            "finish_reason": "tool_calls",
            "index": 0,
            "message": {
                "content": null,
                "tool_calls": [
                    {
                        "id": "6b6fe6f1-3edb-4a3a-b775-e0edd",
                        "type": "function",
                        "function": {
                            "arguments": "{\"inner_thoughts\":\"Keeping the interaction engaging. Ensuring consistency in responding to Harshit.\",\"message\":\"Hey there, Harshit! Let me know if there's anything specific you need or if you'd just like to chat.\"}",
                            "name": "send_message"
                        }
                    }
                ],
                "role": "assistant",
                "function_call": null
            },
            "logprobs": null,
            "seed": null
        }
    ],
    "created": "2024-12-15T19:25:37.356338Z",
    "model": "gpt-4o-2024-08-06",
    "system_fingerprint": "fp_a79d8dac1f",
    "object": "chat.completion",
    "usage": {
        "completion_tokens": 48,
        "prompt_tokens": 15582,
        "total_tokens": 15630
    }
}
_get_ai_reply response: {
    "id": "message-fd4674bd-0943-4372-a1bd-55f84da2175d",
    "choices": [
        {
            "finish_reason": "tool_calls",
            "index": 0,
            "message": {
                "content": "Keeping the interaction engaging. Ensuring consistency in responding to Harshit.",
                "tool_calls": [
                    {
                        "id": "6b6fe6f1-3edb-4a3a-b775-e0edd",
                        "type": "function",
                        "function": {
                            "arguments": "{\n  \"message\": \"Hey there, Harshit! Let me know if there's anything specific you need or if you'd just like to chat.\"\n}",
                            "name": "send_message"
                        }
                    }
                ],
                "role": "assistant",
                "function_call": null
            },
            "logprobs": null,
            "seed": null
        }
    ],
    "created": "2024-12-15T19:25:37.356338Z",
    "model": "gpt-4o-2024-08-06",
    "system_fingerprint": "fp_a79d8dac1f",
    "object": "chat.completion",
    "usage": {
        "completion_tokens": 48,
        "prompt_tokens": 15582,
        "total_tokens": 15630
    }
}

error causing response:
original response choices in unpack_all_inner_thoughts_from_kwargs: {
    "id": "message-4c44f4dd-95d5-45c1-ac8f-0c118a5b2967",
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "Hello again, Harshit! 👋 How can I assist you today? 😊",
                "tool_calls": null,
                "role": "assistant",
                "function_call": null
            },
            "logprobs": null,
            "seed": null
        }
    ],
    "created": "2024-12-15T19:20:23.558565Z",
    "model": "gpt-4o-2024-08-06",
    "system_fingerprint": "fp_a79d8dac1f",
    "object": "chat.completion",
    "usage": {
        "completion_tokens": 17,
        "prompt_tokens": 8289,
        "total_tokens": 8306
    }
}
_get_ai_reply response: {
    "id": "message-4c44f4dd-95d5-45c1-ac8f-0c118a5b2967",
    "choices": [
        null
    ],
    "created": "2024-12-15T19:20:23.558565Z",
    "model": "gpt-4o-2024-08-06",
    "system_fingerprint": "fp_a79d8dac1f",
    "object": "chat.completion",
    "usage": {
        "completion_tokens": 17,
        "prompt_tokens": 8289,
        "total_tokens": 8306
    }
}

Please describe your setup

• How did you install letta?
  • pip install letta? pip install letta-nightly? git clone?
  • git clone
• Describe your setup
  • What's your OS (Windows/MacOS/Linux)?
  • Ubuntu
  • How are you running letta? (cmd.exe/Powershell/Anaconda Shell/Terminal)
  • poetry run letta server

Screenshots
If applicable, add screenshots to help explain your problem.
correct response:

error response:

Additional context
Add any other context about the problem here.
I have also modified the parameters of _get_ai_reply, a little:

def _get_ai_reply(
        self,
        message_sequence: List[Message],
        function_call: str = "auto",
        first_message: bool = False,  # hint
        stream: bool = False,  # TODO move to config?
        fail_on_empty_response: bool = True, # this was initially False
        empty_response_retry_limit: int = 0 # this was initially 3
    ) -> ChatCompletionResponse:

Letta Config
Please attach your ~/.letta/config file or copy past it below.

---

If you're not using OpenAI, please provide additional information on your local LLM setup:

Local LLM details

If you are trying to run Letta with local LLMs, please provide the following information:

• The exact model you're trying to use (e.g. dolphin-2.1-mistral-7b.Q6_K.gguf)
• The local LLM backend you are using (web UI? LM Studio?)
• Your hardware for the local LLM backend (local computer? operating system? remote RunPod?)

[cpacker]

Are you able to reproduce this error in the latest version?

[HG2407]

Should I try with the latest version ? because this thing should have been fixed way before v5.5, so I am not sure.

[cpacker]

If you're able to reproduce with the latest version that would be great and help us debug the ticket faster (if the bug is still happening), whereas we're not able to actively debug issues for older version (eg v0.5.5)

[cpacker]

But whenever this issue arises the message.choices doesn't contain any function call

For example I don't think this should be happening when we use structured outputs, which should be on by default for gpt-4o-2024-08-06 in the latest version (but I'm not sure about v0.5.5).

[AliSayyah]

I have the exact issue with version 0.6.5 for existing agents. new agents work fine.
btw, I'm using gpt-4o-mini.

[AliSayyah]

Tested the 0.6.6 and the issue still remains.

[HG2407]

I suspect this error is arising because we are trying to make openai call the send message which is a custom function we have provided, but due to the system message, agent persona and human persona added to the context of the current message, which may include 2-3 function calls, leads to context limit being filled in openai. leading to the send message not being called. just a thought.

[cpacker]

I suspect this error is arising because we are trying to make openai call the send message which is a custom function we have provided, but due to the system message, agent persona and human persona added to the context of the current message, which may include 2-3 function calls, leads to context limit being filled in openai. leading to the send message not being called. just a thought.

I don't think this is the case since OpenAI throws a special context overflow related error when this happens, which Letta has a special catch for ( "finish_reason": "stop", will be something specific).

Tested the 0.6.6 and the issue still remains.

@AliSayyah interesting - so you get this error on old agents, but not new agents. Are you able to share more details about the agent payloads?

For example, it would really help debug if you're able to share the output of /dumpraw in the Letta CLI for both the old agents that aren't working, and the new agents that are working. That might help us ID if the issue has to do with something in the context window (or it could be a model setting being off and not triggering the same backend settings?).

[HG2407]

@cpacker does your latest pr solve the issue ? Can you explain how you solved it ?

[cpacker]

@HG2407 / @AliSayyah yep I think the PR should fix the bug! Basically, there was a bug in the interpretation of some API calls that were missing tool calls, which turns the entire message choice into null, instead of just the tool choice section.

Will be part of the next release which should be out asap! If you want to try out the fix now you can try pulling main or use the nightly builds.

[AliSayyah]

@cpacker I tried the nightly build and the issue remains, unfortunately.

[HG2407]

I can confirm @cpacker I am using the commit #2311

[cpacker]

@AliSayyah @HG2407 are either of you able to provide me a quick way to reproduce the issue on the latest docker or source?

For example, something like:

1. Run docker run with latest nightly
2. Create a new agent (select X starter kit) in the ADE
3. Set model to gpt-4o-mini
4. Say "Y" in the chat
5. ^if you do (4) N times, you'll reproduce the error

[AliSayyah]

@cpacker I haven't found any particular pattern to reproduce it, but I'll try again today.

[HG2407]

@cpacker this issue usually arises when the agent needs to call two three functions in one go, sequentially. At that time, usually 2-3 functions are called then the request ends.

[AliSayyah]

 Letta.letta.server.server - ERROR - Error in server._step: HTTP error occurred: 400 Client Error: Bad Request for url: https://api.openai.com/v1/chat/completions | Status code: 400, Message: {
   "error": {
     "message": "Invalid value for 'content': expected a string, got null.",
     "type": "invalid_request_error",
     "param": "messages.[161].content",
     "code": null
   }
 }
 Traceback (most recent call last):
   File "/app/letta/llm_api/helpers.py", line 142, in make_post_request
     response.raise_for_status()
   File "/app/.venv/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
     raise HTTPError(http_error_msg, response=self)

On the latest version (0.6.7) I get this error message. maybe a different issue?

[cpacker]

interesting - @AliSayyah is it possible to share the payload (request) going to OpenAI here? Feel free to replace any sensitive strings - I'm more curious how this payload ended up happening.

Is it a user role message that's empty? Is it an assistant role message that has no tools + no content? (since I think no content is fine if there's tools?)

[cpacker]

Also - is the agent here effectively "bricked"? E.g. every step/message causes this error? Or is it stochastic?

[AliSayyah]

interesting - @AliSayyah is it possible to share the payload (request) going to OpenAI here? Feel free to replace any sensitive strings - I'm more curious how this payload ended up happening.

Is it a user role message that's empty? Is it an assistant role message that has no tools + no content? (since I think no content is fine if there's tools?)

Do I need to edit the source code to do this?is there an easier way?
Yes, it's user role messages.
and this error raises every step.