Update chatbot templates to support agent serving endpoints #21
Conversation
…t message from agent response) Signed-off-by: Sid Murching <[email protected]>
Signed-off-by: Sid Murching <[email protected]>
Signed-off-by: Sid Murching <[email protected]>
Signed-off-by: Sid Murching <[email protected]>
smurching
requested review from
jerrylian-db and
theof-db
and removed request for
theof-db
Signed-off-by: Sid Murching <[email protected]>
| return res["messages"] | ||
| elif "choices" in res: | ||
| return [res["choices"][0]["message"]] | ||
| raise Exception("This app can only run against Databricks model serving endpoints that serve one of the conversational agent schemas documented in https://docs.databricks.com/aws/en/generative-ai/agent-framework/author-agent") |
There was a problem hiding this comment.
@smurching - Curious this would still continue to support foundational models?
There was a problem hiding this comment.
Ah yes, I need to update this message, thanks!
| @@ -1 +1,2 @@ | |||
| openai | |||
| mlflow | |||
| streamlit | |||
There was a problem hiding this comment.
Can we try to pin the library versions in the requirements.txt files across the templates? That would help the templates be stable across time. I've actually seen some templates break when restarted because the libraries were not pinned.
There was a problem hiding this comment.
You could inspect the app logs to see what library versions were used.
|
|
||
| if __name__ == '__main__': | ||
| app.run_server(debug=True) | ||
| app.run(debug=True) |
There was a problem hiding this comment.
Thanks for fixing this! The template is actually broke in production right now because dash 3+ does not support the run_server method anymore. 😅
| return [res["choices"][0]["message"]] | ||
| raise Exception("This app can only run against:" | ||
| "1) Databricks foundation model or external model endpoints with the chat task type (described in https://docs.databricks.com/aws/en/machine-learning/model-serving/score-foundation-models#chat-completion-model-query)" | ||
| "2) Databricks agent serving endpoints that implmeent the conversational agent schema documented " |
| "in https://docs.databricks.com/aws/en/generative-ai/agent-framework/author-agent") | ||
|
|
||
| def query_endpoint(endpoint_name, messages, max_tokens): | ||
| return _query_endpoint(endpoint_name, messages, max_tokens)[-1] |
There was a problem hiding this comment.
would it be possible to document that we will only take the last msg of the list of chatagent messages?
| "in https://docs.databricks.com/aws/en/generative-ai/agent-framework/author-agent") | ||
|
|
||
| def query_endpoint(endpoint_name, messages, max_tokens): | ||
| return _query_endpoint(endpoint_name, messages, max_tokens)[-1] |
There was a problem hiding this comment.
def doesn't have to be done here, but should we add a test to ensure that this file is the same across the diff chatbot templates to reduce room for potential error? cc @jerrylian-db
There was a problem hiding this comment.
Yeah, that's a good idea - it seems we don't have CI set up yet in this repo, but we should, and probably should have some tests for these apps
There was a problem hiding this comment.
Basic tests are probably sufficient (we can just follow best practices for testing a dash, streamlit, gradio app etc) and hopefully not too confusing to give to end users (we could also leave the tests outside the template, I'm fine with that too, but IMO they seem actually useful)
There was a problem hiding this comment.
I can send a follow-up with that change after this one, will discuss offline with @aakrati @jerrylian-db on what makes sense
There was a problem hiding this comment.
(filed #22 on top of the current PR, can iterate/follow up on this one after this initial PR lands)
Signed-off-by: Sid Murching <[email protected]>
jerrylian-db
left a comment
jerrylian-db
left a comment
There was a problem hiding this comment.
Thanks so much for this!
4 participants
Description
Updates chatbot templates to support running the template against agents deployed with Mosaic AI agent framework (https://docs.databricks.com/aws/en/generative-ai/agent-framework/author-agent)
How is this tested?
Tested manually by deploying the following apps against an agent serving & standard GPT4-o external model endpoint
Dash chatbot
Agent serving endpoint:
External model serving endpoint:
Gradio chatbot
Agent serving endpoint:
External model serving endpoint:
Shiny chatbot
Agent serving endpoint:
External model serving endpoint:
Streamlit chatbot
Agent serving endpoint:
External model serving endpoint:
