Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Bad token estimation in Prompty #3809

Open
ianchi opened this issue Oct 11, 2024 · 1 comment · May be fixed by #3821
Open

[BUG] Bad token estimation in Prompty #3809

ianchi opened this issue Oct 11, 2024 · 1 comment · May be fixed by #3821
Labels
bug Something isn't working

Comments

@ianchi
Copy link

ianchi commented Oct 11, 2024

Describe the bug
Token estimation is wrong, as it doesn't take into account the functions/tools definition as part of the prompt.
In cases with heavy use of tools this can make a big diference and make you take a wrong decision regarding the token limit of a request.

How To Reproduce the bug
Steps to reproduce the behavior, how frequent can you experience the bug:

  1. Create a prompty file with a large tools definition
  2. Make an estimation with a small question
  3. compare with the actual token consumption reported by oai response

Expected behavior
A clear and concise description of what you expected to happen.

  • Add an additional function to also estimate the tools definition token consumption

Screenshots
If applicable, add screenshots to help explain your problem.

Running Information(please complete the following information):

  • Promptflow Package Version using pf -v: [e.g. 0.0.102309906]
  • Operating System: [e.g. Ubuntu 20.04, Windows 11]: Debian
  • Python Version using python --version: [e.g. python==3.10.12]: 3.11.5

Additional context
Add any other context about the problem here.
@ianchi ianchi added the bug Something isn't working label Oct 11, 2024
@ianchi
Copy link
Author

ianchi commented Oct 13, 2024 •
edited
Loading

Additionally token from messages is also incorrect.
It doesn't take into account when an assistant's message has tool_calls

And when using Azure OpenAI the wrong model type may be selected.
num_tokens_from_messages expects the model as string using one of OpenAI model's labels, but in this case the azure_deployment is used. This is a user generated label and not necessarily a valid model name, so the token count raises error and can´t be used.

@ianchi ianchi linked a pull request Oct 19, 2024 that will close this issue
Open
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

prompty: improve token count estimation ianchi/promptflow
1 participant
@ianchi