Token Usage

[nurkmez2]

Hi @marcominerva,
Thanks for your help regarding the token usage.

Can you explain what you mean by the following fields?
Which values will provide the total tokens I consume for this request in the azure model?

For example;

Answer.TokenUsage =
{
"serviceType": "Azure OpenAI",
"modelType": "TextGeneration",
"modelName": "gpt-4o",
"tokenizerTokensIn": 6512,
"tokenizerTokensOut": 454,
"serviceTokensIn": 6519,
"serviceTokensOut": 454,
"serviceReasoningTokens": 0
}

Total Consumed Token is "serviceTokensIn": 6519 + "serviceTokensOut": 454 = 6973 or not ?

Thanks

[dluc]

Total Consumed Token is "serviceTokensIn": 6519 + "serviceTokensOut": 454 = 6973 or not ?

That's correct, though you should also consider serviceReasoningTokens. In other words, tokens whose name start with "service" are the values being billed by the remote service (Azure, OpenAI, Anthropic, etc):

• "serviceTokensIn": 6519,
• "serviceTokensOut": 454,
• "serviceReasoningTokens": 0

The other two values:

• "tokenizerTokensIn": 6512,
• "tokenizerTokensOut": 454,

are local measurements done by KM tokenizers, which can be useful in case the remote model doesn't expose any metrics, such as Ollama.

[dluc]

Note that the report is a "list" of reports. Right now, the list includes just one report, if and when future implementations will make multiple LLM requests, the answer will include multiple reports, one for each LLM request.

In fact, we could already improve the report, adding the cost of the embedding generation. /cc @marcominerva

[marcominerva]

@dluc The GenerateEmbeddingsAsync method provided by Sematic Kernel ITextEmbeddingGeneratorService interface returns the embeddings only, it does not provide information about token usage. So, in this case we can only set the tokenizerTokensIn value, are you agree?

[dluc]

in this case we can only set the tokenizerTokensIn value, are you agree?

yes, that's the best we can provide for now

[nurkmez2]

Thanks 👍