[BUG] Running Prompt flow locally produces errors #3751

rlerch · 2024-09-13T15:55:22Z

Describe the bug
Running a flow using pf test seems to work but exceptions are reported locally.

How To Reproduce the bug
Run pf test on a flow locally. The flow executes successfully but exceptions are generated when collecting token metrics for openai.

Expected behavior
A clean run without exceptions

Screenshots

WARNING:opentelemetry.attributes:Invalid type NoneType for attribute 'computed.cumulative_token_count.completion' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types
WARNING:opentelemetry.attributes:Invalid type NoneType for attribute 'llm.usage.completion_tokens_details' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types
WARNING:opentelemetry.attributes:Invalid type NoneType for attribute 'computed.cumulative_token_count.completion' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types

Running Information(please complete the following information):
{
"promptflow": "1.15.0",
"promptflow-azure": "1.15.0",
"promptflow-core": "1.15.0",
"promptflow-devkit": "1.15.0",
"promptflow-tracing": "1.15.0"
}

Executable 'c:\git\azure-ai-prompt-flow.venv\Scripts\python.exe'
Python (Windows) 3.10.5 (tags/v3.10.5:f377153, Jun 6 2022, 16:14:13) [MSC v.1929 64 bit (AMD64)]
Additional context
Seems like the exception is in here

    def collect_openai_tokens_for_parent_span(self, span):
        tokens = self.try_get_openai_tokens(span.get_span_context().span_id)
        if tokens:
            if not hasattr(span, "parent") or span.parent is None:
                return
            parent_span_id = span.parent.span_id
            with self._lock:
                if parent_span_id in self._span_id_to_tokens:
                    merged_tokens = {
                        key: self._span_id_to_tokens[parent_span_id].get(key, 0) + tokens.get(key, 0)
                        for key in set(self._span_id_to_tokens[parent_span_id]) | set(tokens)
                    }
                    self._span_id_to_tokens[parent_span_id] = merged_tokens
                else:
                    self._span_id_to_tokens[parent_span_id] = tokens

On the line key: self._span_id_to_tokens[parent_span_id].get(key, 0) + tokens.get(key, 0) . Not all steps in my flow are LLM steps. I wonder if this could be causing the issue where we have some steps in the flow that don't have any tokens.

The text was updated successfully, but these errors were encountered:

berndku · 2024-09-13T17:45:56Z

We see something similar, however in our case it is an error and failing the execution:

File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/promptflow/tracing/_trace.py", line 144, in <dictcomp>
key: self._span_id_to_tokens[parent_span_id].get(key, 0) + tokens.get(key, 0)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'

jwieler · 2024-09-13T18:08:51Z

Seeing similar errors on my end, this is halting flow execution:

2024-09-13 14:04:35 -0400 46456 execution.flow WARNING Failed to calculate metrics due to exception: unsupported operand type(s) for +: 'int' and 'dict'.

mack-adknown · 2024-09-13T21:08:06Z

Same issue, resulting in flow being terminated:
WARNING:opentelemetry.attributes:Invalid type dict for attribute '__computed__.cumulative_token_count.completion' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types 2024-09-13 16:53:20 -0400 28812 execution.flow WARNING Failed to calculate metrics due to exception: unsupported operand type(s) for +: 'int' and 'dict'. 2024-09-13 16:53:20 -0400 28812 execution.flow ERROR Flow execution has failed. Cancelling all running nodes: extract_data.

SULAPIS · 2024-09-14T05:36:37Z

Some keys in the tokens have values that are not int, causing the issue.

azure_open_ai: tokens = {'completion_tokens': 13, 'prompt_tokens': 673, 'total_tokens': 686, 'completion_tokens_details': None}
open_ai: {'completion_tokens': 13, 'prompt_tokens': 669, 'total_tokens': 682, 'completion_tokens_details': {'reasoning_tokens': 0}}

As a temporary solution, tracing can be disabled by setting PF_DISABLE_TRACING=true.

asos-oliverfrost · 2024-09-15T09:44:22Z

Rolling back the version of openai also works, openai<=1.44.1 resolves the error. Looks like promptflow-tracing could be incompatible with 1.45.0 https://pypi.org/project/openai/#history

jomalsan · 2024-09-16T01:37:07Z

@asos-oliverfrost thank you for finding that! Rolling back worked for me. It looks like this is the specific line that is breaking the behavior in prompt flow, because the new field completion_token_details is optional: openai/openai-python@v1.44.1...v1.45.0#diff-d85f41ac9f419751206af46c34ef5c8c74258660be492aa703dcbebcfc96a41bR25

olopezqubika · 2024-09-19T15:47:46Z

Rolling back OpenAI did not work for me, Pydantic is so dynamic that I still get completion_tokens_details={'reasoning_tokens': 0}. In summary:

An OpenAI endpoint returns a dictionary inside another dictionary: {"completion_token_details":{"foo_bar":0}}.
Then it's processed by promflow.tracing, which does not support that.

cfoster0 · 2024-09-20T01:28:14Z

Also having this issue, with similar errors:

2024-09-19 18:05:38 -0700   89862 execution          ERROR    Node extract_result in line 0 failed. Exception: Execution failure in 'extract_result': (TypeError) unsupported operand type(s) for +: 'dict' and 'dict'.
Traceback (most recent call last):
  File "/[...]/pypoetry/virtualenvs/promptflow-test-gto235yr-py3.11/lib/python3.11/site-packages/promptflow/_core/flow_execution_context.py", line 182, in _invoke_tool_inner
    return f(**kwargs)
           ^^^^^^^^^^^
  File "/[...]/pypoetry/virtualenvs/promptflow-test-gto235yr-py3.11/lib/python3.11/site-packages/promptflow/tracing/_trace.py", line 561, in wrapped
    token_collector.collect_openai_tokens_for_parent_span(span)
  File "/[...]/pypoetry/virtualenvs/promptflow-test-gto235yr-py3.11/lib/python3.11/site-packages/promptflow/tracing/_trace.py", line 143, in collect_openai_tokens_for_parent_span
    merged_tokens = {
                    ^
  File "/[...]/pypoetry/virtualenvs/promptflow-test-gto235yr-py3.11/lib/python3.11/site-packages/promptflow/tracing/_trace.py", line 144, in <dictcomp>
    key: self._span_id_to_tokens[parent_span_id].get(key, 0) + tokens.get(key, 0)
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for +: 'dict' and 'dict'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/[...]/pypoetry/virtualenvs/promptflow-test-gto235yr-py3.11/lib/python3.11/site-packages/promptflow/_core/flow_execution_context.py", line 90, in invoke_tool
    result = self._invoke_tool_inner(node, f, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/[...]/pypoetry/virtualenvs/promptflow-test-gto235yr-py3.11/lib/python3.11/site-packages/promptflow/_core/flow_execution_context.py", line 206, in _invoke_tool_inner
    raise ToolExecutionError(node_name=node_name, module=module) from e
promptflow._core._errors.ToolExecutionError: Execution failure in 'extract_result': (TypeError) unsupported operand type(s) for +: 'dict' and 'dict'
2024-09-19 18:05:38 -0700   89862 execution.flow     WARNING  Failed to calculate metrics due to exception: unsupported operand type(s) for +: 'int' and 'dict'.
2024-09-19 18:05:38 -0700   89862 execution.flow     ERROR    Flow execution has failed. Cancelling all running nodes: extract_result.
pf.flow.test failed with UserErrorException: TypeError: Execution failure in 'extract_result': (TypeError) unsupported operand type(s) for +: 'dict' and 'dict'

This happens when running a version of the chat-math-variant example, edited so that extract_text.py calls the OpenAI ChatCompletions endpoint. One of my coworkers is also seeing this error, seemingly from an LLM tool call.

mallapraveen · 2024-09-25T05:30:37Z

Rolling back the version of openai also works, openai<=1.44.1 resolves the error. Looks like promptflow-tracing could be incompatible with 1.45.0 https://pypi.org/project/openai/#history

This worked for me

HakjunMIN · 2024-09-26T01:27:44Z

oh. @mallapraveen @jomalsan YOu saved me. Thank you millions.

JatinGargQA · 2024-09-27T07:07:58Z

Traceback (most recent call last):
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py", line 182, in _invoke_tool_inner
return f(**kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 561, in wrapped
token_collector.collect_openai_tokens_for_parent_span(span)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 143, in collect_openai_tokens_for_parent_span
merged_tokens = {
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 144, in
key: self._span_id_to_tokens[parent_span_id].get(key, 0) + tokens.get(key, 0)
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

Getting this error.

matheuslazarin · 2024-09-27T17:38:25Z

I'm able to run locally, but I get TypeError: unsupported operand type(s) for +: 'int' and 'NoneType' after deploying using Azure AI Studio

oussamachiboub · 2024-09-29T16:06:20Z

Rolling back the version of openai also works, openai<=1.44.1 resolves the error. Looks like promptflow-tracing could be incompatible with 1.45.0 https://pypi.org/project/openai/#history

This did the trick for me !! thank you so much <3

hiroki0525 · 2024-10-01T00:35:23Z

I have found PR that may be relevant to this issue.

#3793

teebu · 2024-10-02T00:10:21Z

I'm also noticing this issue recently (was working a month ago) when deploying prompt flow from Azure AI Studio, created from Chat playground.

JanWerder · 2024-10-02T07:58:41Z

We encountered the problem as well.
Adding the following lines to the promptflow YAML worked:

environment_variables:
  PF_DISABLE_TRACING: true

espanto28 · 2024-10-02T17:51:07Z

As @JanWerder suggested this did the trick for me

environment_variables:
  PF_DISABLE_TRACING: true

luigiw · 2024-10-09T07:29:04Z

This is fixed with promptflow-tracing 1.16.1

WilliamWatsonEAI · 2024-10-18T01:34:53Z

@luigiw It does not seem that this fix works with the base docker promptflow runtime image. As the image is still based on Python 3.9

scherici · 2024-10-23T08:08:06Z

@luigiw after updating to 1.16.1 the error goes away, and that is good, but a lot of warning still persists:

WARNING:opentelemetry.attributes:Invalid type NoneType for attribute '__computed__.cumulative_token_count.completion' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute '__computed__.cumulative_token_count.prompt' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute 'llm.usage.completion_tokens_details' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute 'llm.usage.prompt_tokens_details' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute '__computed__.cumulative_token_count.completion' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute '__computed__.cumulative_token_count.prompt' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types

And what I notice on AppInsights if that the logs for llm tools shows token consumptions (completion, prompt and total); but the "same" metrics for te whole flow shows only cosumption token for total token and "0" for the other two:

Leaf tool nod:

1 level up:

root:

scherici · 2024-10-23T09:21:45Z

@luigiw after updating to 1.16.1 the error goes away, and that is good, but a lot of warning still persists:

WARNING:opentelemetry.attributes:Invalid type NoneType for attribute '__computed__.cumulative_token_count.completion' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute '__computed__.cumulative_token_count.prompt' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute 'llm.usage.completion_tokens_details' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute 'llm.usage.prompt_tokens_details' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute '__computed__.cumulative_token_count.completion' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types WARNING:opentelemetry.attributes:Invalid type NoneType for attribute '__computed__.cumulative_token_count.prompt' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types

And what I notice on AppInsights if that the logs for llm tools shows token consumptions (completion, prompt and total); but the "same" metrics for te whole flow shows only cosumption token for total token and "0" for the other two:

Leaf tool nod:

1 level up:

root:

I have found PR that may be relevant to this issue: #3806

rlerch added the bug Something isn't working label Sep 13, 2024

hiroki0525 mentioned this issue Oct 7, 2024

[BUG]: ERROR "(TypeError) unsupported operand type(s) for +: 'int' and 'NoneType'" in flow_execution_context.py / _trace.py #3796

Closed

luigiw closed this as completed Oct 9, 2024

ckittel mentioned this issue Oct 16, 2024

Fix missing identity role assignment Azure-Samples/openai-end-to-end-basic#14

Merged

WilliamWatsonEAI mentioned this issue Oct 18, 2024

[Feature Request] Change/Update the Python version for the promptflow-runtime:latest docker image #3817

Open

ckittel mentioned this issue Oct 18, 2024

Transition to Azure AI Studio Azure-Samples/openai-end-to-end-basic#19

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Running Prompt flow locally produces errors #3751

[BUG] Running Prompt flow locally produces errors #3751

rlerch commented Sep 13, 2024

berndku commented Sep 13, 2024

jwieler commented Sep 13, 2024

mack-adknown commented Sep 13, 2024

SULAPIS commented Sep 14, 2024

asos-oliverfrost commented Sep 15, 2024 •

edited

Loading

jomalsan commented Sep 16, 2024

olopezqubika commented Sep 19, 2024 •

edited

Loading

cfoster0 commented Sep 20, 2024

mallapraveen commented Sep 25, 2024

HakjunMIN commented Sep 26, 2024

JatinGargQA commented Sep 27, 2024

matheuslazarin commented Sep 27, 2024

oussamachiboub commented Sep 29, 2024

hiroki0525 commented Oct 1, 2024

teebu commented Oct 2, 2024 •

edited

Loading

JanWerder commented Oct 2, 2024

espanto28 commented Oct 2, 2024

luigiw commented Oct 9, 2024

WilliamWatsonEAI commented Oct 18, 2024

scherici commented Oct 23, 2024

scherici commented Oct 23, 2024

[BUG] Running Prompt flow locally produces errors #3751

[BUG] Running Prompt flow locally produces errors #3751

Comments

rlerch commented Sep 13, 2024

berndku commented Sep 13, 2024

jwieler commented Sep 13, 2024

mack-adknown commented Sep 13, 2024

SULAPIS commented Sep 14, 2024

asos-oliverfrost commented Sep 15, 2024 • edited Loading

jomalsan commented Sep 16, 2024

olopezqubika commented Sep 19, 2024 • edited Loading

cfoster0 commented Sep 20, 2024

mallapraveen commented Sep 25, 2024

HakjunMIN commented Sep 26, 2024

JatinGargQA commented Sep 27, 2024

matheuslazarin commented Sep 27, 2024

oussamachiboub commented Sep 29, 2024

hiroki0525 commented Oct 1, 2024

teebu commented Oct 2, 2024 • edited Loading

JanWerder commented Oct 2, 2024

espanto28 commented Oct 2, 2024

luigiw commented Oct 9, 2024

WilliamWatsonEAI commented Oct 18, 2024

scherici commented Oct 23, 2024

scherici commented Oct 23, 2024

asos-oliverfrost commented Sep 15, 2024 •

edited

Loading

olopezqubika commented Sep 19, 2024 •

edited

Loading

teebu commented Oct 2, 2024 •

edited

Loading