What happens
temporalio.contrib.pydantic.PydanticJSONPlainPayloadConverter.from_payload constructs a fresh TypeAdapter for every payload:
return TypeAdapter(_type_hint).validate_json(payload.data)
For hints that are BaseModel subclasses this is masked by pydantic's class-level core-schema cache. But activity arg/result hints are frequently not classes — Annotated discriminated unions, list[Union[...]], generics — and for those TypeAdapter.__init__ rebuilds the core schema on every single payload. Construction dominates validation by orders of magnitude.
Measured
Toy 3-member discriminated union (runnable below), pydantic 2.x, Python 3.12:
build+validate per payload: 0.134 ms
validate only: 0.0023 ms
ratio: 57x
Production shape (pydantic-ai's CallToolResult, a wide Annotated discriminated union — PydanticAIPlugin inherits this converter): 2.818 ms build vs 0.005 ms validate, 521×. In a wedged activation resolving 792 concurrent activity completions, 11.935 s of the 12.089 s activation was TypeAdapter.__init__ — enough on its own to breach a Lambda worker's deadlock-detection ceiling. It also makes replay and continue-as-new rebuilds scale with history size × union width.
Repro
import time
from typing import Annotated, Literal, Union
from pydantic import BaseModel, Field, TypeAdapter
class TextPart(BaseModel):
kind: Literal["text"] = "text"
text: str
class DataPart(BaseModel):
kind: Literal["data"] = "data"
payload: dict[str, str]
class ErrorPart(BaseModel):
kind: Literal["error"] = "error"
message: str
code: int
Part = Annotated[Union[TextPart, DataPart, ErrorPart], Field(discriminator="kind")]
ListPart = list[Part]
doc = '[{"kind":"text","text":"' + "x" * 200 + '"},{"kind":"data","payload":{"a":"b"}},{"kind":"error","message":"m","code":1}]'
N = 200
t0 = time.perf_counter()
for _ in range(N):
TypeAdapter(ListPart).validate_json(doc)
per_payload = (time.perf_counter() - t0) / N * 1000
adapter = TypeAdapter(ListPart)
t0 = time.perf_counter()
for _ in range(N):
adapter.validate_json(doc)
validate_only = (time.perf_counter() - t0) / N * 1000
print(f"build+validate per payload: {per_payload:.3f} ms")
print(f"validate only: {validate_only:.4f} ms")
print(f"ratio: {per_payload / validate_only:.0f}x")Suggested fix
Memoize the adapter on the type hint. We run exactly that in production as a converter subclass — an LRU keyed on the hint (falling back to the un-memoized path for unhashable hints), which took our peak activation from 3.900 s → 0.200 s with no behavior change: the memo returns exactly what a fresh adapter returns for the same inputs, so histories replay identically. Happy to turn that into a PR against contrib/pydantic.py if you'd take it — the only design question is cache scope/eviction (process-global lru_cache(maxsize=...) on a module function was enough for us).
What happens
temporalio.contrib.pydantic.PydanticJSONPlainPayloadConverter.from_payloadconstructs a freshTypeAdapterfor every payload:For hints that are
BaseModelsubclasses this is masked by pydantic's class-level core-schema cache. But activity arg/result hints are frequently not classes —Annotateddiscriminated unions,list[Union[...]], generics — and for thoseTypeAdapter.__init__rebuilds the core schema on every single payload. Construction dominates validation by orders of magnitude.Measured
Toy 3-member discriminated union (runnable below), pydantic 2.x, Python 3.12:
Production shape (pydantic-ai's
CallToolResult, a wideAnnotateddiscriminated union —PydanticAIPlugininherits this converter): 2.818 ms build vs 0.005 ms validate, 521×. In a wedged activation resolving 792 concurrent activity completions, 11.935 s of the 12.089 s activation wasTypeAdapter.__init__— enough on its own to breach a Lambda worker's deadlock-detection ceiling. It also makes replay and continue-as-new rebuilds scale with history size × union width.Repro
Suggested fix
Memoize the adapter on the type hint. We run exactly that in production as a converter subclass — an LRU keyed on the hint (falling back to the un-memoized path for unhashable hints), which took our peak activation from 3.900 s → 0.200 s with no behavior change: the memo returns exactly what a fresh adapter returns for the same inputs, so histories replay identically. Happy to turn that into a PR against
contrib/pydantic.pyif you'd take it — the only design question is cache scope/eviction (process-globallru_cache(maxsize=...)on a module function was enough for us).