I created a initial Draft on how this could be done in #1622, but this is not DRY. Once this has been vetted I would refactor the code and write the appropriate tests.

I'm facing the same issue when my model outputs a numpy.array of string. It works fine when invoked through the REST API, but failed when invoked via Kafka server. I believe this is because of the difference on how InferenceResponse object is serialized. FastAPI uses the fastapi.encoders.jsonable_encoder function, while KafkaMessage uses json.dumps(InferenceResponse.dict()) [code]

I tried adding jsonable_encoder prior to json.dumps in KafkaMessage encoder function and it seems to have solved the issue.

from fastapi.encoders import jsonable_encoder

def _encode_value(v: dict) -> bytes:
    v = jsonable_encoder(v)

    if orjson is None:
        dumped = json.dumps(v)
        return dumped.encode("utf-8")

    return orjson.dumps(v)

Using pydantic's encoder as the default encoder is also a possible solution (for pydantic<2).

from pydantic.json import pydantic_encoder

def _encode_value(v: dict) -> bytes:
    if orjson is None:
        dumped = json.dumps(v, default=pydantic_encoder)
        return dumped.encode("utf-8")

    return orjson.dumps(v, default=pydantic_encoder)

@hansen-young we merged a contribution from @DerTiedemann #1622 that fixes this issue. Please submit a new ticket if there are things still that need to be fixed.