I experienced this from a slightly different direction. Writing up my experience in case anyone has ideas for me or in case this will help anyone else:

I return responses including various custom types which aren't json-serializable and which aren't serializable by pydantic by default. Usually this looks something like this:

class SomeModel(BaseModel):
    regular_field: str
    nothing_special_here: int
    arbitrary_data: Dict[str, Any]

I didn't want to add custom serialization at the pydantic level in each model because it would be repetitive.
And I wasn't able to modify the types or subclass them to add custom pydantic serialization at the type level.

Before upgrading to pydantic 2, I added custom serialization at the FastAPI level using a custom response class and setting it as the default response class (along the lines of what was recommended here). Something like this:

def serialize_arbitrary_data(obj: Any) -> Any:
    ...

class MyJsonResponse(JSONResponse):
   def render(self, content: Any) -> bytes:
        return json.dumps(content, default=serialize_arbitrary_data)

This worked because the behavior with pydantic 1 is to leave unknown types alone when serializing.

When I upgraded to pydantic 2, this stopped working, because pydantic tries (and fails) to convert the custom types to json-serializable primitives before my custom response's render() is called. Even if the fields are typed as Any or annotated with SerializeAsAny this is the case, because FastAPI asks pydantic to model_dump(mode="json") before sending the data to the response class.

The same is true for adding encoders to fastapi.encoders.encoders_by_class_tuples; it is too late by the time they are used.

This is a reasonable design decision (to say that all conversion to json-serializable primitives must occur at the pydantic level, and only the JSON serialization itself can be controlled in the response class) but it definitely removed functionality that cannot be achieved in another way (that I have found so far).

In case any of the maintainers read this, would you be open to a feature allowing opt-in behavior of calling model_dump() with mode="python", or some similar workaround?

The next-best workaround I know of is to use the json_encoders config field in pydantic, but that's less than ideal for a few reasons:

1. It's "deprecated and will likely be removed in the future" according to the docs. It's worth noting that the reason it was not deleted yet is that there is no "1:1 replacement", so it may be reasonable to hope that if/when it is removed, an equivalent or better mechanism will be added to replace it.
2. If I want to get the behavior globally on all of my models, I need to subclass BaseModel and use it everywhere, or something like that. But that's also not great, since any models which define additional json_encoders would need to manually merge with the config of the base class, etc.
3. It won't work for pydantic models from third-party packages whose source I don't control.

In the meantime, I have (very redundantly) done the serialization in pydantic by annotating all of the models, using something like this:

def serialize_arbitrary_data(obj: Any) -> Any:
    ...

def recursively_serialize_arbitrary_data(obj: Any) -> Any:
    # recursively walk a nested structure, calling serialize_arbitrary_data() on leaves
    ...

ArbitraryData = Annotated[
    Dict[str, Any], PlainSerializer(lambda x: recursively_serialize_arbitrary_data(x))
]

class SomeModel(BaseModel):
    regular_field: str
    nothing_special_here: int
    arbitrary_data: ArbitraryData