Support defining configuration and stream schemas using Pydantic
Pydantic is the most popular data validation and serialization library for Python at the moment. It is used by the new-ish and increasingly popular web framework FastAPI and by other frameworks wanting to support reliable data validation and serialization (like odmantic for MongoDB and pydantic-sqlalchemy).
Alternatives include mashumaro, marshmallow and dataclasses-json. All of these support only a subset of Pydantic's features (no validation, serialization but no deserialization).
I propose leveraging Pydantic to allow the SDK user to define a plugin's configuration and inline stream schemas using a known, powerful and well-documented library. The implementation would look something like the following:
from pydantic import BaseModel
class BaseSchema(BaseModel):
# Created by the SDK developers to support the specifics of Singer schemas
class TapTestConfig(BaseSchema):
# Plugin config created by the SDK user
class ExampleStreamSchema(BaseSchema):
# Stream schema created by the SDK user
Notes:
-
BaseSchema
needs to extend pydantic to make sure schemas are Singer-compatible: https://pydantic-docs.helpmanual.io/usage/schema/#schema-customization - Initial discussion for pydantic use: https://gitlab.com/meltano/singer-sdk/-/issues/10#note_519595958