Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Address missing type in for SDK-based type conforming #299

Conversation

edgarrmondragon
Copy link
Member

@edgarrmondragon edgarrmondragon force-pushed the edgarrmondragon/fix/sdk-conforming-missing-type-key branch 2 times, most recently from af91bed to 5c3e833 Compare November 28, 2023 18:08
@Lawiss
Copy link

Lawiss commented Nov 30, 2023

Hello @edgarrmondragon, I like the explicit list of possible schema instead of empty object. In term of implementation, it seems that the SDK needs some modifications.

I'm using the data mentioned in #239 (comment).

The first example value is '[{"type": "DEPOT_BAG", "other": "", "quantity": 1}]'.

Using your branch, during the processing of the record (in my tests it is a list of objects ), the step of translation between record values and JSON compatible type will hit this if branch :
https://github.com/meltano/sdk/blob/d6c9ef09bdaee6af039440be7e0b47c97e0090bf/singer_sdk/helpers/_typing.py#L464

Then when trying to convert the list as a "primitive" type, the type matched will be the boolean type (https://github.com/meltano/sdk/blob/d6c9ef09bdaee6af039440be7e0b47c97e0090bf/singer_sdk/helpers/_typing.py#L493) :

    if is_boolean_type(property_schema):
        return None if elem is None else elem != 0

The record is here converted to True and the record is thus non conform to the data extracted.

I don't know if the is a risk to, in case of a list, directly output the record without any modification (relying entirely on sqlalchemy for the type) ?

@edgarrmondragon
Copy link
Member Author

@Lawiss Yeah I started running into the same problems in my own testing. The SDK is probably too eager when conforming types, e.g. I don't think it should try to cast a {"type": ["bool", "string"]} to boolean. Gotta re-think the approach here.

@edgarrmondragon edgarrmondragon force-pushed the edgarrmondragon/fix/sdk-conforming-missing-type-key branch from 5c3e833 to 7ac47d9 Compare December 1, 2023 00:15
@edgarrmondragon
Copy link
Member Author

Superseded by #307

@edgarrmondragon edgarrmondragon deleted the edgarrmondragon/fix/sdk-conforming-missing-type-key branch December 21, 2023 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants