Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(taps,targets): Support the x-singer.decimal JSON Schema extension #2786

Merged
merged 23 commits into from
Mar 8, 2025

Conversation

edgarrmondragon
Copy link
Collaborator

@edgarrmondragon edgarrmondragon commented Nov 30, 2024

TODO:

  • Docs
  • Capability/Built-in setting

Related:


📚 Documentation preview 📚: https://meltano-sdk--2786.org.readthedocs.build/en/2786/

Summary by Sourcery

Adds support for the singer.decimal JSON Schema extension. This allows numbers to be represented as strings with the singer.decimal format instead of as numbers. A new use_singer_decimal option is added to the SQLToJSONSchema class to control whether numbers are represented as string with the singer.decimal format or as number.

New Features:

  • Adds support for the singer.decimal JSON Schema extension, allowing numbers to be represented as strings with the singer.decimal format instead of as numbers.

Enhancements:

  • Adds a use_singer_decimal option to the SQLToJSONSchema class to control whether numbers are represented as string with the singer.decimal format or as number.

Tests:

  • Adds a test case for converting numeric types to singer.decimal format.
  • Adds a test case for converting singer.decimal types to SQLAlchemy DECIMAL types.

@edgarrmondragon edgarrmondragon added SQL Support for SQL taps and targets Type/Target Singer targets Type/Tap Singer taps labels Nov 30, 2024
@edgarrmondragon edgarrmondragon self-assigned this Nov 30, 2024
Copy link

codecov bot commented Nov 30, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.37%. Comparing base (aa9b2b2) to head (d710ac4).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2786      +/-   ##
==========================================
+ Coverage   91.35%   91.37%   +0.02%     
==========================================
  Files          63       63              
  Lines        5214     5227      +13     
  Branches      675      676       +1     
==========================================
+ Hits         4763     4776      +13     
  Misses        319      319              
  Partials      132      132              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link

codspeed-hq bot commented Nov 30, 2024

CodSpeed Performance Report

Merging #2786 will not alter performance

Comparing edgarrmondragon/feat/singer-decimal (d710ac4) with main (aa9b2b2)

Summary

✅ 7 untouched benchmarks

@edgarrmondragon edgarrmondragon force-pushed the edgarrmondragon/refactor/jsonschematosql-fromconfig branch from de53f85 to 42a702d Compare November 30, 2024 20:27
@s7clarke10
Copy link
Contributor

That's awesome, thanks once again @edgarrmondragon 🙌

@edgarrmondragon edgarrmondragon force-pushed the edgarrmondragon/feat/singer-decimal branch from ff67a40 to 8e95d23 Compare January 29, 2025 15:20
@edgarrmondragon edgarrmondragon added this to the v0.45 milestone Jan 29, 2025
@edgarrmondragon edgarrmondragon force-pushed the edgarrmondragon/feat/singer-decimal branch from e27b663 to 9df828b Compare January 30, 2025 00:38
@edgarrmondragon edgarrmondragon force-pushed the edgarrmondragon/feat/singer-decimal branch from 584d3b5 to 6ec48ae Compare January 30, 2025 00:41
Base automatically changed from edgarrmondragon/refactor/jsonschematosql-fromconfig to main February 24, 2025 16:38
@edgarrmondragon edgarrmondragon marked this pull request as ready for review February 24, 2025 16:39
Copy link
Contributor

sourcery-ai bot commented Feb 24, 2025

Reviewer's Guide by Sourcery

This pull request introduces support for the singer.decimal JSON Schema extension. It modifies the SQLToJSONSchema class to allow representing numeric values as strings with the singer.decimal format. It also modifies the JSONSchemaToSQL class to handle the singer.decimal format and return the appropriate SQLAlchemy type. Finally, it adds a SingerDecimalType to typing.py.

Class diagram for SingerDecimalType

classDiagram
  class SingerDecimalType {
    +string_format = "singer.decimal"
  }
  SingerDecimalType --|> StringType
  class StringType
Loading

File-Level Changes

Change Details Files
Added support for the singer.decimal JSON Schema extension, allowing numeric values to be represented as strings with a specific format.
  • Added use_singer_decimal parameter to SQLToJSONSchema to control whether numbers are represented as string with the singer.decimal format.
  • Implemented float_to_jsonschema to return SingerDecimalType when use_singer_decimal is enabled.
  • Added _handle_singer_decimal to JSONSchemaToSQL to handle the singer.decimal format and return the appropriate SQLAlchemy type.
  • Added SingerDecimalType to typing.py.
singer_sdk/connectors/sql.py
tests/core/test_connector_sql.py
singer_sdk/typing.py

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!
  • Generate a plan of action for an issue: Comment @sourcery-ai plan on
    an issue to generate a plan of action for it.

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @edgarrmondragon - I've reviewed your changes - here's some feedback:

Overall Comments:

  • Consider adding a default value for max_varchar_length in the JSONSchemaToSQL constructor to avoid breaking changes.
  • It might be worth adding a test case that checks the precision and scale are passed through correctly to the SQLAlchemy DECIMAL type.
Here's what I looked at during the review
  • 🟢 General issues: all looks good
  • 🟢 Security: all looks good
  • 🟡 Testing: 2 issues found
  • 🟢 Complexity: all looks good
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

edgarrmondragon and others added 2 commits February 24, 2025 11:20
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
@edgarrmondragon edgarrmondragon added the Release Highlight Call this out in the release notes label Mar 7, 2025
@edgarrmondragon edgarrmondragon changed the title feat(taps,targets): Support the singer.decimal JSON Schema extension feat(taps,targets): Support the x-singer.decimal JSON Schema extension Mar 7, 2025
@edgarrmondragon edgarrmondragon requested review from a team as code owners March 7, 2025 20:03
@edgarrmondragon
Copy link
Collaborator Author

@sourcery-ai review

@edgarrmondragon edgarrmondragon merged commit 1bc35ff into main Mar 8, 2025
36 checks passed
@edgarrmondragon edgarrmondragon deleted the edgarrmondragon/feat/singer-decimal branch March 8, 2025 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Release Highlight Call this out in the release notes SQL Support for SQL taps and targets Type/Tap Singer taps Type/Target Singer targets
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: Support Singer Decimal as a config item
2 participants