Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add default limit for fetching failed rows in DbSample #1991

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add default limit for fetching failed rows in DbSample #1991

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0a4d9f8
Add default limit for fetching failed rows in DbSample
Jan 18, 2024
d2fcf82
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions soda/core/soda/sampler/db_sample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

from soda.sampler.sample import Sample
from soda.sampler.sample_schema import SampleColumn, SampleSchema
from soda.sampler.sampler import DEFAULT_FAILED_ROWS_SAMPLE_LIMIT


class DbSample(Sample):
Expand All @@ -11,9 +12,13 @@ def __init__(self, cursor, data_source):
self.rows = None

def get_rows(self) -> Tuple[Tuple]:
# This might be dangerous if a big number of rows is fetched, consider cleaning up the memory when this object is not needed any more.
# Fetch the default number of failed rows
# TODO: respect the limit set in the config
if not self.rows:
self.rows = self.cursor.fetchall()
try:
self.rows = self.cursor.fetchmany(DEFAULT_FAILED_ROWS_SAMPLE_LIMIT)
except:
self.rows = self.cursor.fetchall()

return self.rows

Expand Down
Loading