Skip to content

Commit

Permalink
fix: postgres errors
Browse files Browse the repository at this point in the history
Having tried to implement meilisync on a postgres database I encountered some errors

- `meilisync check` did not work because `RealDictCursor` was used as `cursor_factory` and `get_count` used integer indexing, which  `RealDictCursor` does not support

    - I have made my own custom Row and Cursor types that support key indexing and integer indexing

- When deleting something from the database `columntypes` is not included in the payload. `.get` is used to fetch `columntypes` where the length of it is then computed, but `.get` is used without a default, so when it is not included, it would call `len` of `None` which gives an error

    - Added an empty list as default for the various `column` field in the payload

- I am not sure if this is an error, but in the way I want to use meilisync it is. When setting fields for an index in the `config.yml` file, it updates the documents of the index with all the fields for the affected row. This makes search results inconsistent, and could potentially reveal information to the user it should not have.

    - Made it such that when giving fields in the config, it only updates the documents with those given fields. If no fields are given it will include all fields.
  • Loading branch information
lasseintree committed Jan 30, 2024
1 parent 5f6c2e0 commit 9d0a1c4
Show file tree
Hide file tree
Showing 2 changed files with 22 additions and 5 deletions.
2 changes: 1 addition & 1 deletion meilisync/schemas.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,6 @@ def mapping_data(self, fields_mapping: Optional[dict] = None):
if fields_mapping is not None and k in fields_mapping:
real_k = fields_mapping[k] or k
data[real_k] = v
else:
elif fields_mapping is None:
data[k] = v
return data or self.data
25 changes: 21 additions & 4 deletions meilisync/source/postgres.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,23 @@
from meilisync.source import Source


class CustomDictRow(psycopg2.extras.RealDictRow):
def __getitem__(self, key):
try:
return super().__getitem__(key)
except KeyError as exc:
if isinstance(key, int):
return super().__getitem__(list(self.keys())[key])
raise exc


class CustomDictCursor(psycopg2.extras.RealDictCursor):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
kwargs["row_factory"] = CustomDictRow
self.row_factory = CustomDictRow


class Postgres(Source):
type = SourceType.postgres
slot = "meilisync"
Expand All @@ -34,7 +51,7 @@ def __init__(
self.cursor.execute("SELECT pg_current_wal_lsn()")
self.start_lsn = self.cursor.fetchone()[0]
self.conn_dict = psycopg2.connect(
**self.kwargs, cursor_factory=psycopg2.extras.RealDictCursor
**self.kwargs, cursor_factory=CustomDictCursor
)

async def get_current_progress(self):
Expand Down Expand Up @@ -81,9 +98,9 @@ def _consumer(self, msg: ReplicationMessage):
table = change.get("table")
if table not in self.tables:
return
columnnames = change.get("columnnames")
columnvalues = change.get("columnvalues")
columntypes = change.get("columntypes")
columnnames = change.get("columnnames", [])
columnvalues = change.get("columnvalues", [])
columntypes = change.get("columntypes", [])

for i in range(len(columntypes)):
if columntypes[i] == "json":
Expand Down

0 comments on commit 9d0a1c4

Please sign in to comment.