How to troubleshoot missing column value for Logical rep.

[halilduygulu]

Hi,
I am using logical replication from postgres to redshift. For one table's text type column which contains string dump of json, we are sometimes getting NULL value instead of long string json dump.
I was wondering if you have any suggestions to help me identify cause of this issue.
for more context,

• this is a text not null in postgres 13,
• I am using pipelinewise-tap-postgres (1.8.3) with pipelinewise in CDC mode.
• I have another text notnull column in same table and this problem does not occur for that.
• I checked csv file generated by target-redshift, column value is NULL in file too, so it is not a copy command ignore thing
• FastSync full load is getting correct data, I reload table to fix all NULLs and now CDC replication inserting nulls again
• I was using Transform to set some other columns to NULL but I disabled that to see if that is the bug, no effect.

[halilduygulu]

oh, after posting this I dig into wal2json closed issues, can my case also be TOAST not being in update event? I see many columns of this table is in extended storage, but not all of them comes as null though.
https://wiki.postgresql.org/wiki/TOAST

eulerto/wal2json#205
eulerto/wal2json#223
eulerto/wal2json#196

[halilduygulu]

Just confirmed by following query to see TOASTed column is not in change data column list sometimes.
I will leave this question here for someone hopefully find it on google search.

SELECT *
FROM pg_logical_slot_peek_changes('pipelinewise_xxx_postgres_db_logical', NULL, NULL, 'pretty-print', '1', 'add-msg-prefixes', 'wal2json')
where data ilike '%_your_table_name_%'