Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setting filter_schemas not working as intended during catalog discovery #226

Closed
ShahBinoy opened this issue Sep 7, 2023 · 5 comments
Closed

Setting filter_schemas not working as intended during catalog discovery #226

ShahBinoy opened this issue Sep 7, 2023 · 5 comments

Comments

@ShahBinoy
Copy link

I am connecting to a 3rd Party database where i only have connectivity permissions for 1 schema on Postgres DB. But after using version 0.0.3 the issue still persists.

My original issue started when I experienced the error #215. But with the fix in 0.0.3 the issue still persists and I get error deep in catalog discovery phase.

This is my meltano.yml section for extractor pdr. We have permissions to only this schema

- name: pdr
    inherit_from: tap-postgres
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git
    config:
      filter_schemas:
       - 'pdr'
    select:
      - "pdr-mv_participant_all.participant_id"
      - "pdr-mv_participant_all.participant_origin"
      - "pdr-mv_participant_all.is_ghost_id"
      - "pdr-mv_participant_all.withdrawal_status"
      - "pdr-mv_participant_all.suspension_status"
      - "pdr-mv_participant_all.deceased_status"
      - "pdr-mv_participant_all.deceased_status_date"
      - "pdr-mv_participant_all.test_participant"
      - "pdr-mv_participant_all.addr_state"
      - "pdr-mv_participant_all.hpo"
      - "pdr-mv_participant_all.hpo_id"
      - "pdr-mv_participant_all.organization_id"
      - "pdr-mv_participant_all.site_id"
      - "pdr-mv_participant_all.enrollment_status"
      - "pdr-mv_participant_all.withdrawal_status"
      - "pdr-mv_participant_all.withdrawal_date"
      - "pdr-mv_participant_all.suspension_time"
      - "pdr-mv_participant_all.sign_up_time"
      - "pdr-mv_participant_all.enrollment_member"
      - "pdr-mv_participant_all.enrollment_core_stored"
      - "pdr-mv_participant_all.enrollment_core_ordered"
      - "pdr-mv_participant_all.ehr_status"
      - "pdr-mv_participant_all.ehr_receipt"
      - "pdr-mv_participant_all.ehr_update"
      - "pdr-mv_participant_all.biobank_id"
      - "pdr-mv_participant_all.ubr_overall"
      - "pdr-mv_participant_all.ubr_ethnicity"
      - "pdr-mv_participant_all.ubr_age_at_consent"
      - "pdr-mv_participant_all.ubr_sex"
      - "pdr-mv_participant_all.ubr_sexual_gender_minority"
      - "pdr-mv_participant_all.ubr_income"
      - "pdr-mv_participant_all.ubr_education"
      - "pdr-mv_participant_all.ubr_geography"
      - "pdr-mv_participant_all.ubr_disability"
    metadata:
      'pdr-mv_participant_all':
        replication-method: INCREMENTAL
        replication-key: modified
        view-key-properties: [ id ]

This is the error I get when trying to connect to remote server where I only have permissions to check on one schema pdr. It is still trying to look for something in pdr_ops schema during catalog discovery.

{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/run/elt/pdr-data/e00d718e-b4c2-484f-887e-90fe7345d3be/tap.c9a805cd-1c12-4a54-a812-004b1acfdd6c.config.json', '--state', '/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/run/elt/pdr-data/e00d718e-b4c2-484f-887e-90fe7345d3be/state.json', '--discover'] returned 1 with stderr:", "level": "info", "timestamp": "2023-09-06T18:57:28.782135Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": " Traceback (most recent call last):", "level": "info", "timestamp": "2023-09-06T18:57:28.784226Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1910, in _execute_context", "level": "info", "timestamp": "2023-09-06T18:57:28.784561Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    self.dialect.do_execute(", "level": "info", "timestamp": "2023-09-06T18:57:28.784782Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py\", line 736, in do_execute", "level": "info", "timestamp": "2023-09-06T18:57:28.784969Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    cursor.execute(statement, parameters)", "level": "info", "timestamp": "2023-09-06T18:57:28.785963Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "psycopg2.errors.InsufficientPrivilege: permission denied for schema pdr_ops", "level": "info", "timestamp": "2023-09-06T18:57:28.786479Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "", "level": "info", "timestamp": "2023-09-06T18:57:28.786688Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "", "level": "info", "timestamp": "2023-09-06T18:57:28.786877Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "The above exception was the direct cause of the following exception:", "level": "info", "timestamp": "2023-09-06T18:57:28.787059Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "", "level": "info", "timestamp": "2023-09-06T18:57:28.787282Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "Traceback (most recent call last):", "level": "info", "timestamp": "2023-09-06T18:57:28.787498Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres\", line 8, in <module>", "level": "info", "timestamp": "2023-09-06T18:57:28.787689Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    sys.exit(TapPostgres.cli())", "level": "info", "timestamp": "2023-09-06T18:57:28.787874Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 1157, in __call__", "level": "info", "timestamp": "2023-09-06T18:57:28.788055Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    return self.main(*args, **kwargs)", "level": "info", "timestamp": "2023-09-06T18:57:28.788233Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 1077, in main", "level": "info", "timestamp": "2023-09-06T18:57:28.788434Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    with self.make_context(prog_name, args, **extra) as ctx:", "level": "info", "timestamp": "2023-09-06T18:57:28.788622Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 943, in make_context", "level": "info", "timestamp": "2023-09-06T18:57:28.788787Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    self.parse_args(ctx, args)", "level": "info", "timestamp": "2023-09-06T18:57:28.789124Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 1408, in parse_args", "level": "info", "timestamp": "2023-09-06T18:57:28.789326Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    value, args = param.handle_parse_result(ctx, opts, args)", "level": "info", "timestamp": "2023-09-06T18:57:28.789507Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 2400, in handle_parse_result", "level": "info", "timestamp": "2023-09-06T18:57:28.789678Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    value = self.process_value(ctx, value)", "level": "info", "timestamp": "2023-09-06T18:57:28.789848Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 2362, in process_value", "level": "info", "timestamp": "2023-09-06T18:57:28.790013Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    value = self.callback(ctx, self, value)", "level": "info", "timestamp": "2023-09-06T18:57:28.790178Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py\", line 527, in cb_discover", "level": "info", "timestamp": "2023-09-06T18:57:28.790340Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    tap.run_discovery()", "level": "info", "timestamp": "2023-09-06T18:57:28.790502Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py\", line 287, in run_discovery", "level": "info", "timestamp": "2023-09-06T18:57:28.790664Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    catalog_text = self.catalog_json_text", "level": "info", "timestamp": "2023-09-06T18:57:28.790885Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py\", line 307, in catalog_json_text", "level": "info", "timestamp": "2023-09-06T18:57:28.791118Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    return json.dumps(self.catalog_dict, indent=2)", "level": "info", "timestamp": "2023-09-06T18:57:28.791300Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/tap.py\", line 466, in catalog_dict", "level": "info", "timestamp": "2023-09-06T18:57:28.791536Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    result[\"streams\"].extend(self.connector.discover_catalog_entries())", "level": "info", "timestamp": "2023-09-06T18:57:28.791732Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/connectors/sql.py\", line 506, in discover_catalog_entries", "level": "info", "timestamp": "2023-09-06T18:57:28.791907Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    catalog_entry = self.discover_catalog_entry(", "level": "info", "timestamp": "2023-09-06T18:57:28.792075Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/connectors/sql.py\", line 444, in discover_catalog_entry", "level": "info", "timestamp": "2023-09-06T18:57:28.792240Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    for column_def in inspected.get_columns(table_name, schema=schema_name):", "level": "info", "timestamp": "2023-09-06T18:57:28.792406Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py\", line 497, in get_columns", "level": "info", "timestamp": "2023-09-06T18:57:28.792569Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    col_defs = self.dialect.get_columns(", "level": "info", "timestamp": "2023-09-06T18:57:28.792733Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"<string>\", line 2, in get_columns", "level": "info", "timestamp": "2023-09-06T18:57:28.792899Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py\", line 55, in cache", "level": "info", "timestamp": "2023-09-06T18:57:28.793061Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    ret = fn(self, con, *args, **kw)", "level": "info", "timestamp": "2023-09-06T18:57:28.793225Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py\", line 3918, in get_columns", "level": "info", "timestamp": "2023-09-06T18:57:28.793387Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    c = connection.execute(s, dict(table_oid=table_oid))", "level": "info", "timestamp": "2023-09-06T18:57:28.793552Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1385, in execute", "level": "info", "timestamp": "2023-09-06T18:57:28.793716Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)", "level": "info", "timestamp": "2023-09-06T18:57:28.793879Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/sql/elements.py\", line 334, in _execute_on_connection", "level": "info", "timestamp": "2023-09-06T18:57:28.794042Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    return connection._execute_clauseelement(", "level": "info", "timestamp": "2023-09-06T18:57:28.794220Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1577, in _execute_clauseelement", "level": "info", "timestamp": "2023-09-06T18:57:28.794386Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    ret = self._execute_context(", "level": "info", "timestamp": "2023-09-06T18:57:28.794553Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1953, in _execute_context", "level": "info", "timestamp": "2023-09-06T18:57:28.794717Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    self._handle_dbapi_exception(", "level": "info", "timestamp": "2023-09-06T18:57:28.794879Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 2134, in _handle_dbapi_exception", "level": "info", "timestamp": "2023-09-06T18:57:28.795044Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    util.raise_(", "level": "info", "timestamp": "2023-09-06T18:57:28.795207Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py\", line 211, in raise_", "level": "info", "timestamp": "2023-09-06T18:57:28.795368Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    raise exception", "level": "info", "timestamp": "2023-09-06T18:57:28.795532Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1910, in _execute_context", "level": "info", "timestamp": "2023-09-06T18:57:28.795693Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    self.dialect.do_execute(", "level": "info", "timestamp": "2023-09-06T18:57:28.795854Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/aniketshirsat/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py\", line 736, in do_execute", "level": "info", "timestamp": "2023-09-06T18:57:28.796019Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    cursor.execute(statement, parameters)", "level": "info", "timestamp": "2023-09-06T18:57:28.796191Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InsufficientPrivilege) permission denied for schema pdr_ops", "level": "info", "timestamp": "2023-09-06T18:57:28.796352Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "", "level": "info", "timestamp": "2023-09-06T18:57:28.796514Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "[SQL:", "level": "info", "timestamp": "2023-09-06T18:57:28.796676Z"}
@visch
Copy link
Member

visch commented Sep 7, 2023

@ShahBinoy Can you double check you have the latest version installed ie meltano install tap-postgres --clean we haven't pushed a new version number out to pypi (since you're not pointing to pypi this doesn't matter actually I believe you just have to run the clean command everytime as the version is set to 0.0.0 on this repo itself)

@ShahBinoy
Copy link
Author

@visch We had verified the installation with github SHA which matched with latest, I will verify after running clean command too

Meanwhile attaching full log

(venv) $ meltano % meltano --log-level=debug elt --transform=skip --state-id=pdr-data --select=pdr-mv_participant_all pdr data-warehouse-pdr --force

{"event": "Environment 'dev' is active", "level": "info", "timestamp": "2023-09-06T18:44:12.761729Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "event": "Running extract & load...", "level": "info", "timestamp": "2023-09-06T18:44:40.494477Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/run/elt/pdr-data/e00d718e-b4c2-484f-887e-90fe7345d3be/tap.c9a805cd-1c12-4a54-a812-004b1acfdd6c.config.json', '--state', '/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/run/elt/pdr-data/e00d718e-b4c2-484f-887e-90fe7345d3be/state.json', '--discover'] returned 1 with stderr:", "level": "info", "timestamp": "2023-09-06T18:57:28.782135Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": " Traceback (most recent call last):", "level": "info", "timestamp": "2023-09-06T18:57:28.784226Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1910, in _execute_context", "level": "info", "timestamp": "2023-09-06T18:57:28.784561Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    self.dialect.do_execute(", "level": "info", "timestamp": "2023-09-06T18:57:28.784782Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py\", line 736, in do_execute", "level": "info", "timestamp": "2023-09-06T18:57:28.784969Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    cursor.execute(statement, parameters)", "level": "info", "timestamp": "2023-09-06T18:57:28.785963Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "psycopg2.errors.InsufficientPrivilege: permission denied for schema pdr_ops", "level": "info", "timestamp": "2023-09-06T18:57:28.786479Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "", "level": "info", "timestamp": "2023-09-06T18:57:28.786688Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "", "level": "info", "timestamp": "2023-09-06T18:57:28.786877Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "The above exception was the direct cause of the following exception:", "level": "info", "timestamp": "2023-09-06T18:57:28.787059Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "", "level": "info", "timestamp": "2023-09-06T18:57:28.787282Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "Traceback (most recent call last):", "level": "info", "timestamp": "2023-09-06T18:57:28.787498Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres\", line 8, in <module>", "level": "info", "timestamp": "2023-09-06T18:57:28.787689Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    sys.exit(TapPostgres.cli())", "level": "info", "timestamp": "2023-09-06T18:57:28.787874Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 1157, in __call__", "level": "info", "timestamp": "2023-09-06T18:57:28.788055Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    return self.main(*args, **kwargs)", "level": "info", "timestamp": "2023-09-06T18:57:28.788233Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 1077, in main", "level": "info", "timestamp": "2023-09-06T18:57:28.788434Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    with self.make_context(prog_name, args, **extra) as ctx:", "level": "info", "timestamp": "2023-09-06T18:57:28.788622Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 943, in make_context", "level": "info", "timestamp": "2023-09-06T18:57:28.788787Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    self.parse_args(ctx, args)", "level": "info", "timestamp": "2023-09-06T18:57:28.789124Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 1408, in parse_args", "level": "info", "timestamp": "2023-09-06T18:57:28.789326Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    value, args = param.handle_parse_result(ctx, opts, args)", "level": "info", "timestamp": "2023-09-06T18:57:28.789507Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 2400, in handle_parse_result", "level": "info", "timestamp": "2023-09-06T18:57:28.789678Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    value = self.process_value(ctx, value)", "level": "info", "timestamp": "2023-09-06T18:57:28.789848Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py\", line 2362, in process_value", "level": "info", "timestamp": "2023-09-06T18:57:28.790013Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    value = self.callback(ctx, self, value)", "level": "info", "timestamp": "2023-09-06T18:57:28.790178Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py\", line 527, in cb_discover", "level": "info", "timestamp": "2023-09-06T18:57:28.790340Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    tap.run_discovery()", "level": "info", "timestamp": "2023-09-06T18:57:28.790502Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py\", line 287, in run_discovery", "level": "info", "timestamp": "2023-09-06T18:57:28.790664Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    catalog_text = self.catalog_json_text", "level": "info", "timestamp": "2023-09-06T18:57:28.790885Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py\", line 307, in catalog_json_text", "level": "info", "timestamp": "2023-09-06T18:57:28.791118Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    return json.dumps(self.catalog_dict, indent=2)", "level": "info", "timestamp": "2023-09-06T18:57:28.791300Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/tap.py\", line 466, in catalog_dict", "level": "info", "timestamp": "2023-09-06T18:57:28.791536Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    result[\"streams\"].extend(self.connector.discover_catalog_entries())", "level": "info", "timestamp": "2023-09-06T18:57:28.791732Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/connectors/sql.py\", line 506, in discover_catalog_entries", "level": "info", "timestamp": "2023-09-06T18:57:28.791907Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    catalog_entry = self.discover_catalog_entry(", "level": "info", "timestamp": "2023-09-06T18:57:28.792075Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/connectors/sql.py\", line 444, in discover_catalog_entry", "level": "info", "timestamp": "2023-09-06T18:57:28.792240Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    for column_def in inspected.get_columns(table_name, schema=schema_name):", "level": "info", "timestamp": "2023-09-06T18:57:28.792406Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py\", line 497, in get_columns", "level": "info", "timestamp": "2023-09-06T18:57:28.792569Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    col_defs = self.dialect.get_columns(", "level": "info", "timestamp": "2023-09-06T18:57:28.792733Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"<string>\", line 2, in get_columns", "level": "info", "timestamp": "2023-09-06T18:57:28.792899Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py\", line 55, in cache", "level": "info", "timestamp": "2023-09-06T18:57:28.793061Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    ret = fn(self, con, *args, **kw)", "level": "info", "timestamp": "2023-09-06T18:57:28.793225Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py\", line 3918, in get_columns", "level": "info", "timestamp": "2023-09-06T18:57:28.793387Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    c = connection.execute(s, dict(table_oid=table_oid))", "level": "info", "timestamp": "2023-09-06T18:57:28.793552Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1385, in execute", "level": "info", "timestamp": "2023-09-06T18:57:28.793716Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)", "level": "info", "timestamp": "2023-09-06T18:57:28.793879Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/sql/elements.py\", line 334, in _execute_on_connection", "level": "info", "timestamp": "2023-09-06T18:57:28.794042Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    return connection._execute_clauseelement(", "level": "info", "timestamp": "2023-09-06T18:57:28.794220Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1577, in _execute_clauseelement", "level": "info", "timestamp": "2023-09-06T18:57:28.794386Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    ret = self._execute_context(", "level": "info", "timestamp": "2023-09-06T18:57:28.794553Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1953, in _execute_context", "level": "info", "timestamp": "2023-09-06T18:57:28.794717Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    self._handle_dbapi_exception(", "level": "info", "timestamp": "2023-09-06T18:57:28.794879Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 2134, in _handle_dbapi_exception", "level": "info", "timestamp": "2023-09-06T18:57:28.795044Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    util.raise_(", "level": "info", "timestamp": "2023-09-06T18:57:28.795207Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py\", line 211, in raise_", "level": "info", "timestamp": "2023-09-06T18:57:28.795368Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    raise exception", "level": "info", "timestamp": "2023-09-06T18:57:28.795532Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py\", line 1910, in _execute_context", "level": "info", "timestamp": "2023-09-06T18:57:28.795693Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    self.dialect.do_execute(", "level": "info", "timestamp": "2023-09-06T18:57:28.795854Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "  File \"/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py\", line 736, in do_execute", "level": "info", "timestamp": "2023-09-06T18:57:28.796019Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "    cursor.execute(statement, parameters)", "level": "info", "timestamp": "2023-09-06T18:57:28.796191Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InsufficientPrivilege) permission denied for schema pdr_ops", "level": "info", "timestamp": "2023-09-06T18:57:28.796352Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "", "level": "info", "timestamp": "2023-09-06T18:57:28.796514Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "[SQL:", "level": "info", "timestamp": "2023-09-06T18:57:28.796676Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "            SELECT a.attname,", "level": "info", "timestamp": "2023-09-06T18:57:28.796853Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "              pg_catalog.format_type(a.atttypid, a.atttypmod),", "level": "info", "timestamp": "2023-09-06T18:57:28.797017Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "              (", "level": "info", "timestamp": "2023-09-06T18:57:28.797179Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                SELECT pg_catalog.pg_get_expr(d.adbin, d.adrelid)", "level": "info", "timestamp": "2023-09-06T18:57:28.797340Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                FROM pg_catalog.pg_attrdef d", "level": "info", "timestamp": "2023-09-06T18:57:28.797502Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum", "level": "info", "timestamp": "2023-09-06T18:57:28.797662Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                AND a.atthasdef", "level": "info", "timestamp": "2023-09-06T18:57:28.797823Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "              ) AS DEFAULT,", "level": "info", "timestamp": "2023-09-06T18:57:28.797983Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "              a.attnotnull,", "level": "info", "timestamp": "2023-09-06T18:57:28.798151Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "              a.attrelid as table_oid,", "level": "info", "timestamp": "2023-09-06T18:57:28.798314Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "              pgd.description as comment,", "level": "info", "timestamp": "2023-09-06T18:57:28.798474Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "              a.attgenerated as generated,", "level": "info", "timestamp": "2023-09-06T18:57:28.798635Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                              (SELECT json_build_object(", "level": "info", "timestamp": "2023-09-06T18:57:28.798796Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                    'always', a.attidentity = 'a',", "level": "info", "timestamp": "2023-09-06T18:57:28.798956Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                    'start', s.seqstart,", "level": "info", "timestamp": "2023-09-06T18:57:28.799116Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                    'increment', s.seqincrement,", "level": "info", "timestamp": "2023-09-06T18:57:28.799282Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                    'minvalue', s.seqmin,", "level": "info", "timestamp": "2023-09-06T18:57:28.799445Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                    'maxvalue', s.seqmax,", "level": "info", "timestamp": "2023-09-06T18:57:28.799605Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                    'cache', s.seqcache,", "level": "info", "timestamp": "2023-09-06T18:57:28.799767Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                    'cycle', s.seqcycle)", "level": "info", "timestamp": "2023-09-06T18:57:28.799927Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                FROM pg_catalog.pg_sequence s", "level": "info", "timestamp": "2023-09-06T18:57:28.800086Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                JOIN pg_catalog.pg_class c on s.seqrelid = c.\"oid\"", "level": "info", "timestamp": "2023-09-06T18:57:28.800252Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                WHERE c.relkind = 'S'", "level": "info", "timestamp": "2023-09-06T18:57:28.800429Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                AND a.attidentity != ''", "level": "info", "timestamp": "2023-09-06T18:57:28.800592Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                AND s.seqrelid = pg_catalog.pg_get_serial_sequence(", "level": "info", "timestamp": "2023-09-06T18:57:28.800753Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                    a.attrelid::regclass::text, a.attname", "level": "info", "timestamp": "2023-09-06T18:57:28.800914Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                )::regclass::oid", "level": "info", "timestamp": "2023-09-06T18:57:28.801076Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                ) as identity_options", "level": "info", "timestamp": "2023-09-06T18:57:28.801241Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "            FROM pg_catalog.pg_attribute a", "level": "info", "timestamp": "2023-09-06T18:57:28.801404Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "            LEFT JOIN pg_catalog.pg_description pgd ON (", "level": "info", "timestamp": "2023-09-06T18:57:28.801565Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "                pgd.objoid = a.attrelid AND pgd.objsubid = a.attnum)", "level": "info", "timestamp": "2023-09-06T18:57:28.801726Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "            WHERE a.attrelid = %(table_oid)s", "level": "info", "timestamp": "2023-09-06T18:57:28.801888Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "            AND a.attnum > 0 AND NOT a.attisdropped", "level": "info", "timestamp": "2023-09-06T18:57:28.802062Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "            ORDER BY a.attnum", "level": "info", "timestamp": "2023-09-06T18:57:28.802389Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "        ]", "level": "info", "timestamp": "2023-09-06T18:57:28.802592Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "[parameters: {'table_oid': 68924}]", "level": "info", "timestamp": "2023-09-06T18:57:28.802768Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "(Background on this error at: https://sqlalche.me/e/14/f405)", "level": "info", "timestamp": "2023-09-06T18:57:28.802936Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": ".", "level": "info", "timestamp": "2023-09-06T18:57:28.803103Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag.", "level": "info", "timestamp": "2023-09-06T18:57:28.803264Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "Note that you can also check the generated log file at '/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/logs/elt/pdr-data/e00d718e-b4c2-484f-887e-90fe7345d3be/elt.log'.", "level": "info", "timestamp": "2023-09-06T18:57:28.803429Z"}
{"name": "meltano", "run_id": "e00d718e-b4c2-484f-887e-90fe7345d3be", "state_id": "pdr-data", "stdio": "stderr", "cmd_type": "elt", "event": "For more information on debugging and logging: https://docs.meltano.com/reference/command-line-interface#debugging", "level": "info", "timestamp": "2023-09-06T18:57:28.803607Z"}
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/run/elt/pdr-data/e00d718e-b4c2-484f-887e-90fe7345d3be/tap.c9a805cd-1c12-4a54-a812-004b1acfdd6c.config.json', '--state', '/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/run/elt/pdr-data/e00d718e-b4c2-484f-887e-90fe7345d3be/state.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.InsufficientPrivilege: permission denied for schema pdr_ops


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 527, in cb_discover
    tap.run_discovery()
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 287, in run_discovery
    catalog_text = self.catalog_json_text
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/tap_base.py", line 307, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/tap_postgres/tap.py", line 466, in catalog_dict
    result["streams"].extend(self.connector.discover_catalog_entries())
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/connectors/sql.py", line 506, in discover_catalog_entries
    catalog_entry = self.discover_catalog_entry(
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/singer_sdk/connectors/sql.py", line 444, in discover_catalog_entry
    for column_def in inspected.get_columns(table_name, schema=schema_name):
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 497, in get_columns
    col_defs = self.dialect.get_columns(
  File "<string>", line 2, in get_columns
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 55, in cache
    ret = fn(self, con, *args, **kw)
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py", line 3918, in get_columns
    c = connection.execute(s, dict(table_oid=table_oid))
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1385, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/extractors/tap-postgres/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InsufficientPrivilege) permission denied for schema pdr_ops

[SQL: 
            SELECT a.attname,
              pg_catalog.format_type(a.atttypid, a.atttypmod),
              (
                SELECT pg_catalog.pg_get_expr(d.adbin, d.adrelid)
                FROM pg_catalog.pg_attrdef d
                WHERE d.adrelid = a.attrelid AND d.adnum = a.attnum
                AND a.atthasdef
              ) AS DEFAULT,
              a.attnotnull,
              a.attrelid as table_oid,
              pgd.description as comment,
              a.attgenerated as generated,
                              (SELECT json_build_object(
                    'always', a.attidentity = 'a',
                    'start', s.seqstart,
                    'increment', s.seqincrement,
                    'minvalue', s.seqmin,
                    'maxvalue', s.seqmax,
                    'cache', s.seqcache,
                    'cycle', s.seqcycle)
                FROM pg_catalog.pg_sequence s
                JOIN pg_catalog.pg_class c on s.seqrelid = c."oid"
                WHERE c.relkind = 'S'
                AND a.attidentity != ''
                AND s.seqrelid = pg_catalog.pg_get_serial_sequence(
                    a.attrelid::regclass::text, a.attname
                )::regclass::oid
                ) as identity_options                
            FROM pg_catalog.pg_attribute a
            LEFT JOIN pg_catalog.pg_description pgd ON (
                pgd.objoid = a.attrelid AND pgd.objsubid = a.attnum)
            WHERE a.attrelid = %(table_oid)s
            AND a.attnum > 0 AND NOT a.attisdropped
            ORDER BY a.attnum
        ]
[parameters: {'table_oid': 68924}]
(Background on this error at: https://sqlalche.me/e/14/f405)
.
For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag.
Note that you can also check the generated log file at '/Users/johndoe/Documents/code/data-lake-workflows/src/meltano/.meltano/logs/elt/pdr-data/e00d718e-b4c2-484f-887e-90fe7345d3be/elt.log'.
For more information on debugging and logging: https://docs.meltano.com/reference/command-line-interface#debugging

@visch
Copy link
Member

visch commented Sep 7, 2023

@ShahBinoy Thanks for putting this in, the error is clear. It's something to do with https://github.com/meltano/sdk/blob/main/singer_sdk/connectors/sql.py#L499 not filtering / querying the way we were expecting if you still double check that will be helpful as this looks fairly confusing

At a minimum we should add a test that creates a user with limited privilege's and runs the tap limited to one schema

@visch
Copy link
Member

visch commented Sep 7, 2023

@ShahBinoy just wanted to verify you ran clean and were on the latest version!

@ShahBinoy
Copy link
Author

Hi @visch I think asking for --clean was a good call. We did in local and clean made it work. Secondly Kubernetes/docker deployments should also utilize --clean because of docker filesystem layers which can be cached.

Closing the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
MeltanoLabs Overview
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ShahBinoy @visch