Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Relations public.Withdrawal and public.Epoch does not exist #897

Open
2 of 6 tasks
mkocur-ledger opened this issue Sep 19, 2024 · 14 comments
Open
2 of 6 tasks

Relations public.Withdrawal and public.Epoch does not exist #897

mkocur-ledger opened this issue Sep 19, 2024 · 14 comments
Labels
BUG

Comments

@mkocur-ledger
Copy link

mkocur-ledger commented Sep 19, 2024 •
edited
Loading

Summary

Hello!

I have restored a snapshot with db-sync:
(https://update-cardano-mainnet.iohk.io/cardano-db-sync/13.5/db-sync-snapshot-schema-13.5-block-10823207-x86_64.tgz)

[db-sync-node:Info:83] [2024-09-19 00:51:47.84 UTC] Reached EpochNo 508
[db-sync-node:Info:92] [2024-09-19 00:56:57.60 UTC] Asynchronously wrote a ledger snapshot to /config/state/134092758-670ca68c3d-507.lstate in 323.408766108s.
[db-sync-node:Info:83] [2024-09-19 01:02:12.81 UTC] Removing old epoch boundary file /config/state/130636777-c1ecf5af89-499.lstate
[db-sync-node:Info:83] [2024-09-19 01:02:28.43 UTC] Reached EpochNo 509
[db-sync-node:Info:83] [2024-09-19 01:02:30.04 UTC] Received block which is not in the db with HeaderFields {headerFieldSlot = SlotNo 134525754, headerFieldBlockNo = BlockNo 10823208, headerFieldHash = 1fbfe5fd0f3e99d6e1f4ccffd8ad7202e36e57c18cf8863bf6aec57eb1062db9}. Time to restore consistency.
[db-sync-node:Info:83] [2024-09-19 01:02:30.05 UTC] Starting at epoch 509
[db-sync-node:Info:83] [2024-09-19 01:02:32.80 UTC] Insert Conway Block: epoch 509, slot 134525754, block 10823208, hash 1fbfe5fd0f3e99d6e1f4ccffd8ad7202e36e57c18cf8863bf6aec57eb1062db9
[db-sync-node:Info:83] [2024-09-19 01:02:32.85 UTC] Setting ConsistencyLevel to Consistent
[db-sync-node:Info:92] [2024-09-19 01:05:35.24 UTC] Asynchronously wrote a ledger snapshot to /config/state/134524753-29011cc132-508.lstate in 202.425990791s.
[db-sync-node:Info:83] [2024-09-19 01:08:29.04 UTC] Insert Conway Block: epoch 509, slot 134563412, block 10825000, hash e6f71ef9d62932dd429b61051e63fd97929b605f9c33612817cc5205eb6b4ca2
[db-sync-node:Info:83] [2024-09-19 01:23:30.51 UTC] Inserted 1335280 EpochStake for EpochNo 510
[db-sync-node:Info:83] [2024-09-19 01:31:00.87 UTC] Insert Conway Block: epoch 509, slot 134664823, block 10830000, hash ec59c8053d5f30299ebbfb272083ead1e968f4aab60ab868203a9c8b93cd7201
[db-sync-node:Info:83] [2024-09-19 01:31:00.87 UTC] Voting Anchor Offchain metadata fetch: 0 results, 1000 fetch errors
[db-sync-node:Info:83] [2024-09-19 01:31:01.69 UTC] Pool Offchain metadata fetch: 0 results, 1000 fetch errors
[db-sync-node:Info:83] [2024-09-19 01:41:25.76 UTC] Insert Conway Block: epoch 509, slot 134767193, block 10835000, hash 11cace016758192b7459071aade17f575b305ebf374f6970c78f4ae936bead5e
[db-sync-node:Warning:83] [2024-09-19 01:47:59.59 UTC] prepareTxMetadata: Column 'json' in table 'metadata' was recorded as null, due to a Unicode NUL character found when trying to parse the json.
[db-sync-node:Warning:83] [2024-09-19 01:47:59.79 UTC] prepareTxMetadata: Column 'json' in table 'metadata' was recorded as null, due to a Unicode NUL character found when trying to parse the json.
[db-sync-node:Info:83] [2024-09-19 01:49:47.05 UTC] Insert Conway Block: epoch 509, slot 134867805, block 10840000, hash 4a858262fe88e24fabffd4f7e2b4ef6b51699cbd3aea350e63f748f98f78d175

hasura/graphql-engine:v2.43.0.cli-migrations-v2 is reporting public.Withdrawal and public.Epoch relations missing:

{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.10.97","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database q
uery error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \
"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_
to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"
Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_
execution_time":3.047279e-3,"request_id":"fc74d816-c063-47d9-ae71-cf1ac17e8c2b","request_mode":"error","request_read_time":6.125e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-has
ura-role":"cardano-graphql"}},"request_id":"fc74d816-c063-47d9-ae71-cf1ac17e8c2b"},"level":"error","span_id":"db6b71dd35db4c48","timestamp":"2024-09-19T06:25:06.376+0000","trace_id":"4753ed8ad279d7b47c8bf
4e4ac8a7194","type":"http-log"}

{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-09-19T06:19:34.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.10.97","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database q
uery error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \
"public.Withdrawal\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  json_build_object('aggregate', json_build_object('sum', json_build_object('amount', sum(\"_root\".\"root.pg
.amount\" ) ) ) ) AS \"root\" FROM  (SELECT  \"_root.base\".\"amount\" AS \"root.pg.amount\" FROM  (SELECT  *  FROM \"public\".\"Withdrawal\"  WHERE ('true')     ) AS \"_root.base\"      ) AS \"_root\"
   "},"path":"$"},"query":{"type":null},"query_execution_time":265.310693138,"request_id":"06d7e070-be6b-4a1a-98a5-5f10d6330822","request_mode":"error","request_read_time":5.408e-6,"response_size":669,"un
compressed_response_size":669,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"06d7e070-be6b-4a1a-98a5-5f10d6330822"},"level":"error","span_id":"4c832a518ef4b722","timestamp":"2024-09-19T06:
25:05.375+0000","trace_id":"fdb208a2ea0746a898ea78d82148a64d","type":"http-log"}

The sync status:

curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardanoDbMeta { initialized syncPercentage }}"}' \
  http://localhost:3100/graphql
{"data":{"cardanoDbMeta":{"initialized":true,"syncPercentage":100}},"extensions":{"tracing":{"version":1,"startTime":"2024-09-19T06:49:12.102Z","endTime":"2024-09-19T06:49:12.115Z","duration":13500570,"execution":{"resolvers":[{"path":["cardanoDbMeta"],"parentType":"Query","fieldName":"cardanoDbMeta","returnType":"CardanoDbMeta!","startOffset":502588,"duration":12826503},{"path":["cardanoDbMeta","initialized"],"parentType":"CardanoDbMeta","fieldName":"initialized","returnType":"Boolean!","startOffset":13396134,"duration":15571},{"path":["cardanoDbMeta","syncPercentage"],"parentType":"CardanoDbMeta","fieldName":"syncPercentage","returnType":"Percentage!","startOffset":13420908,"duration":5172}]}}}}
 curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "{ cardano { tip { number slotNo epoch { number } } } }"}' http://localhost:3100/graphql
{"data":{"cardano":{"tip":{"number":10854265,"slotNo":"135162299","epoch":{"number":510}}}},"extensions":{"tracing":{"version":1,"startTime":"2024-09-19T06:49:55.759Z","endTime":"2024-09-19T06:49:55.779Z","duration":19672084,"execution":{"resolvers":[{"path":["cardano"],"parentType":"Query","fieldName":"cardano","returnType":"Cardano!","startOffset":2899698,"duration":16592105},{"path":["cardano","tip"],"parentType":"Cardano","fieldName":"tip","returnType":"Block!","startOffset":19533689,"duration":15467},{"path":["cardano","tip","number"],"parentType":"Block","fieldName":"number","returnType":"Int","startOffset":19563055,"duration":9800},{"path":["cardano","tip","slotNo"],"parentType":"Block","fieldName":"slotNo","returnType":"String","startOffset":19582450,"duration":3407},{"path":["cardano","tip","epoch"],"parentType":"Block","fieldName":"epoch","returnType":"Epoch","startOffset":19597505,"duration":5842},{"path":["cardano","tip","epoch","number"],"parentType":"Epoch","fieldName":"number","returnType":"Int!","startOffset":19617981,"duration":4217}]}}}}

I have restarted graphql, tmr, hasura, background components after the sync was completed to face the same issue.

db-sync: "13.5.0.2"
node: "9.1.1"
ogmios-server: "6.6.2"
graphql-engine: "v2.43.0.cli-migrations-v2"
graphql-server: "8.2.2"
graphql-background: "8.2.2-mainnet"
cf-token-metadata-registry-api: "1.0.2"
postgres: 15.6

Steps to reproduce the bug

No response

Actual Result

Some db data missing after snapshot restore.

Expected Result

No errors

Environment

Kubernetes
{
os: {
arch: 'x64',
cpus: { count: 48, model: 'Intel(R) Xeon(R) Gold 6312U CPU @ 2.40GHz' },
release: '5.15.0-117-generic',
totalMem: 269877899264,
type: 'Linux'
},
version: '8.0.0'
}

Platform

  • Linux (Ubuntu)
  • Linux (Other)
  • macOS
  • Windows

Platform version

Ubuntu 22.04.4 LTS

Runtime

  • Node.js
  • Docker

Runtime version

No response
@mkocur-ledger mkocur-ledger changed the title Relation public.Withdrawal, public.Epoch and metadata does not exist Relations public.Withdrawal, public.Epoch and metadata does not exist Sep 19, 2024
@mkocur-ledger mkocur-ledger mentioned this issue Sep 20, 2024
Closed
6 tasks
@Kartiiyer12
Copy link
Collaborator

Hi @mkocur-ledger I think the issue for the Epoch and withdrawal table missing could be issue with the hasura migration and not the db-sync-snapshot. Can you try to see if there are errors on the graphql-background & hasura while applying the migrations?

For the token metadata registry server, are you running the default container as it is from the repo? Table missing seems improbable.

@mkocur-ledger mkocur-ledger changed the title Relations public.Withdrawal, public.Epoch and metadata does not exist Relations public.Withdrawal and public.Epoch does not exist Sep 24, 2024
@mkocur-ledger
Copy link
Author

Hi @Kartiiyer12, thanks for your interest in my bugreport.
About metadata, I have edit my report and removed it. It was fixed here: cardano-foundation/cf-token-metadata-registry#19

About the migration, I didn't create any of my own. These are related log messages:

{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_D
ATABASE_URL"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting
to 9691"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"server timeout is not set, defaulting to 30 seconds"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}}
{"timestamp":"2024-09-24T06:38:27.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 30 for 9691 to be ready"}}
{"detail":{"info":"Already at the latest catalog version (48); nothing to do.","kind":"catalog_migrate"},"level":"info","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"detail":{"info":"Schema sync enabled. Polling at Refined (Milliseconds {milliseconds = 1s})","kind":"schema-sync"},"level":"info","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"detail":{"info":{"instance_id":"4d2b28e4-c896-465b-88f0-3120b600e68c","message":"listener thread started","thread_id":"ThreadId 72"},"kind":"schema-sync"},"level":"info","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-09-24T06:38:27.784+0000","type":"startup"}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_MIGRATIONS_DIR is not set, defaulting to /hasura-migrations"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-09-24T06:38:29.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}

@Kartiiyer12
Copy link
Collaborator

These are migration folder that should be applied - https://github.com/cardano-foundation/cardano-graphql/tree/16ed6d71dbe95ff9e7d959ea51a8476da668684c/packages/api-cardano-db-hasura/hasura/project/migrations/default/1589369664961_init.
This happens from the background-graphql side. Are you running everything as it is from the repo or are there any customizations?

@mkocur-ledger
Copy link
Author

I have made a fresh installation and after applying this: https://github.com/cardano-foundation/cf-token-metadata-registry/blob/47f6cf08515f45c3dc11968a7e15a791875cf59c/api/src/main/resources/db/migration/postgresql/V0__metadata_server_db_init.sql

I see no errors. Created an issue that it is not applied automatically:
cardano-foundation/cf-token-metadata-registry#19

About your question, there are customizations, yes, but I can see messages in background pod that it is actually inserting assets and the assets were synced in graphql to 100%, so I assume it works.

@Kammerlo
Copy link
Member

I'll check the issue in the token-registry next week and fix this. Thanks for finding it! And I'm glad it worked for you.
I close this issue since it's probably a topic for the Token Registry. If there is anything else, let us know.

@mkocur-ledger
Copy link
Author

mkocur-ledger commented Oct 2, 2024 •
edited
Loading

Actually @Kammerlo please reopen this. I have disabled INFO messages and I found those errors again (I have applied the sql file you mentioned on my database but it didn't help):

"relation \"public.Epoch\" does not exist"
relation \"public.Withdrawal\" does not exist"
{"error":{"code":"validation-failed","error":"no mutations exist","path":"$"}
{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"}

Full log:

{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":
"warn","timestamp":"2024-10-01T08:14:36.932+0000","type":"startup"}                                                                             {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not
 exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) A
S \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":n
ull},"query_execution_time":59.342454474,"request_id":"858e9742-dc65-4e28-8906-a6259378869e","request_mode":"error","request_read_time":2.06e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"858e9742-dc65-4e28-8906-a625
9378869e"},"level":"error","span_id":"1856f494e02ce310","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"f070f1320c096d4104734fc7e6db5fa0","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Withdrawal\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  json_build_object('aggregate', json_build_object('sum', json_build_obje
ct('amount', sum(\"_root\".\"root.pg.amount\" ) ) ) ) AS \"root\" FROM  (SELECT  \"_root.base\".\"amount\" AS \"root.pg.amount\" FROM  (SELECT  *  FROM \"public\".\"Withdrawal\"  WHERE ('true')     ) AS \"_root.base\"      ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_e
xecution_time":65.628226835,"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766","request_mode":"error","request_read_time":1.54e-6,"response_size":669,"uncompressed_response_size":669,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766"},"l
evel":"error","span_id":"93a4c54f8be46366","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"0a44ad3329a1255d3af62717187eee0a","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not
 exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) A
S \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":n
ull},"query_execution_time":1.759961e-3,"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fedb441c0","request_mode":"error","request_read_time":2.38e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fe
db441c0"},"level":"error","span_id":"415b0c75b73d3f9b","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"7221d21491e389cadda32b981deeddc9","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[
]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DE
SC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.466746e-3,"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67","request_mode":"error","request_read_time":8.1e-7,"r
esponse_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67"},"level":"error","span_id":"f0fc2276bc19e1ad","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"f13c31a79f12fd3a95fea03cca50d56f","
type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[
]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DE
SC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.23195e-3,"request_id":"dc69b318-34f9-4282-becb-0e487414eac3","request_mode":"error","request_read_time":1.01e-6,"r
esponse_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"dc69b318-34f9-4282-becb-0e487414eac3"},"level":"error","span_id":"83934b4c0740d43f","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"d4b99c418de9e6fcca17a14586011e69","
type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"ca
rdano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[
]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.or.adaPots\".\"adaPots\" AS \"adaPots\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"num
ber\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" LEFT OUTER JOIN LATERAL (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.or.adaPots.base\".\"reserves\" AS \"reserves\"       ) AS \"_e\"      ) ) AS \"adaPots\" FROM  (SELECT  *  FROM \"public\".\"ada_pots\"  WHERE ((\"_root
.base\".\"number\") = (\"epoch_no\"))    LIMIT 1 ) AS \"_root.or.adaPots.base\"      ) AS \"_root.or.adaPots\" ON ('true')    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.229809e-3,"request_id":"5f82fa48-a
433-4b54-af2a-ef488244b533","request_mode":"error","request_read_time":1.07e-6,"response_size":1194,"uncompressed_response_size":1194,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"5f82fa48-a433-4b54-af2a-ef488244b533"},"level":"error","span_id":"ca4807fd288b05b5","timest
amp":"2024-10-01T12:55:28.255+0000","trace_id":"33c6aa0f5dda79245bc95b61916ef6e5","type":"http-log"}                                            {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not
 exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) A
S \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":n
ull},"query_execution_time":1.657673e-3,"request_id":"2fc31b05-0f3c-4dd6-9192-60204fbb4c8d","request_mode":"error","request_read_time":1.72e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"2fc31b05-0f3c-4dd6-9192-60204
fbb4c8d"},"level":"error","span_id":"95066c5648e8a6ec","timestamp":"2024-10-01T12:55:32.259+0000","trace_id":"671b908ec3a31d38c24227731b82c72e","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.122","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"no mutations exist","path":"$"},"query":{"type":null},"query_execution_time":1.67017e-
4,"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a","request_mode":"error","request_read_time":1.66637e-4,"response_size":68,"uncompressed_response_size":68,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a"},"level":"error","span_id":"
e2a38977c75f3546","timestamp":"2024-10-01T12:55:33.259+0000","trace_id":"ba50a6997116f6e3be10e397d77dc9d6","type":"http-log"}                   {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.50166e-4,"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55","request_mode":"error","request_read_time":
2.36e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55"},"level":"error","span_id":"e8d20c1563873073","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"2fe4e636804b493c12da50d98
4004e3c","type":"http-log"}                                                                                                                     {"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"
operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.43936e-4,"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95","request_mode":"error","request_read_time":
2.06e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95"},"level":"error","span_id":"5d374fc3c09d8b7a","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"0416d3597fb9ebc592dbaffe7
483147d","type":"http-log"}

@Kammerlo Kammerlo reopened this Oct 2, 2024
@Kammerlo
Copy link
Member

Kammerlo commented Oct 2, 2024

Could you provide logs from hasura service? Because this service should apply automatically the sql files and create these tables.

@Kammerlo
Copy link
Member

Kammerlo commented Oct 2, 2024

@mkocur-ledger I further investigated what could be the problem. I found an error within the sql file, which hindert hasura to finish the migration. You can find the PR here: #898
I'll finish testing today, merge and build a new fix release.

Could you check if this solves the problems for your issue as well?

@mkocur-ledger
Copy link
Author

mkocur-ledger commented Oct 2, 2024 •
edited
Loading

@Kammerlo I have seen this, I was actually supposed to create a PR or bugreport, but didn't find time yet.

Actually, this is not going to help, I have run those SQL queries already against the database (fixing the missing comma).

// full logs as requested:

{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_DATABASE_URL"}}
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting to 9691"}}
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}}
{"timestamp":"2024-10-01T08:14:35.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 60 for 9691 to be ready"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-01T08:14:35.949+0000","type":"startup"}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"port 9691 is ready"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_MIGRATIONS_DIR is not set, defaulting to /hasura-migrations"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-10-01T08:14:36.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-01T08:14:36.932+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-01T08:14:35.949+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-01T08:14:36.932+0000","type":"startup"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":59.342454474,"request_id":"858e9742-dc65-4e28-8906-a6259378869e","request_mode":"error","request_read_time":2.06e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"858e9742-dc65-4e28-8906-a6259378869e"},"level":"error","span_id":"1856f494e02ce310","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"f070f1320c096d4104734fc7e6db5fa0","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Withdrawal\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  json_build_object('aggregate', json_build_object('sum', json_build_object('amount', sum(\"_root\".\"root.pg.amount\" ) ) ) ) AS \"root\" FROM  (SELECT  \"_root.base\".\"amount\" AS \"root.pg.amount\" FROM  (SELECT  *  FROM \"public\".\"Withdrawal\"  WHERE ('true')     ) AS \"_root.base\"      ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":65.628226835,"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766","request_mode":"error","request_read_time":1.54e-6,"response_size":669,"uncompressed_response_size":669,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"ffdf5de9-e978-4c5c-9cb6-c41b2d4dc766"},"level":"error","span_id":"93a4c54f8be46366","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"0a44ad3329a1255d3af62717187eee0a","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.759961e-3,"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fedb441c0","request_mode":"error","request_read_time":2.38e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1ddc327d-36dc-48e7-b3d1-4e3fedb441c0"},"level":"error","span_id":"415b0c75b73d3f9b","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"7221d21491e389cadda32b981deeddc9","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.466746e-3,"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67","request_mode":"error","request_read_time":8.1e-7,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"f6f2695f-647f-4ac8-b8c4-8174d98d7b67"},"level":"error","span_id":"f0fc2276bc19e1ad","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"f13c31a79f12fd3a95fea03cca50d56f","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.23195e-3,"request_id":"dc69b318-34f9-4282-becb-0e487414eac3","request_mode":"error","request_read_time":1.01e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"dc69b318-34f9-4282-becb-0e487414eac3"},"level":"error","span_id":"83934b4c0740d43f","timestamp":"2024-10-01T12:55:29.256+0000","trace_id":"d4b99c418de9e6fcca17a14586011e69","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.or.adaPots\".\"adaPots\" AS \"adaPots\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\" LEFT OUTER JOIN LATERAL (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.or.adaPots.base\".\"reserves\" AS \"reserves\"       ) AS \"_e\"      ) ) AS \"adaPots\" FROM  (SELECT  *  FROM \"public\".\"ada_pots\"  WHERE ((\"_root.base\".\"number\") = (\"epoch_no\"))    LIMIT 1 ) AS \"_root.or.adaPots.base\"      ) AS \"_root.or.adaPots\" ON ('true')    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.229809e-3,"request_id":"5f82fa48-a433-4b54-af2a-ef488244b533","request_mode":"error","request_read_time":1.07e-6,"response_size":1194,"uncompressed_response_size":1194,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"5f82fa48-a433-4b54-af2a-ef488244b533"},"level":"error","span_id":"ca4807fd288b05b5","timestamp":"2024-10-01T12:55:28.255+0000","trace_id":"33c6aa0f5dda79245bc95b61916ef6e5","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"unexpected","error":"database query error","internal":{"arguments":["(Oid 114,Just (\"{\\\"x-hasura-role\\\":\\\"cardano-graphql\\\"}\",Binary))"],"error":{"description":null,"exec_status":"FatalError","hint":null,"message":"relation \"public.Epoch\" does not exist","status_code":"42P01"},"prepared":true,"statement":"SELECT  coalesce(json_agg(\"root\" ORDER BY \"root.pg.number\" DESC NULLS FIRST), '[]' ) AS \"root\" FROM  (SELECT  row_to_json((SELECT  \"_e\"  FROM  (SELECT  \"_root.base\".\"number\" AS \"number\"       ) AS \"_e\"      ) ) AS \"root\", \"_root.base\".\"number\" AS \"root.pg.number\" FROM  (SELECT  *  FROM \"public\".\"Epoch\"  WHERE ('true')   ORDER BY \"number\" DESC NULLS FIRST LIMIT 1 ) AS \"_root.base\"    ORDER BY \"root.pg.number\" DESC NULLS FIRST  ) AS \"_root\"      "},"path":"$"},"query":{"type":null},"query_execution_time":1.657673e-3,"request_id":"2fc31b05-0f3c-4dd6-9192-60204fbb4c8d","request_mode":"error","request_read_time":1.72e-6,"response_size":826,"uncompressed_response_size":826,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"2fc31b05-0f3c-4dd6-9192-60204fbb4c8d"},"level":"error","span_id":"95066c5648e8a6ec","timestamp":"2024-10-01T12:55:32.259+0000","trace_id":"671b908ec3a31d38c24227731b82c72e","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.122","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"no mutations exist","path":"$"},"query":{"type":null},"query_execution_time":1.67017e-4,"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a","request_mode":"error","request_read_time":1.66637e-4,"response_size":68,"uncompressed_response_size":68,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"362d856b-1c3c-4587-ab1e-a5650317c48a"},"level":"error","span_id":"e2a38977c75f3546","timestamp":"2024-10-01T12:55:33.259+0000","trace_id":"ba50a6997116f6e3be10e397d77dc9d6","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.50166e-4,"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55","request_mode":"error","request_read_time":2.36e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"1fa181ec-87d2-427f-8a4b-6143d23bae55"},"level":"error","span_id":"e8d20c1563873073","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"2fe4e636804b493c12da50d984004e3c","type":"http-log"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"10.42.51.87","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"error":{"code":"validation-failed","error":"field 'epochs' not found in type: 'query_root'","path":"$.selectionSet.epochs"},"query":{"type":null},"query_execution_time":2.43936e-4,"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95","request_mode":"error","request_read_time":2.06e-6,"response_size":116,"uncompressed_response_size":116,"user_vars":{"x-hasura-role":"cardano-graphql"}},"request_id":"0ed47e3e-4e7a-41af-a66e-3dc3c854cb95"},"level":"error","span_id":"5d374fc3c09d8b7a","timestamp":"2024-10-01T12:55:56.281+0000","trace_id":"0416d3597fb9ebc592dbaffe7483147d","type":"http-log"}

//edit
forgot I have warn level. Here's debug from start:

{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"database url for migrations is set by HASURA_GRAPHQL_DATABASE_URL"}}
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"migrations server port env var is not set, defaulting to 9691"}}
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"starting graphql engine temporarily on port 9691"}}
{"timestamp":"2024-10-02T12:37:14.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"waiting 60 for 9691 to be ready"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"admin_secret_set":false,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"console_sentry_dsn":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["metadata"],"enabled_log_types":["startup","http-log","jwk-refresh-log","webhook-log","websocket-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"debug","port":9691,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":null,"use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"Refined (Seconds {seconds = 3s})","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Refined (Seconds {seconds = 5s})}"},"kind":"server_configuration"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"database_url":"postgres://cardano-mainnet:[email protected]:5432/cardano-mainnet","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"port 9691 is ready"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_MIGRATIONS_DIR is not set, defaulting to /hasura-migrations"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-startup","info":"env var HASURA_GRAPHQL_METADATA_DIR is not set, defaulting to /hasura-metadata"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-migrations does not exist, skipping migrations"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"directory /hasura-metadata does not exist, skipping metadata"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"killing temporary server"}}
{"timestamp":"2024-10-02T12:37:15.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-shutdown","info":"graphql-engine will now start in normal mode"}}
{"detail":{"info":"WARNING: No admin secret provided","kind":"no_admin_secret"},"level":"warn","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"admin_secret_set":false,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"console_sentry_dsn":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["metrics","metadata","config","graphql"],"enabled_log_types":["startup","http-log","jwk-refresh-log","webhook-log","websocket-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"debug","port":9090,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":null,"use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"Refined (Seconds {seconds = 3s})","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Refined (Seconds {seconds = 5s})}"},"kind":"server_configuration"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"database_url":"postgres://cardano-mainnet:[email protected]:5432/cardano-mainnet","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":"Thread runCronEventsGenerator (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread processScheduledTriggers (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"Unlocking all locked scheduled events on `hdb_scheduled_events` and `hdb_cron_events` tables","kind":"scheduled_triggers"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread checkForUpdates (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread websocketConnectionReaper (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"Already at the latest catalog version (48); nothing to do.","kind":"catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Schema sync enabled. Polling at Refined (Milliseconds {milliseconds = 1s})","kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"instance_id":"b580b3d4-6fb3-4180-bc52-22dfc2baba95","message":"listener thread started","thread_id":"ThreadId 72"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"message":"source \"default\" has not been initialized yet.","source":"default"},"kind":"source_catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"instance_id":"b580b3d4-6fb3-4180-bc52-22dfc2baba95","message":"processor thread started","thread_id":"ThreadId 95"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Starting in eventing enabled mode","kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Starting workers","kind":"event_triggers"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":"Help us improve Hasura! The graphql-engine server collects anonymized usage stats which allows us to keep improving Hasura at warp speed. To read more or opt-out, visit https://hasura.io/docs/latest/graphql/core/guides/telemetry.html","kind":"telemetry"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":{"info":{"message":"Starting API server","time_taken":0.394567301},"kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread SchemeUpdate.listener (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread SchemeUpdate.processor (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread processEventQueue (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread sourcePingPoller (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread runTelemetry (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"metrics_info: {\"server_metrics\":{\"resource_cpu\":4096,\"resource_memory\":10737418240,\"resource_checker_error_code\":null}}","level":"debug","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Shutting websockets server down","level":"debug","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal updateJWK thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal runTelemetry thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal websocketConnectionReaper thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal sourcePingPoller thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Stopping immortal checkForUpdates thread","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"All in-flight events have finished processing","kind":"scheduled_events"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread ourIdleGC (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread asyncActionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread updateJWK (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":{"info":"Gracefully shutting down server","kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"startup"}
{"detail":"Thread asyncActionSubscriptionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:15.264+0000","type":"unstructured"}
{"detail":"Thread runCronEventsGenerator (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread processScheduledTriggers (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread checkForUpdates (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread sourcePingPoller (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":{"info":"Unlocking all locked scheduled events on `hdb_scheduled_events` and `hdb_cron_events` tables","kind":"scheduled_triggers"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Already at the latest catalog version (48); nothing to do.","kind":"catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Schema sync enabled. Polling at Refined (Milliseconds {milliseconds = 1s})","kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"instance_id":"e9c63890-18ac-4a6f-a0d1-623c32926af0","message":"listener thread started","thread_id":"ThreadId 72"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"EE client credentials not present in the metadata database. Hasura EE features are disabled.","kind":"license_info"},"level":"warn","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"message":"source \"default\" has not been initialized yet.","source":"default"},"kind":"source_catalog_migrate"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"instance_id":"e9c63890-18ac-4a6f-a0d1-623c32926af0","message":"processor thread started","thread_id":"ThreadId 94"},"kind":"schema-sync"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Starting in eventing enabled mode","kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Starting workers","kind":"event_triggers"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":"Help us improve Hasura! The graphql-engine server collects anonymized usage stats which allows us to keep improving Hasura at warp speed. To read more or opt-out, visit https://hasura.io/docs/latest/graphql/core/guides/telemetry.html","kind":"telemetry"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":{"info":{"message":"Starting API server","time_taken":0.443591001},"kind":"server"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"startup"}
{"detail":"Thread SchemeUpdate.listener (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread SchemeUpdate.processor (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread asyncActionSubscriptionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread websocketConnectionReaper (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread runTelemetry (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"metrics_info: {\"server_metrics\":{\"resource_cpu\":4096,\"resource_memory\":10737418240,\"resource_checker_error_code\":null}}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread ourIdleGC (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread processEventQueue (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread updateJWK (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Thread asyncActionsProcessor (re)started","level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"http_success: {\"error\":\"invalid_payload\"}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":{"http_error":{"http_exception":null,"response":"{\"error\":\"invalid_payload\"}","status_code":400,"url":"https://telemetry.hasura.io/v1/http"},"message":"failed to post telemetry","type":"http_error"},"level":"info","timestamp":"2024-10-02T12:37:16.234+0000","type":"telemetry-log"}
{"detail":"metrics_info: {\"topic\":\"server_metrics_v2\",\"data\":{\"metadata_db_uid\":\"156d95bd-4660-480c-9b89-a7a37d9dd9ff\",\"instance_uid\":\"e9c63890-18ac-4a6f-a0d1-623c32926af0\",\"hasura_version\":\"v2.43.0\",\"ci\":null,\"source_metadata\":{\"db_uid\":\"156d95bd-4660-480c-9b89-a7a37d9dd9ff\",\"backend_type\":\"postgres\",\"db_kind\":\"postgres\",\"db_version\":\"150007\"},\"metrics\":{\"tables\":6,\"views\":27,\"enum_tables\":0,\"relationships\":{\"manual\":82,\"auto\":0},\"permissions\":{\"select\":32,\"insert\":1,\"update\":1,\"delete\":1,\"roles\":1},\"event_triggers\":0,\"functions\":0,\"remote_schemas\":0,\"service_timings\":{\"collectionTag\":1727872637,\"serviceTimingMetrics\":[]},\"actions\":{\"synchronous\":0,\"asynchronous\":0,\"query_actions\":0,\"type_relationships\":0,\"custom_types\":0},\"native_queries\":{\"with_parameters\":0,\"without_parameters\":0},\"stored_procedures\":{\"with_parameters\":0,\"without_parameters\":0},\"logical_models\":{\"count\":0}},\"experimental_features\":[]}}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"http_success: {\"message\":\"ok\"}","level":"debug","timestamp":"2024-10-02T12:37:16.234+0000","type":"unstructured"}
{"detail":"Stopping immortal processScheduledTriggers thread","level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"unstructured"}
{"detail":{"num_cron_events_fetched":0,"num_fetches":1,"num_one_off_scheduled_events_fetched":0},"level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"scheduled-trigger-process"}
{"detail":"Stopping immortal runCronEventsGenerator thread","level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"unstructured"}
{"detail":"Stopping immortal asyncActionSubscriptionsProcessor thread","level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"unstructured"}
{"detail":{"info":"All in-flight events have finished processing","kind":"async_actions"},"level":"info","timestamp":"2024-10-02T12:37:25.275+0000","type":"startup"}
{"detail":"Stopping immortal asyncActionsProcessor thread","level":"info","timestamp":"2024-10-02T12:37:26.276+0000","type":"unstructured"}
{"detail":{"info":"All in-flight events have finished processing","kind":"event_triggers"},"level":"info","timestamp":"2024-10-02T12:37:26.276+0000","type":"startup"}
{"detail":"Stopping immortal processEventQueue thread","level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"unstructured"}
{"detail":"Stopping immortal SchemeUpdate.processor thread","level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"unstructured"}
{"detail":"Stopping immortal SchemeUpdate.listener thread","level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"unstructured"}
{"detail":{"num_events_fetched":{},"num_fetches":13},"level":"info","timestamp":"2024-10-02T12:37:27.277+0000","type":"event-trigger-process"}
{"detail":{"http_info":{"content_encoding":null,"http_version":"HTTP/1.1","ip":"51.91.233.11","method":"GET","status":200,"url":"/healthz"},"operation":{"query":{"type":null},"request_id":"14b2edc0-899c-40b8-b84c-dd350104184b","request_mode":"non-graphql","response_size":2,"uncompressed_response_size":2},"request_id":"14b2edc0-899c-40b8-b84c-dd350104184b"},"level":"info","timestamp":"2024-10-02T12:37:38.258+0000","type":"http-log"}

@Kammerlo
Copy link
Member

Kammerlo commented Oct 2, 2024

What about your background service? Is this one running properly?

@mkocur-ledger
Copy link
Author 

{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"Db","msg":"Initializing...","time":"2024-10-02T13:03:43.963Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraBackgroundClient","msg":"Initializing","time":"2024-10-02T13:03:43.997Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraBackgroundClient","msg":"Initialized","time":"2024-10-02T13:03:51.805Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"MetadataFetchClient","msg":"Initializing","time":"2024-10-02T13:03:51.805Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"MetadataFetchClient","msg":"Metadata Server is up and running. Checking Sync Status.","time":"2024-10-02T13:03:51.813Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"MetadataFetchClient","msg":"Initialized","time":"2024-10-02T13:03:51.817Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"ChainFollower","msg":"Initializing","time":"2024-10-02T13:03:51.817Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"ChainFollower","msg":"Initialized","time":"2024-10-02T13:03:51.873Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"Worker","msg":"Starting","time":"2024-10-02T13:03:51.873Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"Worker","msg":"Started","time":"2024-10-02T13:03:52.018Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"id":"a650a3f398ba4a9427ec8c293e9f7156d81fd2f7ca849014d8d2c1156c359b3a","slot":23068800,"msg":"","time":"2024-10-02T13:03:52.018Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"8cfe22b2dd7708ff8ed5e65e5c8d8ce0a65365cc805b3ffa51946550444153","msg":"Adding metadata to asset","time":"2024-10-02T13:03:58.906Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"02b261951d13fddebeca13abe1ea9b096b6e4131364de1b24f32a23d41504f","msg":"Adding metadata to asset","time":"2024-10-02T13:03:58.921Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"e4214b7cce62ac6fbba385d164df48e157eae5863521b4b67ca71d867b4ccb2dd9de12c339a4aa099fe524342ffdc835be131122deacba92d9139f5b","msg":"Adding metadata to asset","time":"2024-10-02T13:03:58.938Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"605f3d2b1bdb339e0712c9b33be8df5da84c8fb8de969dacca5939b24d6f6d73","msg":"Adding metadata to asset","time":"2024-10-02T13:03:58.949Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"e4214b7cce62ac6fbba385d164df48e157eae5863521b4b67ca71d86df4422ce3d80fa783c1001758390c58e5f6a33c63d9622658a80e29565dcd1be","msg":"Adding metadata to asset","time":"2024-10-02T13:03:59.029Z","v":0}
{"name":"background","hostname":"int-ada-mainnet-node-cardano-background-0","pid":7,"level":30,"module":"HasuraClient","assetId":"ed5517ccf67c60004355cee3c546c77226cd89a04b3aaeae6a65589e4d6f6d","msg":"Adding metadata to asset","time":"2024-10-02T13:03:59.044Z","v":0}
[...]

@Kammerlo
Copy link
Member

Kammerlo commented Oct 8, 2024

@mkocur-ledger I doubled checked and tested the whole setup multiple times and couldn't reproduce the issue. Within your logs I see that the Hasurabackgroundclient is initialized. The schemas are created on this line

cardano-graphql/packages/api-cardano-db-hasura/src/HasuraBackgroundClient.ts

Line 59 in 16ed6d7

await this.applySchemaAndMetadata()
which should be executed due to the log that it's initialized

Another Idea I want to check is if db-sync is running properly. Could you check the logs if there is something noticeable?

@mkocur-ledger
Copy link
Author

mkocur-ledger commented Oct 11, 2024 •
edited
Loading

Except what I found in the log at the very top of this ticket, not really. But that was fixed by adding metadata table manually.

Maybe you just miss those messages? I can see them when:

  • I have restored from snapshot
  • Hasura works with background for like 2 hours
  • Hasura logs are on WARN level. It's very easy to miss them on info.

(BTW I start hasura, background, ogmios and graphql when db-sync reaches current epoch)

@Kartiiyer12
Copy link
Collaborator

Hi @mkocur-ledger Can you check in the database directly, the Epoch and the Withdrawal table exists?
Also, what happens when you try to query the data via GraphQL for Epoch related data?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
BUG
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Kammerlo @Kartiiyer12 @mkocur-ledger and others