Failed to collect receipts in exchange for an on-chain query fee voucher

[86b]

I've been seeing the following (along with 6 other allocation ID's) every time I start up the indexer-agent container where it appears the agent is having issues collecting receipts for certain allocations:

{
  "level": 30,
  "time": 1626348829631,
  "pid": 1,
  "hostname": "5f1eb7ae2d18",
  "name": "IndexerAgent",
  "component": "AllocationReceiptCollector",
  "allocation": "0x77a99255419C67884A22f47fa599700E0913b46C",
  "receipts": 616,
  "msg": "Collect receipts for allocation"
}
{
  "level": 50,
  "time": 1626348830322,
  "pid": 1,
  "hostname": "5f1eb7ae2d18",
  "name": "IndexerAgent",
  "component": "AllocationReceiptCollector",
  "allocation": "0x77a99255419C67884A22f47fa599700E0913b46C",
  "err": {
    "type": "IndexerError",
    "message": "Failed to collect receipts in exchange for query fee voucher",
    "stack": "IndexerError: Failed to collect receipts in exchange for query fee voucher\n    at Object.indexerError (/opt/indexer/packages/indexer-common/dist/errors.js:139:12)\n    at AllocationReceiptCollector.<anonymous> (/opt/indexer/packages/indexer-agent/dist/query-fees/allocations.js:205:130)\n    at Generator.throw (<anonymous>)\n    at rejected (/opt/indexer/packages/indexer-agent/dist/query-fees/allocations.js:6:65)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)",
    "code": "IE054",
    "explanation": "https://github.com/graphprotocol/indexer/blob/master/docs/errors.md#ie054",
    "cause": {
      "message": "Request failed with status code 500",
      "stack": "Error: Request failed with status code 500\n    at createError (/opt/indexer/node_modules/axios/lib/core/createError.js:16:15)\n    at settle (/opt/indexer/node_modules/axios/lib/core/settle.js:17:12)\n    at IncomingMessage.handleStreamEnd (/opt/indexer/node_modules/axios/lib/adapters/http.js:260:11)\n    at IncomingMessage.emit (events.js:333:22)\n    at IncomingMessage.EventEmitter.emit (domain.js:482:12)\n    at endReadableNT (_stream_readable.js:1204:12)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)",
      "config": {
        "url": "https://gateway.network.thegraph.com/collect-receipts",
        "method": "post",
        "data": {},
        "headers": {
          "Accept": "application/json, text/plain, */*",
          "Content-Type": "application/octet-stream",
          "User-Agent": "axios/0.21.1",
          "Content-Length": 69012
        },
        "transformRequest": [
          null
        ],
        "transformResponse": [
          null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1
      }
    }
  },
  "msg": "Failed to collect receipts in exchange for an on-chain query fee voucher"
}

I'm running in a docker-compose environment with the following:

graphprotocol/graph-node:v0.23.1
ghcr.io/graphprotocol/indexer-agent:v0.16.0
ghcr.io/graphprotocol/indexer-service:v0.16.0

I have the following set in the indexer-agent:

INDEXER_AGENT_COLLECT_RECEIPTS_ENDPOINT: https://gateway.network.thegraph.com/collect-receipts
INDEXER_AGENT_ALLOCATION_CLAIM_THRESHOLD: 1

[That3Percent]

Can you please attach the request body so I can help figure out what's going on?

[chriswessels]

Seeing this on 10 different allocations. Sample:

{
  "level": 30,
  "time": 1627385952151,
  "pid": 1,
  "hostname": "80ccfd76f085",
  "name": "IndexerAgent",
  "component": "AllocationReceiptCollector",
  "allocation": "0xb709e3FfA8C2191Dd77Dd902a7aAF6bE16Ca1F41",
  "receipts": 26,
  "msg": "Collect receipts for allocation"
}
{
   "level":50,
   "time":1627385952209,
   "pid":1,
   "hostname":"80ccfd76f085",
   "name":"IndexerAgent",
   "component":"AllocationReceiptCollector",
   "allocation":"0xb709e3FfA8C2191Dd77Dd902a7aAF6bE16Ca1F41",
   "err":{
      "type":"IndexerError",
      "message":"Failed to collect receipts in exchange for query fee voucher",
      "stack":"IndexerError: Failed to collect receipts in exchange for query fee voucher\n    at Object.indexerError (/opt/indexer/packages/indexer-common/dist/errors.js:139:12)\n    at AllocationReceiptCollector.<anonymous> (/opt/indexer/packages/indexer-agent/dist/query-fees/allocations.js:206:130)\n    at Generator.throw (<anonymous>)\n    at rejected (/opt/indexer/packages/indexer-agent/dist/query-fees/allocations.js:6:65)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)",
      "code":"IE054",
      "explanation":"https://github.com/graphprotocol/indexer/blob/master/docs/errors.md#ie054",
      "cause":{
         "message":"Request failed with status code 500",
         "stack":"Error: Request failed with status code 500\n    at createError (/opt/indexer/node_modules/axios/lib/core/createError.js:16:15)\n    at settle (/opt/indexer/node_modules/axios/lib/core/settle.js:17:12)\n    at IncomingMessage.handleStreamEnd (/opt/indexer/node_modules/axios/lib/adapters/http.js:260:11)\n    at IncomingMessage.emit (events.js:333:22)\n    at IncomingMessage.EventEmitter.emit (domain.js:482:12)\n    at endReadableNT (_stream_readable.js:1204:12)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)",
         "config":{
            "url":"https://gateway.network.thegraph.com/collect-receipts",
            "method":"post",
            "data":{
               
            },
            "headers":{
               "Accept":"application/json, text/plain, */*",
               "Content-Type":"application/octet-stream",
               "User-Agent":"axios/0.21.1",
               "Content-Length":2932
            },
            "transformRequest":[
               null
            ],
            "transformResponse":[
               null
            ],
            "timeout":0,
            "xsrfCookieName":"XSRF-TOKEN",
            "xsrfHeaderName":"X-XSRF-TOKEN",
            "maxContentLength":-1,
            "maxBodyLength":-1
         }
      }
   },
   "msg":"Failed to collect receipts in exchange for an on-chain query fee voucher"
}

[That3Percent]

Can you try running it again using this PR #293 and sending me the new logs?

This will log the payload as hex data, hopefully allowing me to reproduce the issue using the Gateway's code on your data. If the issue does not reproduce, I'm afraid it's something in our middleware (eg: axios) not handling binary data effectively. Either way, this should eliminate one possibility or the other.

[chriswessels]

Provided logs to @That3Percent and he deployed a fix. I can confirm this is now working for me. Existing receipts have been swapped for vouchers and saved into DB. Thanks!

[86b]

@fordN @That3Percent Looks like I'm still seeing issues with this fix, however I can see additional information now in the logs:

{
  "level": 30,
  "time": 1628400118202,
  "pid": 1,
  "hostname": "62400e060233",
  "name": "IndexerAgent",
  "component": "AllocationReceiptCollector",
  "allocation": "0x3614240e15F43644e24290A29831aa4B557BBfb9",
  "receipts": 15,
  "msg": "Collect receipts for allocation"
}

{
  "level": 20,
  "time": 1628400118204,
  "pid": 1,
  "hostname": "62400e060233",
  "name": "IndexerAgent",
  "component": "AllocationReceiptCollector",
  "allocation": "0x3614240e15F43644e24290A29831aa4B557BBfb9",
  "data": "3614240e15f43644e24290a29831aa4b557bbfb9000000000000000000000000000000000000000000000000000000000000000002ac031ffdb94d653c397abf424b34d5ab8980a92769fd1a49fc9c2687204aec4bfe13b9be92230158b6000e61d10a7e3a682a15724fba38df094997bb62267128821e5646bd54cbd41f4854e959d31c00000000000000000000000000000000000000000000000000000000000000000aab96558faea20c78bf4bdd026d25d0c3c19b9e43fa2b2944966fe5c492ee326e11ab2c4049d094a83f710522010a76dd8317ae10f2d93a05316c2d784a624881e8abff403d7d32f8cf90551db2051b000000000000000000000000000000000000000000000000000000000000000015a4386a5671cd7295f28fe7f14d066b43de24011cd2b4dea5599ddcbc3e402c576913ed6329637d4e732442e385594e32f13a800dc85cbd174fdbf645c0e2bafd34856b797840725b252bf65a33631b00000000000000000000000000000000000000000000000000000000000000001b7077930495b9325599ae509fcb56309107aefdae000cc0eb67350656a3a56a58314809a31132a10857f28ee81a6c6b70deb9e98abd6ebb0effc80c68e7c09359dc0fec6a5793e0352da128afde7d1c000000000000000000000000000000000000000000000000000000000000000027f24e32f462b9d6b36def4ad1583afe51a72d570af3b1444b955e764ba05e957d90b3d734c3a16899be6a9691fcdf7b72f01ffd489e7d8d6dddcd8b8cbe00ea825ba220801fc667c31a9d371941d61c00000000000000000000000000000000000000000000000000000000000000002b2d6198849b5f31920d41624a2c10c3e2b9df615f148874b1f2b1ff975444be23e32a29ae8b38610d90578c2007a7569a71de6c22664a0e873d6a27b9f91b6ed848f52afdc1ed718f29f21adb20321c00000000000000000000000000000000000000000000000000000000000000002cb76ac59004096dd0cc19f65c064b7f35b001e43c15e3b565be2f91f73560bec5c8782c912f556a233556e63ca60677d741777c4b4a5834bea8c97d5d1e6c7a0f7086a94dad44df40e264c06574261b0000000000000000000000000000000000000000000000000000000000000000353b29052d6b0d45cb1a82d4bd4f3d893cde371862c5019f66cd2541ea710ce1dea1349486d2aef840d6a0557159420c483f40bc6fd6c6d663b8092fbee83acb0aa55f5e07574f396bc3ecc57cf32d1b00000000000000000000000000000000000000000000000000000000000000004283503c209a7ce8e18e69a1ed2a47ff6dec2705b694680fb17305ae499335b871f702c6e60525387c1bf6011c18c03a9393aa144ffc9a63057ea848fefbad71ee0d26cdd810dbd021706fbd5f59a81c0000000000000000000000000000000000000000000000000000000000000000666a23a4f802d5be5f336f5a85a840210e2db262b96c777f2c6f708448bfd0ca7c7a5ff3ba9158ea7ebf155a1243c4285dc726bf3023063a64d0c111dfcb53fb305f4f0e29f4c3114c86db1bb31a351b00000000000000000000000000000000000000000000000000000000000000006aacfabc6dfa29df5bbc6c7beaf93dac69dec1710f3b14488d4a4dc0a9394b3d024796abf3ee8ebc4b0cff5e115c6541634401349c411c750de579732da808109bb56241038b231ae8f605f37ba85c1c0000000000000000000000000000000000000000000000000000000000000000723e5591b29054b5b82d09cea88af64019313fcdd2e3205021849beb5fc8a672172e11b59e8a3ff39220831a6b09d6292784029344b9af94f0fc0a082ff431766004c066a3ef2f19c23c88b48cdfb01c0000000000000000000000000000000000000000000000000000000000000000b8e5eecb34b48a3762ee68011e5a0d0c6da6c61e244456a9b734bb3a205cd45ff49f8296d2339fb4e5147c7780988f6683ffbc9a727c32e58654e31fcc4558196ef1086f15e1ea9f10849c0ed953391b0000000000000000000000000000000000000000000000000000000000000000f38219de4909ccdabb68a6c59c7991ffdf0de890043f52e1abed473004ae15eb69c03a745744ccb07712c79c7f3a881be57ffc7cc5f020ce23c58b527f83a640e213bad51f5acc5829aa7638bce4941b0000000000000000000000000000000000000000000000000000000000000000f54af804c60a144d5f9d739be55874ba63abfd246e25a238452860579b2f03cb3214418a487c97bc642dd5908bd3fb248255dd01cfc293d8ef9c4f9f4186d8fefc3adf21565072dafcb855a82a98cf1c",
  "msg": "Attempting to collect receipts"
}

{
  "level": 50,
  "time": 1628400118371,
  "pid": 1,
  "hostname": "62400e060233",
  "name": "IndexerAgent",
  "component": "AllocationReceiptCollector",
  "allocation": "0x3614240e15F43644e24290A29831aa4B557BBfb9",
  "err": {
    "type": "IndexerError",
    "message": "Failed to collect receipts in exchange for query fee voucher",
    "stack": "IndexerError: Failed to collect receipts in exchange for query fee voucher\n    at Object.indexerError (/opt/indexer/packages/indexer-common/dist/errors.js:139:12)\n    at AllocationReceiptCollector.<anonymous> (/opt/indexer/packages/indexer-agent/dist/query-fees/allocations.js:187:130)\n    at Generator.throw (<anonymous>)\n    at rejected (/opt/indexer/packages/indexer-agent/dist/query-fees/allocations.js:6:65)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)",
    "code": "IE054",
    "explanation": "https://github.com/graphprotocol/indexer/blob/master/docs/errors.md#ie054",
    "cause": {
      "message": "Request failed with status code 500",
      "stack": "Error: Request failed with status code 500\n    at createError (/opt/indexer/node_modules/axios/lib/core/createError.js:16:15)\n    at settle (/opt/indexer/node_modules/axios/lib/core/settle.js:17:12)\n    at IncomingMessage.handleStreamEnd (/opt/indexer/node_modules/axios/lib/adapters/http.js:260:11)\n    at IncomingMessage.emit (events.js:333:22)\n    at IncomingMessage.EventEmitter.emit (domain.js:482:12)\n    at endReadableNT (_stream_readable.js:1204:12)\n    at processTicksAndRejections (internal/process/task_queues.js:84:21)",
      "config": {
        "url": "https://gateway.network.thegraph.com/collect-receipts",
        "method": "post",
        "data": {},
        "headers": {
          "Accept": "application/json, text/plain, */*",
          "Content-Type": "application/octet-stream",
          "User-Agent": "axios/0.21.1",
          "Content-Length": 1700
        },
        "transformRequest": [
          null
        ],
        "transformResponse": [
          null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1
      }
    }
  },
  "msg": "Failed to collect receipts in exchange for an on-chain query fee voucher"
}

[That3Percent]

I'll take a look, thanks.

[That3Percent]

The issue for @86b is that none of the receipts have any value. I'm guessing you are indexing a subgraph with cost model default => 0;

The Gateway fails with the message "Receipts have no value". One issue to resolve is that the Gateway is not propagating these errors to the user though they were intended to be shared.

Another thing we should do is not save a receipt in the db if it has 0 value.
But, because there are some receipts with 0 value already in your db we need to filter those out before submitting for a voucher.

The reason that the Gateway fails is because the contract will revert a transaction attempting to collect for 0 value.