Daser is not sampling a block with txs

[Bidon15]

Celestia Node version

2fde72c

OS

Ubuntu 20.04.3 LTS / Testground

Install tools

• install testground
• install docker

Others

No response

Steps to reproduce it

1. Run tc-002-da-sync from big-blocks testplan
2. Wait until 4th block gets filled with txs

Expected result

Full and Light nodes can das 4th block and onwards

Actual result

Full and Light are not dasing the filled 4th block and onwards

Relevant log output

<< validators[002] (338ca4) >> {"height":"4","txhash":"39314ACBDA9B1049A069E99676C736EBF87318B437D29815FB11BF0C032CDB99","codespace":"","code":0,"data":"12200A1E2F7061796D656E742E4D7367506179466F7244617461526573706F6E7365","raw_log":"[{\"msg_index\":0,\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"/payment.MsgPayForData\"}]},{\"type\":\"payfordata\",\"attributes\":[{\"key\":\"signer\",\"value\":\"celestia1vdjkcetnw35kzvtyd4ukcemev468getkx4jkuapedsehydm9de6kzwrdd34nyvr8ve5rwup4vd5q5w3z8x\"},{\"key\":\"size\",\"value\":\"10000\"}]}]}]","logs":[{"msg_index":0,"log":"","events":[{"type":"message","attributes":[{"key":"action","value":"/payment.MsgPayForData"}]},{"type":"payfordata","attributes":[{"key":"signer","value":"celestia1vdjkcetnw35kzvtyd4ukcemev468getkx4jkuapedsehydm9de6kzwrdd34nyvr8ve5rwup4vd5q5w3z8x"},{"key":"size","value":"10000"}]}]}],"info":"","gas_wanted":"1000000000","gas_used":"71816","tx":null,"timestamp":"","events":[{"type":"coin_spent","attributes":[{"key":"c3BlbmRlcg==","value":"Y2VsZXN0aWExZG15bGd5ZXR0ZXY1ZW50OWwzcjdlbnVhOG1sazIwZ2ZoN3A1Y2g=","index":true},{"key":"YW1vdW50","value":"MTAwMDAwMDAwMDAwdXRpYQ==","index":true}]},{"type":"coin_received","attributes":[{"key":"cmVjZWl2ZXI=","value":"Y2VsZXN0aWExN3hwZnZha20yYW1nOTYyeWxzNmY4NHoza2VsbDhjNWxwbmpzM3M=","index":true},{"key":"YW1vdW50","value":"MTAwMDAwMDAwMDAwdXRpYQ==","index":true}]},{"type":"transfer","attributes":[{"key":"cmVjaXBpZW50","value":"Y2VsZXN0aWExN3hwZnZha20yYW1nOTYyeWxzNmY4NHoza2VsbDhjNWxwbmpzM3M=","index":true},{"key":"c2VuZGVy","value":"Y2VsZXN0aWExZG15bGd5ZXR0ZXY1ZW50OWwzcjdlbnVhOG1sazIwZ2ZoN3A1Y2g=","index":true},{"key":"YW1vdW50","value":"MTAwMDAwMDAwMDAwdXRpYQ==","index":true}]},{"type":"message","attributes":[{"key":"c2VuZGVy","value":"Y2VsZXN0aWExZG15bGd5ZXR0ZXY1ZW50OWwzcjdlbnVhOG1sazIwZ2ZoN3A1Y2g=","index":true}]},{"type":"tx","attributes":[{"key":"ZmVl","value":"MTAwMDAwMDAwMDAwdXRpYQ==","index":true}]},{"type":"tx","attributes":[{"key":"YWNjX3NlcQ==","value":"Y2VsZXN0aWExZG15bGd5ZXR0ZXY1ZW50OWwzcjdlbnVhOG1sazIwZ2ZoN3A1Y2gvMQ==","index":true}]},{"type":"tx","attributes":[{"key":"c2lnbmF0dXJl","value":"aThEUkZDNXJzekVFeGNDMElSYW9WcWFKTlRwUVdZK2l1bkd2YVZOdXZ6SW1oblovS3pjMkNSMVhVdlZTZ1BVTmU4SGJPT0VVSFVzU3VWK0ZxVThIVEE9PQ==","index":true}]},{"type":"message","attributes":[{"key":"YWN0aW9u","value":"L3BheW1lbnQuTXNnUGF5Rm9yRGF0YQ==","index":true}]},{"type":"payfordata","attributes":[{"key":"c2lnbmVy","value":"Y2VsZXN0aWExdmRqa2NldG53MzVrenZ0eWQ0dWtjZW1ldjQ2OGdldGt4NGprdWFwZWRzZWh5ZG05ZGU2a3p3cmRkMzRueXZyOHZlNXJ3dXA0dmQ1cTV3M3o4eA==","index":true},{"key":"c2l6ZQ==","value":"MTAwMDA=","index":true}]}]}
<< bridges[002] (2cc42e) >> 2022-10-28T12:40:53.465Z	INFO	header/store	store/store.go:301	new head	{"height": 4, "hash": "0B176A8D35EBC0E5BF105FD5599447DDF0D237F7CD433A1B35DC740EE8F279F4"}
<< lights[000] (c448ad) >> 2022-10-28T12:40:53.466Z	INFO	header/store	store/store.go:301	new head	{"height": 4, "hash": "0B176A8D35EBC0E5BF105FD5599447DDF0D237F7CD433A1B35DC740EE8F279F4"}
<< lights[000] (c448ad) >> 2022-10-28T12:40:53.466Z	DEBUG	pubsub	[email protected]/validation.go:457	validation done; took 571.076µs
<< lights[000] (c448ad) >> 2022-10-28T12:40:53.466Z	DEBUG	header/p2p	p2p/subscription.go:39	received message	{"topic": "header-sub", "sender": "12D3KooWQTTEdvu1C3RxuM5oWGBd8vpBTnQ4upoD4s2BqTcCyWgu"}
<< lights[000] (c448ad) >> 2022-10-28T12:40:53.466Z	DEBUG	header/p2p	p2p/subscription.go:46	received new ExtendedHeader	{"height": 4, "hash": "0B176A8D35EBC0E5BF105FD5599447DDF0D237F7CD433A1B35DC740EE8F279F4"
<< lights[000] (c448ad) >> 2022-10-28T12:40:53.467Z	INFO	das	das/subscriber.go:33	new header received via subscription	{"height": 4}
<< lights[000] (c448ad) >> 2022-10-28T12:40:53.468Z	DEBUG	das	das/worker.go:44	start sampling worker	{"from": 4, "to": 4}
<< lights[000] (c448ad) >> 2022-10-28T12:40:53.468Z	DEBUG	das	das/worker.go:62	got header from header store	{"height": 4, "hash": "0B176A8D35EBC0E5BF105FD5599447DDF0D237F7CD433A1B35DC740EE8F279F4", "square width": 32, "data root": "a8WpChSPfde0wtYjU7NVxFVvkl7zykCRvJfocnGMGNo=", "finished (s)": 0.000002536}
<< lights[000] (c448ad) >> 2022-10-28T12:40:53.468Z	DEBUG	share/light	light/availability.go:60	Validate availability	{"root": "a8WpChSPfde0wtYjU7NVxFVvkl7zykCRvJfocnGMGNo="}

<< fulls[002] (00d797) >> 2022-10-28T12:40:53.471Z	DEBUG	share/eds	eds/retriever.go:236	requesting quadrant	{"axis": 1, "x": 1, "y": 0, "size": 16}
<< fulls[002] (00d797) >> 2022-10-28T12:41:23.472Z	WARN	share/eds	eds/retriever.go:254	quadrant request timeout	{"timeout": "30s", "axis": 1, "x": 1, "y": 0, "size": 16}

Notes

If we have empty blocks -> DASer works fine with dasing latest headers

Logs https://gist.github.com/Bidon15/8613e4881f8e309dfbd2e530ce00be6d

[renaynay]

Height 4 is width 32

[Bidon15]

So, it's an issue lying in how we use modules in testground
celestiaorg/test-infra#97 vs celestiaorg/test-infra#102

On PR 97, this bug is not reproducing as well as in production
However, in PR 102 we can clearly see that bitswap is not kicking in

Bitswap is not kicking in using monorepo: https://gist.github.com/Bidon15/af38abb3e9c35b4947c70e38003eead9
Everything works as expected on non-monorepo(previous): https://gist.github.com/Bidon15/23cc9ff205852a87ee4f8294df1b578a

cc: @derrandz

[derrandz]

@Bidon15 Noted, so our theory about dependencies mismatch is on-point. Will debug this sometime today

[Bidon15]

Closing as we have found the root cause in how monorepo behaves