ClickHouse File Does not Exist Error After Shard Increase

[karthik-thiyagarajan]

I started to see the below error, after I upgraded my clickhouse shards from 6 shards to 8 shards. I also started to see the DistributedFilesInsert count getting increased every 2 to 3 minutes and the total delayedInsert is also getting increased where as it was literally between 10 to 25, where as it has now increased to 125k in the last 2 days.

I see the below error - can you help to debug this ?

---

Defaulted container "clickhouse" out of: clickhouse, clickhouse-backup, clickhouse-log
2024.11.11 16:10:46.450235 [ 604 ] {} myproduct_monitoring.client_stats.DistributedInsertQueue.default: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/store/712/71257fd8-10bf-47e5-9d37-b4d988de7dfc/shard8_all_replicas/10030.bin: , errno: 2, strerror: No such file or directory: While sending /var/lib/clickhouse/store/712/71257fd8-10bf-47e5-9d37-b4d988de7dfc/shard8_all_replicas/10030.bin. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d02cb1b

1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000078e1d0c

2. DB::ErrnoException::ErrnoException<String&>(int, int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000d08783d

3. void DB::ErrnoException::throwFromPath<String&>(int, String const&, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000d0869c8

4. DB::ReadBufferFromFile::ReadBufferFromFile(String const&, unsigned long, int, char*, unsigned long, std::optional, std::shared_ptrDB::Throttler) @ 0x000000000d08a32b

5. DB::DistributedAsyncInsertDirectoryQueue::processFile(String&, DB::SettingsChanges const&) @ 0x00000000123ddc2a

6. DB::DistributedAsyncInsertDirectoryQueue::processFiles(DB::SettingsChanges const&) @ 0x00000000123d4113

7. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptrDB::IDisk const&, String const&, std::shared_ptrDB::ConnectionPoolWithFailover, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x00000000123e1d55

8. DB::BackgroundSchedulePool::threadFunction() @ 0x0000000010661860

9. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000010662907

10. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_deletestd::__thread_struct>, void ThreadPoolImplstd::thread::scheduleImpl(std::function<void ()>, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000d0e55e3

11. ? @ 0x00007fd6a056d609

12. ? @ 0x00007fd6a0488353
    (version 24.6.3.95 (official build))
    2024.11.11 16:11:16.450513 [ 587 ] {} myproduct_monitoring.client_stats.DistributedInsertQueue.default: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/store/712/71257fd8-10bf-47e5-9d37-b4d988de7dfc/shard8_all_replicas/10030.bin: , errno: 2, strerror: No such file or directory: While sending /var/lib/clickhouse/store/712/71257fd8-10bf-47e5-9d37-b4d988de7dfc/shard8_all_replicas/10030.bin. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

13. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d02cb1b

14. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x00000000078e1d0c

15. DB::ErrnoException::ErrnoException<String&>(int, int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000d08783d

16. void DB::ErrnoException::throwFromPath<String&>(int, String const&, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000d0869c8

17. DB::ReadBufferFromFile::ReadBufferFromFile(String const&, unsigned long, int, char*, unsigned long, std::optional, std::shared_ptrDB::Throttler) @ 0x000000000d08a32b

18. DB::DistributedAsyncInsertDirectoryQueue::processFile(String&, DB::SettingsChanges const&) @ 0x00000000123ddc2a

19. DB::DistributedAsyncInsertDirectoryQueue::processFiles(DB::SettingsChanges const&) @ 0x00000000123d4113

20. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptrDB::IDisk const&, String const&, std::shared_ptrDB::ConnectionPoolWithFailover, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x00000000123e1d55

21. DB::BackgroundSchedulePool::threadFunction() @ 0x0000000010661860

22. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000010662907

23. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_deletestd::__thread_struct>, void ThreadPoolImplstd::thread::scheduleImpl(std::function<void ()>, Priority, std::optional, bool)::'lambda0'()>>(void*) @ 0x000000000d0e55e3

24. ? @ 0x00007fd6a056d609

25. ? @ 0x00007fd6a0488353
    (version 24.6.3.95 (official build))

[Slach]

could you share
kubectl get chi -n <your-namespace> <your-chi-name> -o yaml

without sensitive information?