Clickhouse. Operation not permitted: While executing MergeTreeInOrder. (CANNOT_READ_FROM_FILE_DESCRIPTOR).

[odbod-glitch]

Self-Hosted Version

24.8.0

CPU Architecture

x86_64

Docker Version

20.10.16

Docker Compose Version

2.24.1

Steps to Reproduce

I want to upgrade from version 24.2.0 to version 24.8.0. I run install.sh and at the bootstrap and Snuba initialization stage($dcr snuba-api migrations migrate --force)

I don't think it matters that Postgres is not running in a container but on a dedicated host.

Expected Result

migrations have been done

Actual Result

2024.10.10 17:58:25.027091 [ 50 ] {e089b143-7e9f-492d-9e92-3621051dba84} <Error> executeQuery: Code: 74. DB::ErrnoException: Cannot read from file 42, errno: 1, strerror: Operation not permitted: While executing MergeTreeInOrder. (CANNOT_READ_FROM_FILE_DESCRIPTOR) (version 23.8.11.29.altinitystable (altinity build)) (from 192.168.64.5:53946) (in query: SELECT group, migration_id, status FROM migrations_local FINAL WHERE group IN ('system', 'events', 'transactions', 'discover', 'outcomes', 'metrics', 'sessions', 'profiles', 'functions', 'replays', 'generic_metrics', 'search_issues', 'spans', 'group_attributes')), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c61ff37 in /usr/bin/clickhouse
1. DB::ErrnoException::ErrnoException(String const&, int, int, std::optional<String> const&) @ 0x000000000c621494 in /usr/bin/clickhouse
2. DB::ThreadPoolReader::submit(DB::IAsynchronousReader::Request) @ 0x0000000010c30ab6 in /usr/bin/clickhouse
3. DB::AsynchronousReadBufferFromFileDescriptor::asyncReadInto(char*, unsigned long, Priority) @ 0x0000000010c27a67 in /usr/bin/clickhouse
4. DB::AsynchronousReadBufferFromFileDescriptor::nextImpl() @ 0x0000000010c28020 in /usr/bin/clickhouse
5. DB::MergeTreeMarksLoader::loadMarksImpl() @ 0x0000000012d680db in /usr/bin/clickhouse
6. DB::MergeTreeMarksLoader::loadMarks() @ 0x0000000012d672fd in /usr/bin/clickhouse
7. DB::MergeTreeMarksLoader::getMark(unsigned long, unsigned long) @ 0x0000000012d66058 in /usr/bin/clickhouse
8. DB::MergeTreeReaderCompact::initialize() @ 0x0000000012d83abf in /usr/bin/clickhouse
9. DB::MergeTreeReaderCompact::readRows(unsigned long, unsigned long, bool, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x0000000012d8665c in /usr/bin/clickhouse
10. DB::MergeTreeRangeReader::DelayedStream::finalize(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x0000000012dacd22 in /usr/bin/clickhouse
11. DB::MergeTreeRangeReader::read(unsigned long, DB::MarkRanges&) @ 0x0000000012db55c0 in /usr/bin/clickhouse
12. DB::IMergeTreeSelectAlgorithm::readFromPartImpl() @ 0x0000000012da97c8 in /usr/bin/clickhouse
13. DB::IMergeTreeSelectAlgorithm::readFromPart() @ 0x0000000012daa4fa in /usr/bin/clickhouse
14. DB::IMergeTreeSelectAlgorithm::read() @ 0x0000000012da76f5 in /usr/bin/clickhouse
15. DB::MergeTreeSource::tryGenerate() @ 0x00000000135dbd0f in /usr/bin/clickhouse
16. DB::ISource::work() @ 0x0000000013185c06 in /usr/bin/clickhouse
17. DB::ExecutionThreadContext::executeTask() @ 0x000000001319d61a in /usr/bin/clickhouse
18. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000013194410 in /usr/bin/clickhouse
19. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000013193742 in /usr/bin/clickhouse
20. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000131a0faf in /usr/bin/clickhouse
21. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c700e24 in /usr/bin/clickhouse
22. ? @ 0x00007fbb26681609 in ?
23. ? @ 0x00007fbb265a6353 in ?

Event ID

No response

[bijancot]

it's look like you already following the instruction right from docs,

just my two cents, it's kinda happen in low level and sometimes gave me ptsd when debugging C, CMIIW so it's look like happen in clickhouse side related to this issue ClickHouse/ClickHouse#47636 and kinda fixed several times in different part with these pull request ClickHouse/ClickHouse#55976 and ClickHouse/ClickHouse#36254

maybe @hubertdeng123 or @aldy505 got any idea on this one? The one that crossing my mind is updating the clickhouse to v23 lts or even better to v24 lts

[odbod-glitch]

@bijancot
thx for answer.
Do you think a later version of ClickHouse might help?

[aldy505]

Are you running your own dedicated ClickHouse instance, by any chance?

Snuba is only tested for some very specific ClickHouse version, see here: https://getsentry.github.io/snuba/clickhouse/supported_versions.html, I don't really recommend having an instance that's way off those versions.

It's really recommended to just use this specific version instead of upgrading the instance yourself.

https://github.com/getsentry/self-hosted/blob/bdf8d3ff918d92b04d10906f3869900ca41ec3bf/docker-compose.yml#L178

[odbod-glitch]

@aldy505
In our installation, only the server with PostgreSQL is separated. The same version of ClickHouse remains in the Docker Compose as it was for 24.8.0:
BASE_IMAGE: "altinity/clickhouse-server:23.8.11.29.altinitystable"
Unfortunately, I am encountering the error shown in the log above.

Now I’m confused. If Snuba only supports versions 20.3, 20.7, and 21.8 (https://getsentry.github.io/snuba/clickhouse/supported_versions.html), why are we trying to upgrade to higher versions? Maybe I should just stay on version 21.8.13.1.altinitystable and comment out these upgrade lines in the install/upgrade-clickhouse.sh script?

if [[ "$version" == "21.8.13.1.altinitystable" || "$version" == "21.8.12.29.altinitydev.arm" ]]; then
$dc down clickhouse
$dcb --build-arg BASE_IMAGE=altinity/clickhouse-server:22.8.15.25.altinitystable clickhouse
$dc up -d clickhouse
wait_for_clickhouse
$dc down clickhouse
$dcb --build-arg BASE_IMAGE=altinity/clickhouse-server:23.3.19.33.altinitystable clickhouse
$dc up -d clickhouse
wait_for_clickhouse

[bc-sentry]

Assigning to getsentry/snuba for product area triage.