Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to 0.24 without downtime? #1550

Open
DaanDD opened this issue Oct 30, 2024 · 3 comments
Open

Upgrade to 0.24 without downtime? #1550

DaanDD opened this issue Oct 30, 2024 · 3 comments

Comments

@DaanDD
Copy link

DaanDD commented Oct 30, 2024

When looking at the instructions on how to upgrade to 0.24 it involves deleting the old CHK installation and creating a new one. The ClickHouse instance I'm running is constantly being written to so I cannot just turn off Keeper. Is there a way to upgrade to 0.24 without downtime?
Maybe I could create a new Keeper installation that syncs data with the existing one, then hot swap between old and new Keeper?

@UnamedRus
Copy link

Hi,

We are currently in process of writing procedure to migrate CHK from old to new syntax.

Maybe I could create a new Keeper installation that syncs data with the existing one, then hot swap between old and new Keeper?

Yeah, it's basically how it will work:

  1. Detach old CHK from operator management.
  2. Upgrade Operator + CRDs
  3. Create new CHK, which will also include old nodes in it's files.xml configuration.
  4. Update config for old keeper installation (ie include new nodes)
  5. Check sync
  6. Switch CH to new keeper ensemble.
  7. Remove old nodes from ensemble
  8. Drop old nodes.

@zeev1079
Copy link

zeev1079 commented Nov 7, 2024

have a couple of questions regarding this migration process(which looks much straight forward):

  1. What happens if the old Keeper nodes have different volume mount paths, such as:
keeper_server/log_storage_path: /var/lib/clickhouse-keeper/logs 
keeper_server/snapshot_storage_path: /var/lib/clickhouse-keeper/snapshots
keeper_server/storage_path: /var/lib/clickhouse-keeper
  1. Is adding the old Keeper nodes in the files.xml configuration the correct way to include them in the new installation? For example:
  configuration:
    files:
      files.xml: |
        <yandex>
          <zookeeper>
            <node>
              <host>old-keeper-node-1</host>
              <port>2181</port>
            </node>
            <node>
              <host>old-keeper-node-2</host>
              <port>2181</port>
            </node>
            <node>
              <host>old-keeper-node-3</host>
              <port>2181</port>
            </node>
            <node>
              <host>new-keeper-node-1</host>
              <port>2181</port>
            </node>
            <node>
              <host>new-keeper-node-2</host>
              <port>2181</port>
            </node>
            <node>
              <host>new-keeper-node-3</host>
              <port>2181</port>
            </node>
          </zookeeper>
        </yandex>```

@clement94310
Copy link

hello
i read the upgrade doc :
https://github.com/Altinity/clickhouse-operator/blob/0.24.0/docs/keeper_migration_from_23_to_24.md

in point 7
how i have to modify the pvc if i have 3 servers ?
volumeClaimTemplates:
- name: default
spec:
...
volumeName: $PV

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants