Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Pd init #47

Open
wants to merge 17 commits into
base: unstable
Choose a base branch
from
Open

feat: Pd init #47

wants to merge 17 commits into from

Conversation

panlei-coder
Copy link
Collaborator

@panlei-coder panlei-coder commented Sep 21, 2024

1、进展:完成了 multi-raft 的改造,并将 PD 模块搭建起来了,目前已经完成了 store 上线的注册功能。
2、后续安排:
(1)添加测试,目前只是简单验证了正确性。
(2)添加根据 key 的范围选择 region 的逻辑,region 对 client 透明。
(3)把 join 命令进一步完善,添加获取 region id 的逻辑。
3、测试:
(1)配置信息设置:
pd 节点:默认端口为 7777,pikiwidb.conf 中 as-pd 设置为 yes。
image
普通节点:默认端口为 8888,pikiwidb.conf 中 as-pd 设置为 no。
image
这里不管是 pd 节点还是普通节点都需要设置 pd 的成员配置信息:
1724550616692
如果想要构建 pd 三节点的话,可以自己适当的增加成员配置。
(2)启动 pd 节点:
image
启动 pd 节点,默认会自己创建 region 0(即 db 0),用来存储集群的元数据。
(3)启动普通节点:
1724551304319
(4)查询注册信息:
image
4、设计文档:
https://m0jlcucyhxm.feishu.cn/docx/F2qMd0eRUotvrdxAHBdcpx5yn5Q

Summary by CodeRabbit

  • New Features

    • Introduced a new PlacementDriverService with RPC methods for managing clusters, including adding/removing stores and retrieving cluster information.
    • Added support for Protocol Buffers to enhance communication between components.
    • Implemented a singleton PlacementDriverServer to manage placement driver operations.
  • Configuration Changes

    • Updated configuration parameters for databases, including a reduction in the number of databases and new settings for using the Raft consensus algorithm.
  • Bug Fixes

    • Improved error handling and logging across various components to enhance robustness.
  • Documentation

    • Enhanced documentation for various services and methods to improve clarity and usability for developers.

Copy link

coderabbitai bot commented Sep 21, 2024

Walkthrough

The changes encompass the integration of Protocol Buffers into the build configuration of the pikiwidb project, enhancements to the command execution logic, and the introduction of a new Placement Driver service. Key modifications include updates to CMake files for building new libraries, adjustments in configuration files, and the implementation of various RPC methods for managing cluster operations. Additionally, the PRaft class has been refactored to improve node management and command handling, along with the addition of new methods for managing backend databases and statistics.

Changes

Files Change Summary
CMakeLists.txt, src/pd/CMakeLists.txt, src/praft/CMakeLists.txt Added new subdirectories and libraries for Protocol Buffers integration, including custom commands for generating source files from .proto definitions.
pikiwidb.conf Updated configuration parameters for database connections, enabling Raft consensus, and introducing new settings for the placement driver.
src/base_cmd.cc, src/base_cmd.h Enhanced command execution logic with additional checks for database existence and PRAFT initialization; minor formatting changes in method signatures.
src/client.cc, src/client.h Modified PClient interactions with PRAFT to utilize backend-specific instances; added a new constant for channel timeout.
src/cmd_admin.cc, src/cmd_admin.h Adjusted command flags for SelectCmd and improved encapsulation in InfoCmd by using local references to PRAFT.
src/cmd_raft.cc, src/cmd_raft.h Introduced group ID handling in RaftNodeCmd, updated command execution logic, and integrated new RPC mechanisms for node management.
src/config.cc, src/config.h Added new parameters related to the placement driver in the configuration class, including boolean and string types.
src/db.cc, src/db.h Refactored the DB class to use an ID-based approach instead of an index-based one, along with the integration of a PRaft instance for better management.
src/pd/*.proto, src/pd/pd_server.cc, src/pd/pd_service.cc Introduced new Protocol Buffers definitions and implemented the PlacementDriverService for managing clusters, including methods for adding/removing stores and regions.
src/praft/*.proto, src/praft/praft_service.cc Updated RPC service definitions for adding and removing nodes in the Raft consensus algorithm, replacing previous dummy methods with functional implementations.
src/store.cc, src/store.h, src/store.proto Enhanced the PStore class with new methods for backend management and RPC server initialization, along with the introduction of a new service for store and region statistics.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant PlacementDriverServer
    participant Database
    participant Raft

    Client->>PlacementDriverServer: Add Store Request
    PlacementDriverServer->>Database: Check Store Existence
    alt Store Exists
        PlacementDriverServer->>Raft: Add Node
        Raft-->>PlacementDriverServer: Success
    else Store Does Not Exist
        PlacementDriverServer-->>Client: Error
    end
    PlacementDriverServer-->>Client: Store Added Successfully
Loading

🐇 "In the garden where bunnies play,
New features hop in, brightening the day.
With Protocol Buffers, we build and grow,
A Placement Driver to help us flow.
Commands refined, like a well-tuned flute,
Hooray for changes, so cute and astute!" 🐇


Note

Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    -- I pushed a fix in commit <commit_id>, please review it.
    -- Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    -- @coderabbitai generate unit testing code for this file.
    -- @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    -- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    -- @coderabbitai read src/utils.ts and generate unit testing code.
    -- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    -- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

};

message GetClusterInfoResponse {
bool success = 1;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果是使用status可能可以描述更多失败的场景。

: fake_(fake), pd_group_id_(std::move(pd_group_id)), initial_pd_server_list_(std::move(initial_pd_server_list)) {}
~PlacementDriverOptions() = default;

bool GetFake(bool fake) { return fake_; }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个bool fake参数好像没有用。

return {false, -1};
}

max_region_id += 1;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果只是+1,写成++就可以。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants