Improve code review prompts and add prompt testing tools

[kratos06]

1. Update code review prompts to better identify effective code changes
2. Add detailed working hours estimation guidelines
3. Create test_prompt.py for testing individual diff files
4. Create batch_test_prompts.py for batch testing from GitLab
5. Create test_codedog_prompt.py for testing with CodeDog prompts
6. Add test cases with high and low score examples
7. Add documentation for prompt testing tools

Summary by CodeRabbit

• New Features
  • Introduced orchestrated code review, enabling collaboration of specialized agents for security, performance, readability, architecture, and documentation reviews.
  • Added configuration option to enable or disable orchestrated code review via environment variable.
• Enhancements
  • Improved code review workflow with a factory-based approach for selecting traditional or orchestrated review modes.
  • Expanded support for comprehensive, structured code review reports with detailed scoring and recommendations.
• Documentation
  • Updated environment variable descriptions and section headers for clarity and consistency.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

This update introduces a comprehensive orchestration framework for code review within the CodeDog project. It adds a new environment variable (USE_ORCHESTRATION) to enable orchestrated code reviews using specialized agents. New modules implement a multi-agent system, including a factory for creating either standard or orchestrated code review chains, a coordinator to manage workflow, and specialized agents for security, performance, readability, architecture, and documentation reviews. The main script (run_codedog.py) is updated to use the factory and select the appropriate review chain based on configuration. The .env.sample file and some comments are also updated for clarity and configuration support.

Changes

File(s)	Change Summary
.env.sample	Added "Orchestration Configuration" section with USE_ORCHESTRATION variable; updated section headers and log level description to English.
codedog/chains/__init__.py	Exported OrchestratedCodeReviewChain and CodeReviewChainFactory for public use.
codedog/chains/code_review/factory.py	Introduced CodeReviewChainFactory with a static method to instantiate either standard or orchestrated code review chains based on configuration and provided models.
codedog/chains/code_review/orchestrated.py	Added OrchestratedCodeReviewChain class for orchestrated, asynchronous code review using multiple specialized agents coordinated by an orchestrator.
codedog/orchestration/__init__.py	New module exporting orchestration components: coordinator, orchestrator, and specialized review agents.
codedog/orchestration/agents.py	Added specialized agent classes (SecurityReviewAgent, PerformanceReviewAgent, ReadabilityReviewAgent, ArchitectureReviewAgent, DocumentationReviewAgent) for targeted code review, each with structured prompts and async review methods.
codedog/orchestration/base_agent.py	Added abstract BaseReviewAgent with async review interface, prompt generation, and parsing methods for extracting scores and estimated hours from review text.
codedog/orchestration/coordinator.py	Added ReviewCoordinator to analyze context, distribute review tasks to agents, integrate results, and generate formatted code review reports asynchronously.
codedog/orchestration/orchestrator.py	Added CodeReviewOrchestrator to manage multi-agent code review, coordinate specialized agents, handle error logging, and aggregate results for each file or batch.
run_codedog.py	Updated to use CodeReviewChainFactory for conditional instantiation of standard or orchestrated code review chains; added print statements indicating review mode; updated GitLab commit and diff retrieval to use get_all=True.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant run_codedog.py
    participant CodeReviewChainFactory
    participant CodeReviewChain
    participant OrchestratedCodeReviewChain
    participant CodeReviewOrchestrator
    participant ReviewCoordinator
    participant SpecializedAgents

    User->>run_codedog.py: Trigger code review (with USE_ORCHESTRATION)
    run_codedog.py->>CodeReviewChainFactory: create_chain(llm, use_orchestration, models)
    alt use_orchestration == false
        CodeReviewChainFactory->>CodeReviewChain: from_llm(llm)
        CodeReviewChain-->>run_codedog.py: Chain instance
    else use_orchestration == true
        CodeReviewChainFactory->>OrchestratedCodeReviewChain: from_llms(models)
        OrchestratedCodeReviewChain-->>run_codedog.py: Orchestrated chain instance
    end
    run_codedog.py->>OrchestratedCodeReviewChain: acall(inputs)
    OrchestratedCodeReviewChain->>CodeReviewOrchestrator: review_files(change_files)
    CodeReviewOrchestrator->>ReviewCoordinator: analyze_context(code_diff)
    CodeReviewOrchestrator->>ReviewCoordinator: distribute_work(code_diff, context, agents)
    ReviewCoordinator->>SpecializedAgents: review(code_diff, context)
    SpecializedAgents-->>ReviewCoordinator: review results
    ReviewCoordinator->>ReviewCoordinator: integrate_reviews(reviews)
    ReviewCoordinator->>ReviewCoordinator: generate_report(integrated_review, context)
    ReviewCoordinator-->>CodeReviewOrchestrator: CodeReview
    CodeReviewOrchestrator-->>OrchestratedCodeReviewChain: CodeReview(s)
    OrchestratedCodeReviewChain-->>run_codedog.py: Aggregated code reviews
    run_codedog.py-->>User: Output/report

Loading

Possibly related PRs

• Main #6: Introduced the initial .env.sample file. This PR is related as the current changes extend the environment configuration by adding the orchestration variable and supporting code.

Poem

In burrows deep, the agents meet,
To scan your code with nimble feet.
Orchestrated paws review each line,
Security, docs, and style combine!
With every hop, the feedback grows—
A warren of wisdom, as CodeDog knows.
🐇✨ Let’s code and review, together in prose!

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4ca489c and fa4296d.

📒 Files selected for processing (11)

• .env.sample (1 hunks)
• codedog/chains/__init__.py (1 hunks)
• codedog/chains/code_review/factory.py (1 hunks)
• codedog/chains/code_review/orchestrated.py (1 hunks)
• codedog/orchestration/__init__.py (1 hunks)
• codedog/orchestration/agents.py (1 hunks)
• codedog/orchestration/base_agent.py (1 hunks)
• codedog/orchestration/coordinator.py (1 hunks)
• codedog/orchestration/orchestrator.py (1 hunks)
• example.diff (1 hunks)
• run_codedog.py (6 hunks)

✨ Finishing Touches

• 📝 Generate Docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
codedog
analyze_code.py	19	19	0%	6–74
localization.py	17	2	88%	17, 30
codedog/actors/reporters
code_review.py	107	43	60%	48–92, 96–111, 116, 144, 146, 148, 150, 157
codedog/chains/code_review
base.py	64	31	52%	34, 42, 50, 57–72, 79–94, 100–111, 114–117, 122–125, 135
factory.py	14	7	50%	40–54
orchestrated.py	41	14	66%	44, 52, 60, 71, 83–92, 113–120
translate_code_review_chain.py	48	48	0%	1–103
codedog/chains/pr_summary
base.py	82	20	76%	54, 62, 70, 96–116, 133–136, 178–180, 194–201
translate_pr_summary_chain.py	60	35	42%	40–49, 61–69, 77–85, 91–96, 101–117, 120–130, 135–151
codedog/orchestration
agents.py	74	55	26%	22–51, 55–119, 136–165, 169–233, 250–279, 283–347, 364–393, 397–461, 478–507, 511–575
base_agent.py	49	35	29%	32–33, 48, 61–76, 88–124, 136–150
coordinator.py	125	106	15%	32, 44–65, 79–91, 103–118, 131–145, 149–165, 169–188, 193–201, 205–251, 255–272, 277–292, 296–326
orchestrator.py	40	28	30%	41–44, 62–93, 105–127
codedog/retrievers
github_retriever.py	100	41	59%	79, 87, 94–95, 98–99, 102, 111–114, 130–134, 150–151, 154, 163–167, 172–179, 197, 202, 205–209, 217–222, 225–228, 231, 242, 252
gitlab_retriever.py	119	76	36%	43–56, 62, 66, 70, 74, 78, 81–85, 88–92, 95, 104, 116, 124–141, 146–149, 155–163, 166–176, 179–203, 206–212, 216, 219–226, 234, 237–240, 243, 254–255, 258
codedog/utils
code_evaluator.py	1382	1303	6%	27–48, 90–113, 130–138, 143–197, 201–237, 241–268, 290–298, 311–338, 355–472, 517, 525–573, 587–657, 662–890, 894–1203, 1207–1228, 1241–1260, 1271–1371, 1382–1409, 1421–1573, 1585–1775, 1789–2091, 2102–2142, 2165–2490, 2510–2803, 2820–2996, 3015–3228, 3246–3247, 3263–3354, 3358–3403, 3428–3566
email_utils.py	70	59	16%	31–49, 71–113, 134–158
git_hooks.py	45	45	0%	1–146
git_log_analyzer.py	170	146	14%	41–114, 133–172, 185–216, 229–262, 291–316, 329–345, 370–448
langchain_utils.py	237	186	22%	25–41, 64, 68, 78–152, 162–305, 315, 322–343, 350–371, 378–399, 405–416, 422–433, 455–485
remote_repository_analyzer.py	105	105	0%	1–248
TOTAL	3421	2404	30%

Tests	Skipped	Failures	Errors	Time
46	1 💤	1 ❌	1 🔥	4.160s ⏱️