Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] - Introduce a "Manager" agent for task evaluation and feedback #154

Open
dannleft opened this issue Nov 18, 2024 · 0 comments
Labels
feature New feature or request feedback discord

Comments

@dannleft
Copy link
Collaborator

Description

Is your feature request related to a problem? Please describe.
Currently, there is no centralized mechanism within KaibanJS to evaluate and provide feedback on agents' outputs or the final results of their collective tasks. This makes it challenging to ensure logical correctness (e.g., scheduling conflicts) and meet user preferences (e.g., avoiding certain conditions like heat or snow) in complex workflows.

Describe the solution you'd like
Introduce a "Manager" agent or framework that can:

  1. Evaluate: Assess both individual agents' outputs and the final results of their combined tasks based on two criteria:
    • Logical correctness: Ensure outputs meet predefined rules (e.g., no scheduling conflicts, task feasibility).
    • Preferences: Allow personalized criteria (e.g., user dislikes certain weather conditions).
  2. Feedback Mechanism: Automatically provide feedback to agents, enabling them to refine their work in subsequent iterations.
  3. Cost Optimization: Enable the use of smaller, narrow-scope models for agents, while the "Manager" leverages larger reasoning models (e.g., self-hosted or local models) for evaluation and feedback tasks.

Describe alternatives you've considered

  • Using external validation systems to evaluate agent outputs and final results, which increases the complexity of integrations.
  • Adding manual checkpoints in the workflow to review outputs, which is time-consuming and defeats the purpose of automation.

Additional context
This feature would enable:

  • Automated quality control for agent workflows, ensuring logical and preference-based correctness.
  • Advanced use cases such as trip planning, collaborative projects, and multi-agent simulations.
  • Cost optimization by balancing smaller, task-specific models with larger reasoning models for evaluation.
@dannleft dannleft added feature New feature or request feedback discord labels Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request feedback discord
Projects
None yet
Development

No branches or pull requests

1 participant