Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Update / 2024.04.30] Multi Agent Architecture Optimization & Future Plan #5

Closed
2 of 12 tasks
MIRACLE-cowf opened this issue Apr 16, 2024 · 3 comments
Closed
2 of 12 tasks

[Update / 2024.04.30] Multi Agent Architecture Optimization & Future Plan #5

MIRACLE-cowf opened this issue Apr 16, 2024 · 3 comments
Assignees
@MIRACLE-cowf
Labels
documentation Improvements or additions to documentation enhancement New feature or request idea New idea or feature task What should I/You do update log The update log of this project

Comments

@MIRACLE-cowf
Copy link
Owner

MIRACLE-cowf commented Apr 16, 2024 •
edited
Loading

OverView

  • You can check (641e47b) this commit for detailed code.
  • Applying and advancing the multi-agent architecture mentioned in [Idea / 2024.04.14] Testing Multi-agent with supervisor architecture #4.
  • Detailed flow of the PAR multi-agent version will be upload soon.
  • Some prompts have been modified.
  • As the MultiAgent architecture is advanced, the main function no longer parses the last results of each section.

Current Situation

  • The actual code applying this architecture can be found and tested in the AgentTeam branch.(641e47b)
  • It's not unstable, but the code is not yet clean.
  • Prompts for each search engine-specific agent, document creation agent, and PM agent have been written.
  • Each search agent and document creation agent are still implemented as a single LangGraph. [Idea / 2024.04.14] Testing Multi-agent with supervisor architecture #4
  • Previously, each LLM summarized results for each search engine in parallel. Now, the results from each search engine are integrated, and a single LLM performs the overall summary and extracts content. [Idea / 2024.04.14] Testing Multi-agent with supervisor architecture #4
    • This was started with the intention of somewhat alleviating the rate limit hit (TPM), but it is my personal judgment that the performance is better than before.

Future Plans

  • The checkboxes mentioned in the Future Plans of [Idea / 2024.04.14] Testing Multi-agent with supervisor architecture #4 will be carried out one by one.
  • Overall flow improvements are expected.
  • Plan to insert images into the generated document after analyzing them using multi-modal based on Claude 3.
  • Add GitHub search functionality
  • Add Stack Overflow search functionality
  • Plan to introduce quick search using SerpAPI
  • Handle fallbacks and exceptions
  • Estimate the final token count
  • Adjust the project to allow testing even without MongoDB and PineCone Vector DB APIs

Flow Improvement

  • Currently, when a user asks a question, a multi-query is generated, and if there are no results after searching the vector DB, it directly enters the THLO stage. (Refer to README)
  • This causes significant time and token waste.
  • Therefore, we are trying to advance the criteria for entering the THLO stage.

If there are no search results

  • Perform a quick web search (Tavily + Haiku / Command-R+) for the question and provide an answer immediately.
  • Simultaneously, determine if the question requires an advanced search.
  • Simultaneously, enter the THLO stage.
  • In other words, if there are no search results, the intention is to perform a quick web search and answer the question while giving the user a choice. The expected scenario is as follows.

Expected Scenario

  • This step divides the THLO stage, which requires web searching when there are no search results in the vector DB, in the overall flow.
  1. Perform a quick web search using Tavily + Haiku / Command-R+ (the model has not been selected yet, but it is highly likely that these two models will be used). (Expected time: within 15~30 seconds)
  2. Simultaneously with 1, use LLM to determine if the user's question requires an advanced search (THLO) or a quick search. (Expected time: within 10~15 seconds)
  3. Simultaneously with 1, enter the THLO stage.
  4. After the result of 2 is obtained, wait briefly. When the search result and response from 1 are available, present the response along with the result of 2.
    • In other words, the user is informed of the quick search result while also being presented with the importance of the question as determined by the LLM, giving the user a choice.
    • If the question requires an advanced search, provide a response like "This question seems to require an advanced search. Should we proceed?" along with the search result from 1.
    • If the question does not require an advanced search, provide a response like "The answer to this question is as follows: {result}. Do you need a more detailed (advanced) search?"
  5. If an advanced search is conducted, it is expected to save time since it is already being performed simultaneously with 1 in step 3.
  6. If an advanced search is not needed, terminate before performing the full multi-agent search (some token consumption is expected, but additional measures seem necessary).
@MIRACLE-cowf MIRACLE-cowf mentioned this issue Apr 16, 2024
Open
@MIRACLE-cowf MIRACLE-cowf self-assigned this Apr 16, 2024
@MIRACLE-cowf MIRACLE-cowf added documentation Improvements or additions to documentation enhancement New feature or request update log The update log of this project task What should I/You do idea New idea or feature labels Apr 16, 2024
@MIRACLE-cowf MIRACLE-cowf pinned this issue Apr 16, 2024
MIRACLE-cowf added a commit that referenced this issue Apr 25, 2024
@MIRACLE-cowf
"Implement custom error handling and query fallbacks"
5b90fab 
- Added custom error handling by raising 'PAR_ERROR' in 'Custom_TavilySearchResults' instead of returning the exception
- Added query fallback logic in 'CustomSearchFunc_v2' to retry execution when an error occurs
- Modified the 'Custom_Error_Handler' to explicitly require a string as a parameter for the error message.

Ref: #5
Signed-off-by: MIRACLE <[email protected]>
@MIRACLE-cowf
Copy link
Owner Author

MIRACLE-cowf commented Apr 25, 2024 •
edited
Loading

2024.04.25

  • Add tavily search tool with fallbacks
    • Sometimes, Tavily Search Tool occur HTTPError with string, so the agent can't work expected, so adding fallbacks first!

MIRACLE-cowf added a commit that referenced this issue Apr 25, 2024
@MIRACLE-cowf
"Implemented agent outcome checker utility and enhanced search functi…
306a9ef 
…onality"

- Created a new utility, `agent_outcome_checker`, that simplifies consistent checking of agent outcomes across different agent scripts.
- Improved the `TavilyInput` model in `Custom_TavilySearchResults.py` by adding a `max_results` parameter so that the agent can control the amount of information returned by the search tool.
- Refactored agent scripts to use the new `agent_outcome_checker` utility, reducing redundancy and improving code maintenance.
- Expanded on the descriptions of the `arXiv` and `generate_search_query_plans` tools in `create_agent.py` and `THLO_Graph.py` respectively, to provide a more detailed understanding.
- Update prompt int LangSmith named 'par_generate_search_query_prompt_public', 'par_high_level_outline_prompt_public'
- Enhanced the `web_search_v2` function in `CustomSearchFunc_v2.py` to receive and process a `max_results` parameter.
- Amended instructions in `Project_Manager_Agent.py` to be more specific, clear, and detailed.

Ref: #5

Written by my own LLM

Signed-off-by: MIRACLE <[email protected]>
MIRACLE-cowf added a commit that referenced this issue Apr 30, 2024
@MIRACLE-cowf
Update components across project to enhance functionality
479f19b 
- Implemented a retry mechanism with delay in the helper module to handle exceptions and retry function execution.
- Restructured agents and improved code quality by adding types and descriptive print statements.
- Updated search tools' descriptions for clarity and added 'max_results' parameter for the arXiv search tool.
- Refactored the THLO graph for clarity, removed unnecessary fallback chains and improved output messages.
- Simplified retrieval and grading operations in the main application entrypoint, and made transition conditions more transparent for better readability and maintainability.

Ref: #5
Signed-off-by: MIRACLE <[email protected]>
@MIRACLE-cowf MIRACLE-cowf changed the title [Update / 2024.04.16] Multi Agent Architecture Optimization & Future Plan [Update / 2024.04.30] Multi Agent Architecture Optimization & Future Plan Apr 30, 2024
@MIRACLE-cowf
Copy link
Owner Author

2024.04.30

  • Now, in the THLO stage, the 'conclusion' section is written based on the document generated from the search results of other sections, excluding the 'conclusion' itself.
  • The PM agent's model has been switched to 'sonnet' in an attempt to prevent rate limit hits (TPM) for other search-specialized agents.
  • Nevertheless, in case of rate limit hits, a retry mechanism has been developed and is currently being validated.
  • For search-specialized agents (Tavily, ArXiv), a 'max_results' parameter has been added to allow the agent to set the maximum number of search results on its own.

MIRACLE-cowf added a commit that referenced this issue May 8, 2024
@MIRACLE-cowf
Update error handling, import RunnableSerializable and refactor agents
edb61e4 
- Implemented a check for PDF files and bypass them during data processing.
- Imported RunnableSerializable class from langchain_core.runnables and refactored processing chain to use it.
- Errors during Tavily API calls are now handled gracefully, providing more informative error details.
- Refactored agent creation and management for better readability and performance.
- Additional minor changes include formatting adjustments and updates in comments.

Ref: #5
Signed-off-by: MIRACLE <[email protected]>
@MIRACLE-cowf
Copy link
Owner Author

2024.05.17

33e05c9

  • Adding Brave Search
  • Code refactoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MIRACLE-cowf MIRACLE-cowf

Labels
documentation Improvements or additions to documentation enhancement New feature or request idea New idea or feature task What should I/You do update log The update log of this project
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@MIRACLE-cowf