-
Notifications
You must be signed in to change notification settings - Fork 213
Provide a base class of trajectory - so Evaluation is independent #205
Copy link
Copy link
Open
Description
@ShuxinLin Note sure you adjusted uniformly everywhere, including the documentation. I do see "Agent" yet in the .md file.
There is also one ask: a base class called Trajectory, which can be nested to capture the multiple levels of detail that will be useful for the downstream Evaluation class to remain independent. Trajectories are of different types (ReAct, ReActXen, Plan-Execute, etc.), making it easy to compare at the Step level, especially in our Evaluation based on Reference-based Scoring.
https://github.com/IBM/ReActXen/blob/main/src/reactxen/agents/react/agents.py#L880
self.json_log.append( { "step": self.step_n, "raw_llm_thought_output": "", "raw_llm_action_output": "", "raw_observation_output": "", "raw_llm_output": "", "thought": "", "action": "", "action_input": "", "observation": "", "state": "", # invalid_action, invalid_thought, etc # "include_in_extraction": False, "is_loop_detected": None, "additional_scratchpad_feedback": "", "step_trajectory_file_name": None, "step_metric_file_name": None, "step_trajectory_json": None, "step_metric_json": None, "llm_error": False, "llm_error_detail": False, }Any thoguht?
Originally posted by @DhavalRepo18 in #170
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels