Improving accuracy - use of text context for history

[wenshutang]

First of all, thank you for the outstanding work on the project. The transparency and willingness to share it with the community are truly commendable.

As I delve into basic evaluations of guiding tasks, I’ve noticed a recurring issue with CogAgent: it occasionally provides inaccurate responses. Sometimes this manifests in becoming unaware of previously recommended and completed actions. This results in providing redundant guides and being stuck, unable to proceed.

Would summarizing previously completed actions and rolling that into the text context improve on accuracy? In other words, condensing interactions into a specific format and integrating them into the history.

Initial attempts with a few examples haven’t yielded any improvement. I wanted to confirm whether history context was included during training. In the paper Mind2Web dataset was converted to natural language with previous actions. Was this specific text format used during training?