Abstract

Large Language Models (LLMs), which are trained on massive text data, have demonstrated remarkable advancements in language understanding capabilities. Nevertheless, it is still an open question whether these LLMs have mastered the knowledge relationships they have been exposed to and to what extent. In this study, we concretize this abstract issue and define a new perspective of 'Understanding Self-Consistency', manifesting its mastery of knowledge relationships through the LLM's self-consistency performance. 'Understanding Self-Consistency' refers to the consistency in expressions among LLMs' inputs and responses. Inspired by human cognitive behavior, we design a self-check action framework named $S^{2}AF$. Wherein, a self-question and answering mechanism is emphasized and forms a logically closed loop including four classes of actions, allowing our $S^{2}AF$ to generate, question, answer, and evaluate autonomously. Experimental results on six LLMs across two logical relationship datasets show that LLMs exhibit objective ability values of the understanding self-consistency and demonstrate their differentiated mastery of knowledge relationships across different reasoning paradigms. Moreover, our findings reveal that LLMs' performance can be improved with their own outputs (which we call 'self-enhanced Feedforward'). Notably, $S^{2}AF$ merely relies on factual logical relationships, this ability can effectively advance the development of embodied artificial intelligence.

Figure