Question about training

[FYNIXqwq]

As the readme of this repository indicates, plentiful game log is necessary in the preparation of training. Why this model does not randomly shuffle and generate the deck (ピーパイとワンパイJapanese), then generate 4 agents as players?

[Cryolite]

Are you asking, "Why doesn't this repository use training based on the game records obtained from self-play?"

[FYNIXqwq]

Yes. I am confusing that why multiple agents self-play with reward is not enough. Card game models of imperfect information game i.e. DouZero is a complete self-play model, and it works well.

[Cryolite]

I see your point. It is very reasonable to have AIs of a certain strength fight against each other and then use those game records to advance further learning processes. This is a standard method in reinforcement learning. I plan to implement this traditional reinforcement learning method using self-play in the future.

However, in recent years, the technique known as offline reinforcement learning, which doesn't require self-play and uses only existing game records, has been rapidly advancing. The advantage of offline reinforcement learning is that it doesn't require self-play, which tends to limit the learning speed in traditional reinforcement learning, therefore, learning can proceed at a much faster rate. One of the intermediate goals of this project is to determine how strong mahjong AIs can become through offline reinforcement learning.

This is the reason why the current repository does not have a learning function through self-play.

[FYNIXqwq]

Thank you for giving this detailed explanation for me and for this repository. However, will Offline RL becoming over-fitting the human action in the "cited example" more exactly, crawled logs (牌譜 in Mahjong Hanchans)?

[Cryolite]

Classical over-fitting is a problem in supervised learning (behavioral cloning) and is unlikely to be an issue in offline reinforcement learning. However, unique problems called distributional shift and unlearning effect may arise in offline reinforcement learning (cf. https://arxiv.org/abs/2005.01643). These issues stem from overestimating Q-values for actions that do not appear in training examples. The IQL implemented in this repository is a method to mitigate these problems.