Prepare for defeat against Transformers on Lichess chess platform
A new study has delved into the world of chess, exploring the use of large transformer models to play the game without memorization or explicit search. The research, which you can discuss further on our Discord server, has some intriguing findings.
The study utilises the ChessBench dataset, a collection of 10 million annotated chess games by Stockfish, a leading chess engine. The transformer models were then tested for Action-Value Prediction, State-Value Prediction, and Behavioural Cloning.
Despite their impressive size, with up to 270 million parameters, the transformer models achieved an Elo rating close to that of grandmasters (Lichess blitz Elo of 2895). However, when it comes to directly predicting chess moves, they performed relatively poorly compared to biologically inspired neural architectures.
For instance, lightweight graph neural network models based on insect brain connectomes, trained on far fewer games (10,000), achieved about 60% move accuracy, which is nearly 10 times better than any size transformer trained on the same dataset.
When compared to traditional chess engines that rely on explicit search algorithms, such as minimax with pruning, the bio-inspired neural architectures, even with moderate search depths (depth-6 minimax), showed very high accuracy (91.7%), surpassing a 9-million parameter Transformer baseline.
In contrast, the pure transformer-based models without search or memorization mechanisms lagged behind both biofidelic models and classic engines.
Here's a summary of the findings:
| Model Type | Training/Parameters | Move Accuracy on ChessBench | Search Used | Comparison Notes | |---------------------------|---------------------------|-----------------------------|-------------|-----------------------------------------| | Large Transformers | Trained on 10 million games| Poor; <10% relative accuracy| No | Much less accurate than biofidelic models| | Biofidelic GNN-BPU (insect connectome) | Trained on 10,000 games | ~60% | No/low depth | Nearly 10x better than transformers | | CNN-BPU + minimax search | ~2 million params with search depth 6| 91.7% | Yes | Surpasses large transformers without search |
In conclusion, large transformers trained on ChessBench do not match traditional chess engine strength or strategic understanding without explicit search or memorization mechanisms, while bio-inspired architectures provide a more efficient and accurate approach to learned chess playing by better capturing structural inductive biases.
As the technology matures, transformers might be applied in various complex, real-world scenarios, from logistics to robotics. However, for now, it seems that biofidelic models offer a promising path for developing efficient and accurate chess-playing AI.
[1] Source: [Paper Title] by [Authors' Names]
The study reveals that large transformer models, despite their significant size, struggle to match the accuracy and strategic understanding of traditional chess engines when playing chess without explicit search or memorization mechanisms. On the other hand, bio-inspired models, such as graph neural networks based on insect brain connectomes, offer a more efficient and accurate approach to learning chess, showcasing a higher move accuracy on the ChessBench dataset.
