3 months, 1 week ago
IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty
The aim of the competition is to predict the difficulty of chess puzzles based on board configurations and moves that the solution to each puzzle consists of. The difficulty level is measured as the rating on the lichess platform. The top 3 solutions will be awarded prizes.
IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty is the sixth data science competition organized in association with the IEEE International Conference on Big Data series (IEEE BigData 2024, https://www3.cs.stonybrook.edu/~ieeebigdata2024/index.html).
See the detailed program of our competition presentations at IEEE BigData 2024 -
https://qedsoftware.com/IEEE_BigData_2024_Chess_and_Granulation.pdf
A chess puzzle is a particular configuration of pieces on a chessboard, where the puzzle taker is instructed to assume the role of one of the players and continue the game from that position. The player has to find from one to several moves, until she delivers mate or obtains a decisive material advantage.
In the online setting, where these are often solved, the puzzle taker only makes moves from one side, while the puzzle publisher provides responses from the other side. One such puzzle solving service is Lichess Training
Solving puzzles is considered one of the primary ways to hone chess skills. However, currently the only way to reliably estimate puzzle difficulty is to present it to a wide variety of chess players and see if they manage to solve it.
The goal of the contest is to predict how difficult a chess puzzle is just by looking at the board setup and the moves in the solution. Puzzle difficulty is measured by its Glicko-2 rating calibrated on the lichess.org website. In simplified terms, it means that lichess models the difficulty of a puzzle by assuming that every attempt at solving a puzzle is a “match”. If a user solves the puzzle correctly, she gains puzzle rating and the puzzle loses rating. The opposite happens when the user doesn’t find the full solution (partial solutions count as “losses”). Both user and puzzle ratings are initialized at 1500. More information about the Glicko rating can be found here.
Each chess puzzle is described by the initial position (using Forsyth–Edwards Notation, or FEN) and the moves included in the puzzle solution, starting with one move leading to the puzzle position and then alternating between the moves that the puzzle solver has to find and those made by the simulated “opponent”.
IEEE Big Data 2024: We will encourage the top 3 winners to submit papers describing their solutions. It is already agreed that the conference will provide the top 3 winners with free registrations. The QED Software’s team, just like in the previous years, intends to organize a workshop devoted to the competition outcomes. According to our experience, the ability to present workshop papers may be an extra incentive for participants to consider active involvement in the competition.
The aim of the competition is to predict the difficulty of chess puzzles based on board configurations and moves that the solution to each puzzle consists of. The difficulty level is measured as the rating on the lichess platform. The top 3 solutions will be awarded prizes. IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty is the sixth data science competition organized in association with the IEEE International Conference on Big Data series (IEEE BigData 2024, https://www3.cs.stonybrook.edu/~ieeebigdata2024/index.html).
See the detailed program of our competition presentations at IEEE BigData 2024 - https://qedsoftware.com/IEEE_BigData_2024_Chess_and_Granulation.pdf
The Competition is Over!
Sincere thanks to all participants and congratulations to the winners!
The top3 teams will receive prizes and free full registration to IEEE Big Data conference.
In addition, selected teams (based on the score and interesting report) will be invited to participate in the special session during IEEE Big Data conference.
The data are provided as two .csv files, one for training dataset and one for testing dataset.
Each row of the testing dataset consists of the following fields:
|
|
|
|
|
|
|
|
|
|
|
|
Based on the above data, the challenge contestants are expected to predict the Rating field (which will be kept secret).
|
|
|
|
The training dataset contains all of the above fields, and also a few additional ones listed below. These fields are sometimes null in the training set and will not be provided for the test set:
RatingDeviation (int): Measure of uncertainty over puzzle’s difficulty.
Popularity (int): Users can ”upvote“ or “downvote” a puzzle. This value is the difference between the number of upvotes and downvotes.
NbPlays (int): Number of attempts at solving the puzzle.
Themes (str): Lichess allows choosing puzzles to solve based on different themes, such as tactical concepts, solution length or puzzle types (e.g. mates in x moves).
GameUrl (str): Lichess puzzles are generated based on games played on lichess.
OpeningTags (str): Information about the opening from which this puzzle originated.
Solution format
Solutions in this competition should be submitted to the online evaluation system as a text file with exactly 2282 lines containing predictions for test instances. Each line in the submission should contain a single integer that indicates the predicted rating of the chess puzzle. The ordering of predictions should be the same as the ordering of the test set.
Evaluation
The quality of submissions will be evaluated using the mean squared error metric.
Solutions will be evaluated online, and the preliminary results will be published on the public leaderboard. The public leaderboard will be available starting May 30th. The preliminary score will be computed on a small subset of the test records, fixed for all participants. The final evaluation will be performed after the completion of the competition using the remaining part of the test records. Those results will also be published online. It is important to note that only teams that submit a report describing their approach before the end of the challenge will qualify for the final evaluation.
There are two data files available to download.
Rank | Team Name | Is Report | Preliminary Score | Final Score | Submissions | |
---|---|---|---|---|---|---|
1 | bread emoji |
True | True | 49141.5359 | 104540.656891 | 47 |
2 | anansch |
True | True | 58810.4586 | 120682.234604 | 38 |
3 | ousou |
True | True | 69890.9227 | 123103.229717 | 50 |
4 | Andryyyyy |
True | True | 61381.3812 | 129245.229228 | 56 |
5 | ToDoFindATeamName |
True | True | 65136.8232 | 132631.424731 | 53 |
6 | alexmolas |
True | True | 74378.0110 | 137839.668622 | 22 |
7 | dymitr |
True | True | 69202.5691 | 141488.501466 | 50 |
8 | Feiwyth |
True | True | 70792.7182 | 146729.234115 | 6 |
9 | NxGTR |
True | True | 73832.3591 | 150757.042522 | 35 |
10 | BigData2024 |
True | True | 74135.4586 | 154905.759042 | 51 |
11 | transformer_enjoyer |
True | True | 75995.0221 | 158292.081623 | 27 |
12 | neuralnite |
True | True | 82049.4972 | 160677.509775 | 6 |
13 | MrAces |
True | True | 78837.4807 | 163552.500000 | 43 |
14 | DML |
True | True | 91728.0055 | 170533.507331 | 133 |
15 | shoggoth |
True | True | 87533.5193 | 171309.505376 | 68 |
16 | AIBrain |
True | True | 91889.9503 | 189554.737537 | 43 |
17 | Marek |
True | True | 119067.6243 | 195828.732649 | 172 |
18 | LcWP |
True | True | 167116.2762 | 244485.950147 | 4 |
19 | JustEngine |
False | True | 67827.4254 | No report file found or report rejected. | 101 |
20 | deep |
False | True | 81429.3204 | No report file found or report rejected. | 4 |
21 | baellouf |
False | True | 82238.9890 | No report file found or report rejected. | 43 |
22 | September |
False | True | 84712.2541 | No report file found or report rejected. | 19 |
23 | Amy |
False | True | 91476.2762 | No report file found or report rejected. | 78 |
24 | JKU-CODA |
False | True | 91664.9890 | No report file found or report rejected. | 20 |
25 | Plats Bruts |
False | True | 98152.9503 | No report file found or report rejected. | 16 |
26 | hieuvq |
False | True | 101897.2818 | No report file found or report rejected. | 10 |
27 | checkmate |
False | True | 101972.7072 | No report file found or report rejected. | 4 |
28 | scotchgame |
False | True | 85906.9503 | No report file found or report rejected. | 5 |
29 | kubapok |
False | True | 120870.0718 | No report file found or report rejected. | 4 |
30 | Fontageau |
False | True | 122314.7127 | No report file found or report rejected. | 2 |
31 | soksly |
False | True | 123260.3646 | No report file found or report rejected. | 9 |
32 | French_bestbytest |
False | True | 131666.9171 | No report file found or report rejected. | 13 |
33 | tafhi |
False | True | 135894.2928 | No report file found or report rejected. | 2 |
34 | fuzz |
False | True | 142167.5028 | No report file found or report rejected. | 3 |
35 | Cavajah |
False | True | 149984.5028 | No report file found or report rejected. | 3 |
36 | Azeezah |
False | True | 169580.3370 | No report file found or report rejected. | 1 |
37 | OrganizerTest |
False | True | 187245.4199 | No report file found or report rejected. | 1 |
- May 08, 2024: start of the competition, datasets become available,
- May 30, 2024: public leaderboard becomes available
- August 31, 2024: deadline for submitting the solutions,
- September 12 (extended), 2024: deadline for sending the reports, end of the competition,
- September 15, 2024: online publication of the final results, sending invitations for submitting papers to the associated workshop at the IEEE Big Data 2024 conference,
- October 13, 2024: deadline for submitting invited papers,
- October 28, 2024: notification of paper acceptance,
- November 17, 2024: camera-ready of accepted papers due.
QED will sponsor the cash prizes:
- 1000 USD for the winning solution
- 500 USD for the 2nd place solution
- 250 USD for the 3rd place solution
Additionally, the IEEE Big Data 2024 conference will provide the top 3 performers with free full registrations.
- Jan Zyśko
- Katarzyna Jagieła
- Maciej Świechowski
- Sebastian Stawicki
- Andrzej Janusz
- Dominik Ślęzak
- Zbigniew Pakleza
Discussion | Author | Replies | Last post | |
---|---|---|---|---|
Paper Submission | Anan | 4 | by Wednesday, September 18, 2024, 14:55:26 |
|
Final Score | 2 | by Wednesday, September 18, 2024, 14:54:17 |
||
Question about final 3 choices | 3 | by Maciej Friday, September 13, 2024, 07:10:34 |
||
Dude, Report submission is working now, please submit your report before the new deadline | M | 0 | by M Wednesday, September 11, 2024, 17:59:04 |
|
Problem sending the report , | chess | 2 | by Wednesday, September 11, 2024, 15:12:59 |
|
could you please open the report submission again for at least one day? as the website was down for the past few days. | M | 5 | by Anan Tuesday, September 10, 2024, 12:30:44 |
|
When exactly is the deadline for submitting solutions? | 8 | by M Monday, September 09, 2024, 09:08:20 |
||
Player initial rating deviation | 1 | by Friday, August 23, 2024, 14:33:17 |
||
Player initial rating deviation | 0 | by Friday, August 23, 2024, 14:08:56 |
||
Test set rating calculations | Anan | 4 | by Anan Thursday, August 22, 2024, 06:06:26 |
|
Final Evaluation Question | 2 | by Wednesday, August 21, 2024, 15:55:48 |
||
Long evaluation time | Szymon | 1 | by Maciej Sunday, August 04, 2024, 17:19:29 |
|
How to add new team memebers in the team | Abdul | 1 | by Maciej Monday, July 15, 2024, 22:19:37 |
|
Test set | MAROUANE | 2 | by MAROUANE Thursday, July 04, 2024, 09:14:02 |
|
Is test set from the same distribution as train set? | Alex | 1 | by Competition Wednesday, June 26, 2024, 17:18:14 |
|
Add other users to my team | Alex | 1 | by Maciej Wednesday, June 26, 2024, 12:25:26 |
|
Evaluation is online! | Maciej | 3 | by Maciej Wednesday, June 26, 2024, 11:03:14 |
|
Use external information | Alex | 1 | by Maciej Wednesday, June 26, 2024, 10:52:48 |
|
Looking for teammates | 2 | by Tuesday, June 25, 2024, 10:05:37 |
||
Puzzle taker vs simulated opponent | Dymitr | 3 | by Dymitr Wednesday, June 19, 2024, 18:04:47 |
|
Duplicate file in Your Team. | Carlos | 5 | by Maciej Monday, June 10, 2024, 13:12:04 |
|
Chess engine | Michal | 1 | by Maciej Tuesday, May 21, 2024, 13:57:38 |
|
Transfer learning | Łukasz | 1 | by Maciej Tuesday, May 14, 2024, 10:55:04 |