3 months, 1 week ago

IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty

The aim of the competition is to predict the difficulty of chess puzzles based on board configurations and moves that the solution to each puzzle consists of. The difficulty level is measured as the rating on the lichess platform. The top 3 solutions will be awarded prizes. IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty is the sixth data science competition organized in association with the IEEE International Conference on Big Data series (IEEE BigData 2024, https://www3.cs.stonybrook.edu/~ieeebigdata2024/index.html).

See the detailed program of our competition presentations at IEEE BigData 2024 -
https://qedsoftware.com/IEEE_BigData_2024_Chess_and_Granulation.pdf

Overview

A chess puzzle is a particular configuration of pieces on a chessboard, where the puzzle taker is instructed to assume the role of one of the players and continue the game from that position. The player has to find from one to several moves, until she delivers mate or obtains a decisive material advantage.

In the online setting, where these are often solved, the puzzle taker only makes moves from one side, while the puzzle publisher provides responses from the other side. One such puzzle solving service is Lichess Training

Solving puzzles is considered one of the primary ways to hone chess skills. However, currently the only way to reliably estimate puzzle difficulty is to present it to a wide variety of chess players and see if they manage to solve it. 

The goal of the contest is to predict how difficult a chess puzzle is just by looking at the board setup and the moves in the solution. Puzzle difficulty is measured by its Glicko-2 rating calibrated on the lichess.org website. In simplified terms, it means that lichess models the difficulty of a puzzle by assuming that every attempt at solving a puzzle is a “match”. If a user solves the puzzle correctly, she gains puzzle rating and the puzzle loses rating. The opposite happens when the user doesn’t find the full solution (partial solutions count as “losses”). Both user and puzzle ratings are initialized at 1500. More information about the Glicko rating can be found here.

Each chess puzzle is described by the initial position (using Forsyth–Edwards Notation, or FEN) and the moves included in the puzzle solution, starting with one move leading to the puzzle position and then alternating between the moves that the puzzle solver has to find and those made by the simulated “opponent”.

IEEE Big Data 2024: We will encourage the top 3 winners to submit papers describing their solutions. It is already agreed that the conference will provide the top 3 winners with free registrations. The QED Software’s team, just like in the previous years, intends to organize a workshop devoted to the competition outcomes. According to our experience, the ability to present workshop papers may be an extra incentive for participants to consider active involvement in the competition. 

The aim of the competition is to predict the difficulty of chess puzzles based on board configurations and moves that the solution to each puzzle consists of. The difficulty level is measured as the rating on the lichess platform. The top 3 solutions will be awarded prizes. IEEE BigData 2024 Cup: Predicting Chess Puzzle Difficulty is the sixth data science competition organized in association with the IEEE International Conference on Big Data series (IEEE BigData 2024, https://www3.cs.stonybrook.edu/~ieeebigdata2024/index.html).

See the detailed program of our competition presentations at IEEE BigData 2024 - https://qedsoftware.com/IEEE_BigData_2024_Chess_and_Granulation.pdf

Terms & Conditions
 
 
Please log in to the system!
News

The Competition is Over!

Sincere thanks to all participants and congratulations to the winners!

The top3 teams will receive prizes and free full registration to IEEE Big Data conference.

In addition, selected teams (based on the score and interesting report) will be invited to participate in the special session during IEEE Big Data conference.

 

The data are provided as two .csv files, one for training dataset and one for testing dataset.

Each row of the testing dataset consists of the following fields:

Field name

Field description

Field type

Example value

PuzzleId

Unique puzzle ID

string

00sHx

FEN (link)

Standard notation for describing a particular board position of a chess game.

string

q3k1nr/1pp1nQpp/3p4/1P2p3/4P3/B1PP1b2/B5PP/5K2 b k - 0 17

Moves

Solution to the puzzle in Portable Game Notation (PGN). Includes the last move made before the puzzle position.

string

e8d7 a2e6 d7d8 f7f8

Based on the above data, the challenge contestants are expected to predict the Rating field (which will be kept secret).

Field name

Field description

Field type

Example value

Rating

Puzzle rating

int

1760

 

The training dataset contains all of the above fields, and also a few additional ones listed below. These fields are sometimes null in the training set and will not be provided for the test set:

RatingDeviation (int): Measure of uncertainty over puzzle’s difficulty.

Popularity (int): Users can ”upvote“ or “downvote” a puzzle. This value is the difference between the number of upvotes and downvotes.

NbPlays (int): Number of attempts at solving the puzzle.

Themes (str): Lichess allows choosing puzzles to solve based on different themes, such as tactical concepts, solution length or puzzle types (e.g. mates in x moves).

GameUrl (str): Lichess puzzles are generated based on games played on lichess.

OpeningTags (str): Information about the opening from which this puzzle originated.

Solution format 

Solutions in this competition should be submitted to the online evaluation system as a text file with exactly 2282 lines containing predictions for test instances. Each line in the submission should contain a single integer that indicates the predicted rating of the chess puzzle. The ordering of predictions should be the same as the ordering of the test set.

Evaluation

The quality of submissions will be evaluated using the mean squared error metric. 

Solutions will be evaluated online, and the preliminary results will be published on the public leaderboard. The public leaderboard will be available starting May 30th. The preliminary score will be computed on a small subset of the test records, fixed for all participants. The final evaluation will be performed after the completion of the competition using the remaining part of the test records. Those results will also be published online. It is important to note that only teams that submit a report describing their approach before the end of the challenge will qualify for the final evaluation.

 

There are two data files available to download.

 

Final results
Rank Team Name Is Report   Preliminary Score Final Score Submissions
1
bread emoji
True True 49141.5359 104540.656891 47
2
anansch
True True 58810.4586 120682.234604 38
3
ousou
True True 69890.9227 123103.229717 50
4
Andryyyyy
True True 61381.3812 129245.229228 56
5
ToDoFindATeamName
True True 65136.8232 132631.424731 53
6
alexmolas
True True 74378.0110 137839.668622 22
7
dymitr
True True 69202.5691 141488.501466 50
8
Feiwyth
True True 70792.7182 146729.234115 6
9
NxGTR
True True 73832.3591 150757.042522 35
10
BigData2024
True True 74135.4586 154905.759042 51
11
transformer_enjoyer
True True 75995.0221 158292.081623 27
12
neuralnite
True True 82049.4972 160677.509775 6
13
MrAces
True True 78837.4807 163552.500000 43
14
DML
True True 91728.0055 170533.507331 133
15
shoggoth
True True 87533.5193 171309.505376 68
16
AIBrain
True True 91889.9503 189554.737537 43
17
Marek
True True 119067.6243 195828.732649 172
18
LcWP
True True 167116.2762 244485.950147 4
19
JustEngine
False True 67827.4254 No report file found or report rejected. 101
20
deep
False True 81429.3204 No report file found or report rejected. 4
21
baellouf
False True 82238.9890 No report file found or report rejected. 43
22
September
False True 84712.2541 No report file found or report rejected. 19
23
Amy
False True 91476.2762 No report file found or report rejected. 78
24
JKU-CODA
False True 91664.9890 No report file found or report rejected. 20
25
Plats Bruts
False True 98152.9503 No report file found or report rejected. 16
26
hieuvq
False True 101897.2818 No report file found or report rejected. 10
27
checkmate
False True 101972.7072 No report file found or report rejected. 4
28
scotchgame
False True 85906.9503 No report file found or report rejected. 5
29
kubapok
False True 120870.0718 No report file found or report rejected. 4
30
Fontageau
False True 122314.7127 No report file found or report rejected. 2
31
soksly
False True 123260.3646 No report file found or report rejected. 9
32
French_bestbytest
False True 131666.9171 No report file found or report rejected. 13
33
tafhi
False True 135894.2928 No report file found or report rejected. 2
34
fuzz
False True 142167.5028 No report file found or report rejected. 3
35
Cavajah
False True 149984.5028 No report file found or report rejected. 3
36
Azeezah
False True 169580.3370 No report file found or report rejected. 1
37
OrganizerTest
False True 187245.4199 No report file found or report rejected. 1
  • May 08, 2024: start of the competition, datasets become available, 
  • May 30, 2024: public leaderboard becomes available
  • August 31, 2024: deadline for submitting the solutions, 
  • September 12 (extended), 2024: deadline for sending the reports, end of the competition, 
  • September 15, 2024: online publication of the final results, sending invitations for submitting papers to the associated workshop at the IEEE Big Data 2024 conference, 
  • October 13, 2024: deadline for submitting invited papers,
  • October 28, 2024: notification of paper acceptance,
  • November 17, 2024: camera-ready of accepted papers due.

QED will sponsor the cash prizes:

  • 1000 USD for the winning solution
  • 500 USD for the 2nd place solution
  • 250 USD for the 3rd place solution

Additionally, the IEEE Big Data 2024 conference will provide the top 3 performers with free full registrations

  • Jan Zyśko
  • Katarzyna Jagieła
  • Maciej Świechowski
  • Sebastian Stawicki
  • Andrzej Janusz
  • Dominik Ślęzak
  • Zbigniew Pakleza
This forum is for all users to discuss matters related to the competition. Good manners apply!
  Discussion Author Replies Last post
Paper Submission Anan 4 by
Wednesday, September 18, 2024, 14:55:26
Final Score 2 by
Wednesday, September 18, 2024, 14:54:17
Question about final 3 choices 3 by Maciej
Friday, September 13, 2024, 07:10:34
Dude, Report submission is working now, please submit your report before the new deadline M 0 by M
Wednesday, September 11, 2024, 17:59:04
Problem sending the report , chess 2 by
Wednesday, September 11, 2024, 15:12:59
could you please open the report submission again for at least one day? as the website was down for the past few days. M 5 by Anan
Tuesday, September 10, 2024, 12:30:44
When exactly is the deadline for submitting solutions? 8 by M
Monday, September 09, 2024, 09:08:20
Player initial rating deviation 1 by
Friday, August 23, 2024, 14:33:17
Player initial rating deviation 0 by
Friday, August 23, 2024, 14:08:56
Test set rating calculations Anan 4 by Anan
Thursday, August 22, 2024, 06:06:26
Final Evaluation Question 2 by
Wednesday, August 21, 2024, 15:55:48
Long evaluation time Szymon 1 by Maciej
Sunday, August 04, 2024, 17:19:29
How to add new team memebers in the team Abdul 1 by Maciej
Monday, July 15, 2024, 22:19:37
Test set MAROUANE 2 by MAROUANE
Thursday, July 04, 2024, 09:14:02
Is test set from the same distribution as train set? Alex 1 by Competition
Wednesday, June 26, 2024, 17:18:14
Add other users to my team Alex 1 by Maciej
Wednesday, June 26, 2024, 12:25:26
Evaluation is online! Maciej 3 by Maciej
Wednesday, June 26, 2024, 11:03:14
Use external information Alex 1 by Maciej
Wednesday, June 26, 2024, 10:52:48
Looking for teammates 2 by
Tuesday, June 25, 2024, 10:05:37
Puzzle taker vs simulated opponent Dymitr 3 by Dymitr
Wednesday, June 19, 2024, 18:04:47
Duplicate file in Your Team. Carlos 5 by Maciej
Monday, June 10, 2024, 13:12:04
Chess engine Michal 1 by Maciej
Tuesday, May 21, 2024, 13:57:38
Transfer learning Łukasz 1 by Maciej
Tuesday, May 14, 2024, 10:55:04