3 weeks, 3 days ago

FedCSIS 2025 Challenge: Predicting Chess Puzzle Difficulty - Second Edition

This is the second edition of chess puzzle competition, this time with much bigger datasets and new data fields. The goal is to build a model to predict the difficulty (measured as Lichess rating) of given chess puzzles. The top 3 solutions will be awarded prizes.

Overview

Main changes from the first edition:

For every puzzle in both datasets we will now provide 22 success probabilitity predictions in both datasets. These are precomputed using chess engines and represent the predicted success chance of a player of given rating divided by 11 levels and two rating types (rapid and blitz). This change is meant to lower the entry bar for contestants without access to specialized hardware.
Different data and bigger datasets, this time the training dataset has over 4.5 million instances, compared to 3.7 million in the first edition.

In a chess puzzle, the player assumes a role of White or Black in a particular configuration of pieces on a chessboard. The goal for the puzzle taker is to find the best sequence of moves, either outright checkmating the opponent or obtaining a winning material advantage.

On the Internet, chess puzzles are often found on chess websites, like Lichess. The moves from the opposing side are made automatically and the puzzle taker is provided with immediate feedback.

Solving puzzles is considered one of the primary ways to hone chess skills. However, currently the only way to reliably estimate puzzle difficulty is to present it to a wide variety of chess players and see if they manage to solve it.

The goal of the contest is to predict how difficult a chess puzzle is from the initial position of the pieces and moves in the solution. Puzzle difficulty is measured by its Glicko-2 rating calibrated by Lichess. In simplified terms, it means that Lichess treats each attempt at solving a puzzle like a match between the user and the puzzle. If the user solves the puzzle correctly, that counts as a win for the user and they gain puzzle rating while the puzzle loses rating. When the user fails to solve the puzzle, that counts as a loss and the opposite happens. Both user and puzzle ratings are initialized at 1500.

Each chess puzzle is described by the initial position (in Forsyth–Edwards Notation, or FEN) and the moves included in the puzzle solution (in Portable Game Notation, or PGN). The solution starts with one move leading to the puzzle position and includes both moves that the puzzle taker has to find and moves by the simulated opponent.

Terms & Conditions

Competition Participation Rules for "FedCSIS 2025 Challenge: Predicting Chess Puzzle Difficulty - Second Edition”

By entering this Competition you accept these official Competition rules.

1. Organizer

The Competition is organized by QED Software sp. z o. o. (the Organizer), registered at ul. Mazowiecka 11/49, 00-052 Warszawa.
The Competition is sponsored by QED Software sp. z o. o. (the Sponsors)
The Competition is organized via the KnowledgePit.ai platform (available at knowledgepit.ai or knowledgepit.ml) and any submissions made outside of the platform will not be admissible.

2. Entry

The Competition is open to all interested researchers, specialists, and students.
Members of the Contest Organizing Committee (see the dedicated section on the Competition page) and employees of the Organizer, the Sponsors, and their affiliated entities are not allowed to participate.
Persons who are residents of, or are affiliated with or employed by or otherwise contractually or legally tied to an organization, educational institution, company, or other entity of the Russian Federation or another state or territory that falls under the scope of international sanctions or controls for reasons of war, terrorism or otherwise, are not allowed to participate. The current list of such legal restrictions in force in the European Union is available at https://sanctionsmap.eu/#/main. If you have any doubt about whether such restriction may apply to you, the Organizer reserves the right to verify eligibility and to adjudicate your eligibility at any time.
Participants must be of legal age of majority (Check your country’s law, in Poland and in most countries it is 18 years of age).
Registration for the Competition is done via the KnowledgePit.ai platform (knowledgepit.ai or knowlegepit.ml).

3. Timeline

A detailed Competition time schedule is available in the Schedule section (below) of the Competition page.

4. Participants and Teams

A person may enroll in the Competition with only one KnowledgePit.ai user account. Using multiple accounts constitutes grounds for exclusion from the Competition.
Participants submit their solutions as members of teams made up of one or more persons.
Each participant may be a member of only one Team enrolled in the challenge.
Each Team must designate one of the Team Members as the Team Leader responsible for communication with the Organizer.
A single KnowledgePit.ai account can only be associated with one Team in a given competition. It is not possible to withdraw from a Team, but Teams can be merged.
Merging is done between Teams by their respective Team Leaders through the KnowledgePit.ai platform and requires the consent of both Teams.
Participation in the Competition is free and voluntary.
Privately sharing elements of the solution such as code or data outside of Teams is not permitted and may result in disqualification of the persons or Teams involved, however, it is permitted to share remarks and ideas with all participants on the KnowledgePit.ai Competition Forums.
Participants must use their own resources, particularly software and other necessary tools and equipment needed to prepare a solution to be submitted.

5. Solutions

Participants submit their solutions through the KnowledgePit.ai platform.
The submitted solution is checked for errors, and if none is found, the solution is immediately included on the public Leaderboard.
There is a strict limit on the total number of solutions that can be submitted by a Team during the Competition and also, on the number of solutions that can be submitted daily.
For this Competition, those limits are set to 500 and 10, respectively.
The daily submission limit resets at 11:59 PM GMT.
When two Teams merge, their solutions accumulate, but the limits for the resulting Team remain unchanged.
If a Team exceeds the limit for the total number of submissions, the Team will be unable to submit new solutions until the Competition ends.
A team can select up to three solutions for the final evaluation, and the best of them will constitute the Team’s final score (Final Solution).
If a Team fails to select its 3 solutions for final evaluation, then this team’s solution with the highest score on the public Leaderboard is automatically selected as the Team’s Final Solution.

6. Evaluation

Each Team is obliged to provide a short Report describing their Final Solution. The Report must contain information such as the name of the Team, the names of all Team Members, and a brief overview of the approach used in the solution. The description should explain all data pre-processing steps and model construction steps. It should be submitted in the KnowledgePit.ai platform submission system by the Report Submission Date specified in the Schedule section of the Competition page (below).
The final evaluation takes place after the expiry of the Report Submission Date.
After the Final Evaluation, three top-ranked Teams will be asked to provide the source codes that can be used to reproduce their Final Solutions and Documentation that would allow running the code. If the code requires to be run within a complex environment (e.g. distributed Hadoop cluster), a detailed setup explanation should be provided as well. The source codes will be used to verify the legitimacy of the solutions and will be shared with the Competition Sponsors in accordance with the provisions of Section 9 below.
For a Team to be eligible for an Award in this Competition, a Team must be one of the three top-ranked teams. Moreover, a Team must provide the report and source codes upon request for the verification of their legitimacy.
In this Competition, the teams are eligible for monetary prizes only if their final solution improves the baseline.
If the Contest Organizing Committee refrains from indicating such a solution in its announcement of winners, the additional prize will not be awarded. This additional prize may be awarded to any team, including one of the winners of the main prizes.
The Organizer holds the right to extend the deadlines for submitting solutions and/or reports. In such a case, participants will be informed about such an extension through the KnowledgePit.ai platform Competition Forum.
The Organizer is not responsible for any consequences of technical issues related to the participant’s ability to access or submit to the KnowledgePit.ai platform, especially for issues relating to Internet connectivity. If due to technical problems within the Organizer’s control, the resulting delays or temporary unavailability of the platform or its components are likely to impact the participant’s ability to timely submit the Team’s solution, the challenge deadline will be extended by the Organizer by the time of such unavailability, and participants will be informed of such extension by email.
The final ranking of the competing Teams will be published based on the final evaluation results.
In the case of draws in the evaluation scores, the time of the submission will be taken into account.

7. Prizes

The Prizes are listed in the dedicated section of the Competition page.
Each prize comprises two components: a monetary amount and the FedCSIS 2025 conference registration fee waiver. The Sponsors shall cover a maximum of one registration fee per winning team. The registration fee is a prize component that is non-monetary and cannot be converted into a monetary value.
Each winning Team must select their representative Team Member to participate in the FedCSIS 2025 and notify their selection to the Sponsors. If no Team Member is selected by his or her Team to participate in the conference, the registration fee shall be deemed forfeited and not constitute part of the prize.
Following the announcement of the winners, The Sponsors shall pay the monetary prizes to the winning teams. Each winning team shall notify the Organizer within 14days following the announcement of how the Team wishes the monetary prize to be paid out (single payment, split payments, etc.).
In order to receive the Prize, each prize recipient is required to submit to the Sponsors all documents necessary for the remittance of such prizes, such as their personal details, including bank account number, their certificate of tax residency issued by the tax authority of their place of residence or other documents required by law. If such necessary documents are not submitted within the time limit indicated in the notification of the award, the monetary prize cannot be paid out and is deemed to have been forfeited by the winner.
The winning participant who fails or refuses to provide the necessary legal documentation such as defined above will be deemed to have forfeited the monetary prize, and such prize will not be paid out. In such a case, the winning Team may select another of its members as the recipient of such a prize.
In accordance with the law, the Sponsors shall cover the tax payments required on the awarded prizes at the rates and on terms prescribed by law in the country of the Sponsors.
The Sponsors are not responsible in any way for any duties, levies, fees, or taxes that the winning participant may be subject to in his or her country of residence. It is the sole responsibility of the winning participant to inquire at the appropriate authorities about applicable documentation and/or tax payments that she or he may be required to submit.

8. Claims

Submissions are not admissible if they are in whole or part illegible, incomplete, damaged, altered, counterfeit, obtained through fraud, or late, or made or submitted in breach of these Rules.
Participant may be disqualified if the Organizer reasonably believes that the participant has attempted to undermine the legitimate operation of the Competition by using multiple accounts, cheating, deception, or other unfair playing practices or abuses, threats, or harassment towards other participants or the Organizer.
Organizer may at its discretion reject any submission or disqualify any participant if the Organizer reasonably believes that it was – respectively - produced in an unfair or illegitimate way or submitted by a person who has broken the challenge rules without providing any additional explanation.
If you think you were excluded without reason, or have any questions, please contact the Organizer at contact@knowledgepit.ai.

9. Data security, Privacy and Copyright

Each participant agrees to use reasonable and suitable measures to prevent persons who have not formally agreed to these Rules from gaining access to the Competition data.
You agree not to transmit, duplicate, publish, redistribute, or otherwise provide or make available the Competition data to any party not participating in the Competition. You agree to notify the Organizer immediately upon learning of any possible unauthorized transmission of or unauthorized access to the Competition data and agree to work with the Organizer to rectify any unauthorized transmission or access.
Each report, paper, and any other type of publication based on the team’s or participant’s research where data from this Competition is used should accredit KnowledgePit.ai, the Organizer and Sponsors as the institutions that provided data for the study.
The fact of accepting the award is equivalent to granting to the Organizers and Sponsors a worldwide, non-exclusive, sub-licensable, transferable, royalty-free, perpetual, and irrevocable right to use, reproduce, distribute, create derivative works of, publicly perform, publicly display, digitally perform, make, have made, sell, offer for sale and/or import, the winning solution submitted and the source code used to generate it, in any media now known or hereafter developed, for any purpose whatsoever, commercial or otherwise, without further approval, and without any payment to the participant or participants who authored or co-authored it. By accepting the award the participants also acknowledge that they have full and unrestricted rights to grant the aforementioned rights.
By enrolling in this Competition, the participant grants his/her consent for the processing of her/his registration data and the submissions and reports, and grants the Organizer the rights to use such data and submissions for the purpose of evaluation of solutions, competition administrative purposes and in post-competition research.
By accepting the award the winning participant grants the Organizer the right to process his or her personal data such as name, address, personal identification or security number, bank account or credit card number, and other necessary details provided for the purposes of prize processing and payment, including the payment of applicable taxes.
By accepting the award the participant grants the Organizer the right to use the participant's name, affiliation, and/or prize information for the purpose of informational and promotional purposes of the Competition and the KnowledgePit.ai platform in any medium without additional compensation.
Participant’s data is administered by the Organizer and shall be processed in accordance with the European data protection and privacy rules (The General Data Protection Regulation (EU) 2016/679). For more information check Organizer’s Privacy Policy.

10. Final arrangements

The Organizer reserves the right to modify the Rules of this Competition, including without limitation for the purpose of clarification, correcting obvious editing mistakes, extending the deadlines for the benefit of participants, or other minor amendments. In the event of any change to the Rules, participants will be informed of them via the Competition forum.
Unless otherwise provided in the Competition Rules above, all claims arising out of or relating to these Rules will be governed by Polish law and will be litigated in Poland. Participants consent to personal jurisdiction in those courts.
If any provision of these Rules is held to be invalid or unenforceable, all remaining provisions of the Rules will remain in full force and effect.

Enroll

Please log in to the system!

News

Additional Mask Competition

For those who qualified for final evaluation, we created an additional competition with an additional award!

The goal of the competition was simple - to create a best mask for the already submitted final solution. 10% of the puzzles were to be masked and scores were calculated based on how close the mask was to the ideal mask.

Task description

The training and testing datasets are provided in two .csv files.

Test dataset consists of the following fields:

Field name	Field description	Field type	Example value
PuzzleId	Unique puzzle ID	string	00sHx
FEN	Standard notation for describing a particular board position of a chess game.	string	q3k1nr/1pp1nQpp/3p4/1P2p3/4P3/B1PP1b2/B5PP/5K2 b k - 0 17
Moves	Solution to the puzzle in Portable Game Notation (PGN). Includes the last move made before the puzzle position.	string	e8d7 a2e6 d7d8 f7f8
Success_prob (22 fields)	Predicted success probabilities representing chance of correctly solving a puzzle by a player of a given rating and type of rating (rapid or blitz).	float	0.2640107

Based on the above data, the challenge contestants are expected to predict the Rating field (which will be kept secret).

Field name	Field description	Field type	Example value
Rating	Puzzle Glicko-2 rating	int	1760

The training dataset contains all of the above fields, and also a few additional ones listed below.

RatingDeviation (int): Measure of uncertainty in the Glicko-2 rating system. It decreases as more players attempt to solve the puzzle.

Popularity (int): Users can ”upvote“ or “downvote” a puzzle. This value is the difference between the number of upvotes and downvotes.

NbPlays (int): Number of attempts at solving the puzzle.

Themes (string): Lichess allows choosing puzzles to solve based on different themes, such as tactical concepts, solution length or puzzle types (e.g. mates in x moves).

GameUrl (string): Lichess puzzles are generated from the games played on the site.

OpeningTags (string): Information about the opening from which this puzzle originated. This field has missing values.

Solution format

Solutions in this competition should be submitted to the online evaluation system as a text file with exactly 2235 lines containing predictions for test instances. Each line in the submission should contain a single integer that indicates the predicted rating of the chess puzzle. The ordering of predictions should be the same as the ordering of the test set.

Evaluation

The quality of submissions will be evaluated using the mean squared error metric.

Solutions will be evaluated online, and the preliminary results will be published on the public leaderboard. The public leaderboard will be available starting April 25th. The preliminary score will be computed on a subset of the test records, fixed for all participants. The final evaluation will be performed after the completion of the competition using the remaining part of the test records. Those results will also be published online. It is important to note that only teams that submit a report describing their approach before the end of the challenge will qualify for the final evaluation.

Data files

(visible only if you are logged in)

Leaderboard

Rank	Team Name	Score	Submission Date
1	ousou	55382.0743	2025-06-26 19:25:17
2	bread emoji	58185.9508	2025-06-27 11:57:25
3	transformer_enjoyer	58891.7234	2025-06-25 20:50:13
4	ToDoFindATeamName	61523.7449	2025-06-23 22:59:52
5	neuro	66485.4458	2025-06-28 16:29:04
6	xyz	66638.4414	2025-06-28 23:08:22
7	DML	66657.5900	2025-06-28 23:26:57
8	Cyan	67071.6598	2025-06-28 15:02:25
9	Ru	70407.9982	2025-06-26 20:51:13
10	Feiwyth	70715.1038	2025-06-28 21:21:41
11	JKing	72369.5085	2025-06-28 22:16:23
12	Nemopilist	82539.2498	2025-06-26 21:33:13
13	Model	83448.3339	2025-06-28 22:19:23
14	Ai_chess	84641.4154	2025-06-23 13:40:58
15	Puzzle	85973.0340	2025-06-17 03:37:30
16	NoName	87117.4754	2025-06-23 21:56:31
17	xiededoudui	87162.2668	2025-06-23 11:04:13
18	No_Name	88284.9669	2025-06-25 21:39:47
19	undefined	90447.4127	2025-06-14 13:20:17
20	Chess-Challenge winner	92782.4073	2025-06-27 11:19:47
21	Glojk team	93009.9266	2025-06-26 18:08:50
22	test	95706.1388	2025-06-26 11:45:32
23	dymitr	98902.7207	2025-04-28 11:54:23
24	wojciech_drochomirecki	103724.9230	2025-06-22 19:02:24
25	Wymi@tacze	106364.2453	2025-06-27 19:54:27
26	Mathurin	107528.4306	2025-05-22 14:39:25
27	incorrect+crossvalidation	109050.8442	2025-06-28 19:53:39
28	flame-team	110692.9543	2025-06-13 05:09:04
29	kkrzywicki	120802.4628	2025-06-27 17:23:35
30	rs	120804.1710	2025-06-25 08:32:08
31	alan	123821.8711	2025-06-28 21:05:31
32	ShyamDGupta	124535.8514	2025-06-17 10:57:24
33	Barnie	124792.1030	2025-06-12 18:09:33
34	kuba_chess	127087.0707	2025-06-19 07:06:31
35	YesChess	132117.6401	2025-06-28 20:58:14
36	mb	133140.6312	2025-06-26 20:07:34
37	SzeleszczącaSzczkupakabra	133880.7466	2025-06-2 09:08:22
38	michalm	135140.1397	2025-05-11 12:07:40
39	Smedbros	148332.0215	2025-06-16 20:41:44
40	Ti-Kat	150904.9722	2025-06-10 09:33:11
41	asgdsj	170959.7619	2025-06-10 09:32:43
42	LN_419328	1628196.3679	2025-06-28 23:56:27

Final results

Rank	Team Name	Is Report		Preliminary Score	Final Score	Submissions
1	ousou	True	True	55382.0743	52310.896243	89
2	bread emoji	True	True	58185.9508	54377.454383	240
3	transformer_enjoyer	True	True	58891.7234	55938.176208	117
4	ToDoFindATeamName	True	True	61523.7449	57492.423077	136
5	Cyan	True	True	67071.6598	61045.490161	93
6	neuro	True	True	66485.4458	62567.827370	81
7	xyz	True	True	66638.4414	62685.150268	25
8	DML	True	True	66657.5900	63009.287120	73
9	Ru	True	True	70407.9982	67467.647585	53
10	Feiwyth	True	True	70715.1038	68073.359571	17
11	JKing	True	True	72369.5085	69641.728086	32
12	Nemopilist	True	True	82539.2498	76733.697674	15
13	Model	True	True	83448.3339	77586.666369	13
14	Puzzle	True	True	85973.0340	79099.969589	10
15	Ai_chess	True	True	84641.4154	80115.372093	7
16	No_Name	True	True	88284.9669	83032.500894	16
17	xiededoudui	True	True	87162.2668	83704.729875	5
18	NoName	True	True	87117.4754	83857.397138	13
19	Glojk team	True	True	93009.9266	84571.562612	2
20	Chess-Challenge winner	True	True	92782.4073	87224.424866	6
21	wojciech_drochomirecki	True	True	103724.9230	95351.983900	1
22	Wymi@tacze	True	True	106364.2453	96394.858676	6
23	incorrect+crossvalidation	True	True	109050.8442	100656.922182	26
24	kkrzywicki	True	True	120802.4628	111648.442755	1
25	kuba_chess	True	True	127087.0707	114532.191413	1
26	alan	True	True	123821.8711	116148.053667	3
27	ShyamDGupta	True	True	124535.8514	119662.008050	18
28	YesChess	True	True	132117.6401	124223.928444	2
29	mb	True	True	133140.6312	127312.208408	1
30	SzeleszczącaSzczkupakabra	True	True	133880.7466	128353.487478	8
31	Smedbros	True	True	148332.0215	141876.092129	1
32	undefined	False	True	90447.4127	No report file found or report rejected.	10
33	test	False	True	95706.1388	No report file found or report rejected.	5
34	dymitr	False	True	98902.7207	No report file found or report rejected.	9
35	Mathurin	False	True	107528.4306	No report file found or report rejected.	11
36	flame-team	False	True	110692.9543	No report file found or report rejected.	7
37	rs	False	True	120804.1710	No report file found or report rejected.	11
38	Barnie	False	True	124792.1030	No report file found or report rejected.	9
39	michalm	False	True	135140.1397	No report file found or report rejected.	4
40	EvaluatorTest2	False	False	999.0000	No report file found or report rejected.	2
41	Ti-Kat	False	True	150904.9722	No report file found or report rejected.	1
42	asgdsj	False	True	170959.7619	No report file found or report rejected.	2
43	LN_419328	False	True	1628196.3679	No report file found or report rejected.	1

Schedule

April 18, 2025: start of the competition
April 25, 2025: submitting solutions and public leaderboard become available
June 27, 2025 (23:59 GMT) June 28, 2025 (23:59 GMT): deadline for submitting the predictions
June 30, 2025 (23:59 GMT): deadline for sending the reports, end of the competition
July 7, 2025: online publication of the final results, sendinginvitations for submitting short papers for the special session at FedCSIS 2025
July 27, 2025: deadline for submitting invited papers
August 7, 2025: notification of paper acceptance
August 17, 2025: camera-ready of accepted papers, and registration for the conference are due

Awards

Authors of the top-ranked solutions (based on the final evaluation scores) will be awarded prizes funded by the Sponsors:

1000 USD for the winning solution + one FedCSIS 2025 registration
500 USD for the 2nd place solution + one FedCSIS 2025 registration
250 USD for the 3rd place solution + one FedCSIS 2025 registration

For the additional mask competition the winner will be awarded one FedCSIS 2025 registration.

Contest organizing committee

Jan Zyśko
Michał Ślęzak
Maciej Świechowski
Dominik Ślęzak

Forum

This forum is for all users to discuss matters related to the competition. Good manners apply!

	Discussion	Author	Replies	Last post
	Deadline setting seems to be wrong	Maria	2	by Maria Saturday, June 28, 2025, 16:16:36
	Additional Dataset Information		5	by Competition Tuesday, May 06, 2025, 06:02:11