Training data in this challenge are hourly aggregated values of various workload characteristics extracted from device logs. They were made available in the form of a CSV table containing ten columns. The first three of these columns are identifiers. They are followed by the mean, standard deviation, and a candlestick aggregation of the corresponding values. In particular, the meanings of the columns in the data set are:

• hostname: an ID of the device
• series: a name of the considered characteristic
• time_window: a timestamp of the aggregation window; the row aggregates values from an hour starting at the indicated timestamp
• Mean: the mean of the values
• SD: the standard deviation of the values
• Open: a value of the first reading during the corresponding hour
• High: the maximum of values
• Low: the minimum of values
• Close:  a value of the last reading during the corresponding hour
• Volume: the number of values

For each hostname-series pair in the data, values can be arranged into a time series spanning for over 80 days. Note, however, that some values can be missing for some pairs. Moreover, hostnames correspond to heterogeneous types of devices for which different sets of characteristics are monitored. Some of these devices are a part of the same system and it is likely that their workloads are highly correlated.

The task and the format of submissions: the task in this challenge is to predict future workload characteristic values of a number of devices from the training data. IDs of the devices (hostname) and their characteristics for which the predictions are to be made (series) are indicated in the solution_template.csv file. This file was made available in the Data files section. Participants of the challenge are asked to predict 168 consecutive values of each indicated time series (one full week) and upload the predictions through the submission system.

The format of submissions should be the same as in the solution_template.csv file. Solutions should be submitted as CSV files containing 170 columns. The first two columns should contain device ID (hostname) and characteristic ID (series), respectively. They should be followed by 168 numeric columns containing predictions – mean values of the corresponding characteristics for the next 168 hours (one week starting at 2020-02-20 12:00:00) after the end of the training data. The file exemplary_solution.csv contains an example of a correctly formatted submission file.

Evaluation: the quality of submissions will be evaluated using the $R^2$ measure, i.e., for each time series, the forecasts will be compared to ground truth values, and their quality will be assessed using the formula:

$$R^2(f, y) = 1 - \frac{RSS(f, y)}{TSS(y)},$$ where $RSS(f, y)$ is the residual sum of squares of forecasts: $$RSS(f, y) = \sum_i (y_i - f_i)^2,$$ and $TSS(y)$ is the total sum of squares: $$TSS(y) =  \sum_i (y_i - \bar{y})^2,$$ and $\bar{y}$ is the mean value of time series $y$ estimated using available training data. The submission score is the average $R^2$ value over all time series from the test set.

Solutions will be evaluated on-line and the preliminary results will be published on the public leaderboard. The preliminary score will be computed on a small subset of the test time series (10%), fixed for all participants. The final evaluation will be performed after completion of the competition using the remaining part of the test data. Those results will also be published online. It is important to note that only teams that submit a report describing their approach before the end of the challenge will qualify for the final evaluation. Moreover, to be eligible for the awards, the winning teams must exceed the score of the baseline solution by at least 10%.

In case of any questions, please post on the competition forum or write an email to contact {at} knowledgepit.ml 

• March 23, 2020: start of the challenge, the data set becomes available
• March 25, 2020: submission system opens
• June 8, 2020 (23:59:59 GMT): submission system closes
• June 10, 2020 (23:59:59 GMT): sending reports due
• June 17, 2020: online publication of the final results, sending invitations for submitting papers
• July 1, 2020: deadline for submitting invited papers
• July 8, 2020: notification of paper acceptance
• July 15, 2020: camera-ready of accepted papers, and registration to the conference due

Authors of the top-ranked solutions (based on the final evaluation scores) were awarded prizes funded by the sponsors:

• First Prize: 1500 USD + one free FedCSIS'20 conference registration,
• Second Prize: 1000 USD + one free FedCSIS'20 conference registration,
• Third Prize: 500 USD + one free FedCSIS'20 conference registration.

The award ceremony took place during the FedCSIS'20 conference. Please note that the winners were eligible for the money prizes only if their final score exceeds the baseline solution score by at least 10%.

• Andrzej Janusz, QED Software & University of Warsaw
• Piotr Biczyk, QED Software
• Artur Bicki, EMCA Software
• Mateusz Przyborowski, QED Software & University of Warsaw

In case of any questions please post on the competition forum or write an email at contact {at} knowledgepit.ml