Submission¶

Submission should be made by uploading the algorithm container. Use this GitHub repository as an example of the algorithm container for submission. Note that this algorithm container should be used for submission purposes only, and model training should be done using different code.

In addition, please note that participants do not have access to the internet to prevent unauthorized access to the test set. Therefore, you need to download the pretrained weights within your container in order to utilize them.

* Important¶

Please take note of the following changes that have been implemented in the submission system:

1. The submission limit during the training phase has been restricted to 15 times.
2. The time limit for each algorithm has been set to 10 minutes for processing 100 images. It is important to ensure that the evaluation of 100 images (one test batch) using the reference container is completed within this time frame.
3. Algorithms that exceed the time limit will be automatically cancelled, and outputs will not be generated.
4. For detailed information about the provided infrastructure regarding GPU support, take a look at the infrastructure section in this link.

Evaluation¶

The final overall score for each algorithm is calculated using the following code, which involves summing up three correlation coefficient values: PLCC, SROCC, and KROCC.

from scipy.stats import pearsonr, spearmanr, kendalltau

  aggregate_results=dict()
  aggregate_results["plcc"] = abs(pearsonr(total_pred, total_gt)[0])
  aggregate_results["srocc"] = abs(spearmanr(total_pred, total_gt)[0])
  aggregate_results["krocc"] = abs(kendalltau(total_pred, total_gt)[0])
  aggregate_results["overall"] = abs(pearsonr(total_pred, total_gt)[0]) + abs(spearmanr(total_pred, total_gt)[0]) + abs(kendalltau(total_pred, total_gt)[0])