Hosting a competition with Codalab
By Nicolas Cazard - 2019/07/30
Codalab was created in 2013 as a joint venture between Microsoft and Stanford University and is an open source framework for running competitions, which is mostly used for machine learning challenge.
Many compeitions are hosted on this platform in which you can try to compete. This article will show you the basics of how to create your own one.
Uploading a competition
A competition consists of a set of files zipped in an archive named a bundle.
Here is an example of a simple bundle competition that you can download and use as a starting point for your own competition.
Here is the procedure to follow for uploading it on Codalab :
- Create an account on Codalab, and sign in.
- Click on "My Competitions" -> "Competitions I'm Running" -> "Create Competition"
- Upload the archive "competition_bundle.zip".
Congratulations, you have just created your first competition with Codalab !
You can now edit this competition online directly for minor modifications, like rescheduling the starting date or modifying the text which describe the dataset.
However, for major updates, you will have to dig into and modify the bundle itself before uploading it again on Codalab.
Updating a competition
If you unzip the previous bundle, you will find several files in it :
- competition.yaml : Probably the most important one, it contains the metadata for the competition. It describes all the attributes in the Codalab competition definition language, using YAML. If you want to change the competition name, add describing files, add a new metric... you will need to update this file accordingly.
- scoring program : This folder contains all the metrics needed to evaluate the participants' submissions. You can modify it or add more than one metric, but keep in mind that you have to follow the standard codalab structure and your function must output a score.txt file containing each metrics referenced in the competition.yaml file.
- datas : The ground truth of your dataset that you want to compare with every participants' submissions must be uploaded to codalab in a zipped folder, and referenced in the competition.yaml file as the reference_data. If your dataset isn't too large, you can also upload it, as weel as useful fonctions or a describing notebook, as public_data.
- HTML pages : Descriptive texts about the dataset and metrics used, instructions for participants.
Once all your modifications are done, don't forget to delete and replace the previous datas and metrics bundles.
You can then build the bundle for the competition with the command :
zip -r competition_bundle.zip *
And upload your brand new competition bundle on Codalab.
To go further
This tutorial shows you how to tune a basic bundle competition to create your own one, but Codalab allows you to tune many others aspects of your competition : You can organise your competition around several phases, allow users to submit their code instead of their results only, you can run your own instance of codalab at home...
This website gathers a lot of in-depth information about a more complex use of Codalab and may be worth a look.