-
Notifications
You must be signed in to change notification settings - Fork 779
Glossary
A challenge, run by some institute or organization, wherein a number of researchers, students and data scientists participate and compete with each other over a period of time. Each challenge has a start time and generally an end time also.
A member of the host team which is organizing a challenge. In our system, it is a form of representing a user. A user can be in the organizing team of many challenges and hence for each challenge, its challenge host will be different.
A group of challenge hosts who are organizing a challenge. They are identified by a unique team name.
A challenge phase split is a relation between a challenge phase and dataset splits for a challenge (many to many relation). This is used to set the privacy of submissions (public/private) to different dataset splits for different challenge phases.
A dataset in EvalAI is the main entity on which an AI challenge is based on. Participants are expected to make submissions corresponding to different splits of the corresponding dataset.
A dataset is generally divided into different parts and each part is called dataset split. Generally, a dataset has three different splits:
- Training set
- Validation set
- Test set
EvalAI is an open-source web platform that aims to evaluate the state of the art in AI. Its goal is to help AI researchers, practitioners, and students to host, collaborate, and participate in AI challenges organized around the globe.
The leaderboard can be defined as a scoreboard listing the name of the teams along with their current scores. Presently each challenge has its own leaderboard.
A challenge can be divided into many phases (or challenge phases). A challenge phase can have the same or different start and end date than the challenge start and end date.
A member of the team who is competing against other teams for any particular challenge. It is a form of representing a user. A user can participate in many challenges, hence for each challenge, its participant entry will be different.
A group of one or more participants who are taking part in a challenge. They are identified uniquely by a team name.
A way of submitting your results to the platform, so that it can be evaluated and ranked amongst others. A submission can be public or private, depending upon how the challenge is.
A python script which processes messages of submission received from a queue. It does the heavy lifting task of receiving a submission, performing mandatory checks and then evaluating the submission and updating its status in the database.
A model, present in web app which helps CloudCV in registering any new contributor as a core team member or simply an open source contributor.
This is generally a file uploaded by a challenge host and is associated with a challenge phase. This file is used for ranking the submission made by a participant. An annotation file can be shared by more than one challenge phase. In the codebase, this is present as a file field attached to challenge phase model.