Glossary¶
- Benchmarking Event¶
Time-bound contest where a tool, pipeline, service or product, i.e. the participant, is compared against other participants using a predefined collection of reference datasets and assessment metrics.
- Benchmarking Workflow¶
Docker-based pipeline prepared by benchmarking event manager/s that calculates the performance metrics for a given participant’s dataset. A Benchmarking Event consumes behind the scences a Benchmarking Workflow with a particular set of golden reference datasets. See more.
- Challenge¶
Each of the categories in which a benchmarking event is divided. In its simplest form, one challenge comprises one reference dataset and one or more evaluation metrics. This can be customised if needed.
- Community¶
The organised group behind the benchmarking.
- Participant¶
Software application (program, server or pipeline) evaluated within a benchmarking event in at least one challenge. The same program can participate multiple times if various versions or parameter settings are benchmarked separately.
- (OpenEBench) Virtual Resarch Environment (VRE)¶
Cloud-based analysis platform where the assessment of the participants’ datasets takes place. The platform executes in a transparent and reproducible way Benchmarking Workflows . URL: https://openebench.bsc.es/vre/ . See See more.