The hardware and bandwidth for this mirror is donated by dogado GmbH, the Webhosting and Full Service-Cloud Provider. Check out our Wordpress Tutorial.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]dogado.de.

prcbench

The aim of the prcbench package is to provide a testing workbench for evaluating precision-recall curves under various conditions. It contains integrated interfaces for the following five tools. It also contains predefined test data sets.

Disclaimer: prcbench was originally develop to help our precrec library in order to provide fast and accurate calculations of precision-recall curves with extra functionality.

Accuracy evaluation of precision-recall curves

prcbench uses pre-defined test sets to help evaluate the accuracy of precision-recall curves.

1. create_toolset: creates objects of different tools for testing (5 different tools)
2. create_testset: selects pre-defined data sets (c1, c2, and c3)
3. run_evalcurve: evaluates the selected tools on the simulation data
4. autoplot: shows the results with ggplot2 and patchwork

## Load library
library(prcbench)

## Plot base points and the result of 5 tools on pre-defined test sets (c1, c2, and c3)
toolset <- create_toolset(c("precrec", "ROCR", "AUCCalculator", "PerfMeas", "PRROC"))
testset <- create_testset("curve", c("c1", "c2", "c3"))
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1, ncol = 3, nrow = 2)

Running-time evaluation of precision-recall curves

prcbench helps create simulation data to measure computational times of creating precision-recall curves.

1. create_toolset: creates objects of different tools for testing
2. create_testset: creates simulation data
3. run_benchmark: evaluates the selected tools on the simulation data

## Load library
library(prcbench)

## Run benchmark for auc5 (5 tools) on b10 (balanced 5 positives and 5 negatives)
toolset <- create_toolset(set_names = "auc5")
testset <- create_testset("bench", "b10")
res <- run_benchmark(testset, toolset)

print(res)

Documentation

• Introduction to prcbench – a package vignette that contains the descriptions of the functions with several useful examples. View the vignette with vignette("introduction", package = "prcbench") in R.

• Help pages – all the functions including the S3 generics have their own help pages with plenty of examples. View the main help page with help(package = "prcbench") in R.

Installation

CRAN

install.packages("prcbench")

Dependencies

AUCCalculator requires a Java runtime environment (>= 6) if AUCCalculator needs to be evaluated.

GitHub

You can install a development version of prcbench from our GitHub repository.

devtools::install_github("evalclass/prcbench")

1. Make sure you have a working development environment.

   • Windows: Install Rtools (available on the CRAN website).

   • Mac: Install Xcode from the Mac App Store.

   • Linux: Install a compiler and various development libraries (details vary across different flavors of Linux).

2. Install devtools from CRAN with install.packages("devtools").

3. Install prcbench from the GitHub repository with devtools::install_github("evalclass/prcbench").

Troubleshooting

microbenchmark

microbenchmark does not work on some OSs. prcbench uses system.time when microbenchmark is not available.

rJava

• Some OSs require en extra configuration step after rJava installation.

sudo R CMD javareconf

• JDKs

1. Oracle JDK
2. OpenJDK

• JDKs for macOS

1. AdoptOpenJDK
2. AdoptOpenJDK with homebrew

• JRI support on macOS Big Sur – see this Stack Overflow thread.

install.packages("rJava", configure.args = "--disable-jri")

Citation

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2017; 33 (1): 145-147.

doi: 10.1093/bioinformatics/btw570

External links

• Classifier evaluation with imbalanced datasets – our web site that contains several pages with useful tips for performance evaluation on binary classifiers.

• The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets – our paper that summarized potential pitfalls of ROC plots with imbalanced datasets and advantages of using precision-recall plots instead.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.
Health stats visible at Monitor.