3. Run a evaluation function with the selected tools and test data sets
The prcbench package currently provides two differnt
types of peformance evaluation.
Accuracy evaluation of precision-recall curves
Running-time evaluation of precision-recall curves
3a. Accuracy evaluation of precision-recall curves
The run_evalcurve function evaluates precision-recall
curves with the following five test cases. The basic idea is that the
function returns the full score as long as the points generated by a
library matches with the manually calculated recall and precision
values.
| Test case | Description |
|---|---|
| fpoint | Check the first point |
| int_pts | Check the intermediate points |
| epoint | Check the end point |
| x_range | Evaluate a range between two recall values |
| y_range | Evaluate a range between two precision values |
Evaluation scores
The run_evalcurve function calculates the scores of the
test cases and summarizes them to a data frame.
## Evaluate precision-recall curves for ROCR and precrec with c1 test set
testset <- create_testset("curve", "c1")
toolset <- create_toolset(c("ROCR", "precrec"))
scores <- run_evalcurve(testset, toolset)
scores## testset toolset toolname score
## 1 c1 ROCR ROCR 5/8
## 2 c1 precrec precrec 8/8The result of each test case can be displayed by specifying
data_type = all of the print
function.
## Print all results
print(scores, data_type = "all")## testset toolset toolname testitem testcat success total
## 1 c1 ROCR ROCR x_range Rg 1 1
## 2 c1 ROCR ROCR y_range Rg 1 1
## 3 c1 ROCR ROCR fpoint SE 0 1
## 4 c1 ROCR ROCR intpts Ip 2 4
## 5 c1 ROCR ROCR epoint SE 1 1
## 6 c1 precrec precrec x_range Rg 1 1
## 7 c1 precrec precrec y_range Rg 1 1
## 8 c1 precrec precrec fpoint SE 1 1
## 9 c1 precrec precrec intpts Ip 4 4
## 10 c1 precrec precrec epoint SE 1 1Visualization of the result
The autoplot shows a plot with the result of the
run_evalcurve function.
## ggplot2 is necessary to use autoplot
library(ggplot2)
## Plot base points and the result of precrec on c1, c2, and c3 test sets
testset <- create_testset("curve", c("c1", "c2", "c3"))
toolset <- create_toolset("precrec")
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1)
## Plot the results of PerfMeas and PRROC on c1, c2, and c3 test sets
toolset <- create_toolset(c("PerfMeas", "PRROC"))
scores2 <- run_evalcurve(testset, toolset)
autoplot(scores2, base_plot = FALSE)
3b. Running-time evaluation of precision-recall curves
The run_benchmark function internally calls the
microbenchmark function provided by the microbenchmark
package. It takes a test set and a tool set and returns the result of
microbenchmark.
## Run microbenchmark for aut5 on b10
testset <- create_testset("bench", "b10")
toolset <- create_toolset(set_names = "auc5")
res <- run_benchmark(testset, toolset)
res## testset toolset toolname min lq mean median uq max neval
## 1 b10 auc5 AUCCalculator 1.883 2.238 4.78 2.733 3.41 13.62 5
## 2 b10 auc5 PRROC 0.137 0.140 0.17 0.145 0.15 0.26 5
## 3 b10 auc5 PerfMeas 0.057 0.059 0.10 0.077 0.12 0.21 5
## 4 b10 auc5 ROCR 1.537 1.543 1.62 1.547 1.58 1.90 5
## 5 b10 auc5 precrec 3.558 3.627 3.73 3.630 3.79 4.03 5
