Package 'mlr3benchmark'

Description

Implements methods for post-hoc analysis and visualisation of benchmark experiments, for 'mlr3' and beyond.

Author(s)

Maintainer: Sebastian Fischer [email protected] (ORCID) [contributor]

Authors:

• Sonabend Raphael [email protected] (ORCID)

• Florian Pfisterer [email protected] (ORCID)

Other contributors:

• Michel Lang [email protected] (ORCID) [contributor]

• Bernd Bischl [email protected] (ORCID) [contributor]

See Also

Useful links:

• https://mlr3benchmark.mlr-org.com

• https://github.com/mlr-org/mlr3benchmark

• Report bugs at https://github.com/mlr-org/mlr3benchmark/issues

Description

Coercion methods to BenchmarkAggr. For mlr3::BenchmarkResult this is a simple wrapper around the BenchmarkAggr constructor called with mlr3::BenchmarkResult⁠$aggregate()⁠.

Usage

as_benchmark_aggr(
  obj,
  task_id = "task_id",
  learner_id = "learner_id",
  independent = TRUE,
  strip_prefix = TRUE,
  ...
)

Arguments

Examples

df = data.frame(tasks = factor(rep(c("A", "B"), each = 5),
                               levels = c("A", "B")),
                learners = factor(paste0("L", 1:5)),
                RMSE = runif(10), MAE = runif(10))

as_benchmark_aggr(df, task_id = "tasks", learner_id = "learners")


if (requireNamespaces(c("mlr3", "rpart"))) {
  library(mlr3)
  task = tsks(c("boston_housing", "mtcars"))
  learns = lrns(c("regr.featureless", "regr.rpart"))
  bm = benchmark(benchmark_grid(task, learns, rsmp("cv", folds = 2)))

  # default measure
  as_benchmark_aggr(bm)

  # change measure
  as_benchmark_aggr(bm, measures = msr("regr.rmse"))
}

Description

This function is deprecated, use as_benchmark_aggr() instead.

Coercion methods to BenchmarkAggr. For mlr3::BenchmarkResult this is a simple wrapper around the BenchmarkAggr constructor called with mlr3::BenchmarkResult⁠$aggregate()⁠.

Usage

as.BenchmarkAggr(
  obj,
  task_id = "task_id",
  learner_id = "learner_id",
  independent = TRUE,
  strip_prefix = TRUE,
  ...
)

Arguments

Examples

df = data.frame(tasks = factor(rep(c("A", "B"), each = 5),
                               levels = c("A", "B")),
                learners = factor(paste0("L", 1:5)),
                RMSE = runif(10), MAE = runif(10))

as_benchmark_aggr(df, task_id = "tasks", learner_id = "learners")


if (requireNamespaces(c("mlr3", "rpart"))) {
  library(mlr3)
  task = tsks(c("boston_housing", "mtcars"))
  learns = lrns(c("regr.featureless", "regr.rpart"))
  bm = benchmark(benchmark_grid(task, learns, rsmp("cv", folds = 2)))

  # default measure
  as_benchmark_aggr(bm)

  # change measure
  as_benchmark_aggr(bm, measures = msr("regr.rmse"))
}

Description

Generates plots for BenchmarkAggr, all assume that there are multiple, independent, tasks. Choices depending on the argument type:

• "mean" (default): Assumes there are at least two independent tasks. Plots the sample mean of the measure for all learners with error bars computed with the standard error of the mean.

• "box": Boxplots for each learner calculated over all tasks for a given measure.

• "fn": Plots post-hoc Friedman-Nemenyi by first calling BenchmarkAggr⁠$friedman_posthoc⁠ and plotting significant pairs in coloured squares and leaving non-significant pairs blank, useful for simply visualising pair-wise comparisons.

• "cd": Critical difference plots (Demsar, 2006). Learners are drawn on the x-axis according to their average rank with the best performing on the left and decreasing performance going right. Any learners not connected by a horizontal bar are significantly different in performance. Critical differences are calculated as:

  $CD = q_{\alpha} \sqrt{\left(\frac{k(k+1)}{6N}\right)}$

  Where $q_\alpha$ is based on the studentized range statistic. See references for further details. It's recommended to crop white space using external tools, or function image_trim() from package magick.

Usage

## S3 method for class 'BenchmarkAggr'
autoplot(
  object,
  type = c("mean", "box", "fn", "cd"),
  meas = NULL,
  level = 0.95,
  p.value = 0.05,
  minimize = TRUE,
  test = "nem",
  baseline = NULL,
  style = 1L,
  ratio = 1/7,
  col = "red",
  friedman_global = TRUE,
  ...
)

Arguments

Value

The generated plot.

References

Demšar J (2006). “Statistical Comparisons of Classifiers over Multiple Data Sets.” Journal of Machine Learning Research, 7(1), 1-30. https://jmlr.org/papers/v7/demsar06a.html.

Examples

if (requireNamespaces(c("mlr3learners", "mlr3", "rpart", "xgboost"))) {
library(mlr3)
library(mlr3learners)
library(ggplot2)

set.seed(1)
task = tsks(c("iris", "sonar", "wine", "zoo"))
learns = lrns(c("classif.featureless", "classif.rpart", "classif.xgboost"))
bm = benchmark(benchmark_grid(task, learns, rsmp("cv", folds = 3)))
obj = as_benchmark_aggr(bm)

# mean and error bars
autoplot(obj, type = "mean", level = 0.95)

if (requireNamespace("PMCMRplus", quietly = TRUE)) {
  # critical differences
  autoplot(obj, type = "cd",style = 1)
  autoplot(obj, type = "cd",style = 2)

  # post-hoc friedman-nemenyi
  autoplot(obj, type = "fn")
}

}

Description

An R6 class for aggregated benchmark results.

Details

This class is used to easily carry out and guide analysis of models after aggregating the results after resampling. This can either be constructed using mlr3 objects, for example the result of mlr3::BenchmarkResult⁠$aggregate⁠ or via as_benchmark_aggr, or by passing in a custom dataset of results. Custom datasets must include at the very least, a character column for learner ids, a character column for task ids, and numeric columns for one or more measures.

Currently supported for multiple independent datasets only.

Active bindings

Methods

Public methods

• BenchmarkAggr$new()

• BenchmarkAggr$print()

• BenchmarkAggr$summary()

• BenchmarkAggr$rank_data()

• BenchmarkAggr$friedman_test()

• BenchmarkAggr$friedman_posthoc()

• BenchmarkAggr$subset()

• BenchmarkAggr$clone()

Method new()

Creates a new instance of this R6 class.

Usage

BenchmarkAggr$new(
  dt,
  task_id = "task_id",
  learner_id = "learner_id",
  independent = TRUE,
  strip_prefix = TRUE,
  ...
)

Arguments

Method print()

Prints the internal data via data.table::print.data.table.

Usage

BenchmarkAggr$print(...)

Arguments

Method summary()

Prints the internal data via data.table::print.data.table.

Usage

BenchmarkAggr$summary(...)

Arguments

Method rank_data()

Ranks the aggregated data given some measure.

Usage

BenchmarkAggr$rank_data(meas = NULL, minimize = TRUE, task = NULL, ...)

Arguments

Method friedman_test()

Computes Friedman test over all tasks, assumes datasets are independent.

Usage

BenchmarkAggr$friedman_test(meas = NULL, p.adjust.method = NULL)

Arguments

Method friedman_posthoc()

Posthoc Friedman Nemenyi tests. Computed with PMCMRplus::frdAllPairsNemenyiTest. If global ⁠$friedman_test⁠ is non-significant then this is returned and no post-hocs computed. Also returns critical difference

Usage

BenchmarkAggr$friedman_posthoc(
  meas = NULL,
  p.value = 0.05,
  friedman_global = TRUE
)

Arguments

Method subset()

Subsets the data by given tasks or learners. Returns data as data.table::data.table.

Usage

BenchmarkAggr$subset(task = NULL, learner = NULL)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

BenchmarkAggr$clone(deep = FALSE)

Arguments

References

'r format_bib("demsar_2006")

Examples

# Not restricted to mlr3 objects
df = data.frame(tasks = factor(rep(c("A", "B"), each = 5),
                               levels = c("A", "B")),
                learners = factor(paste0("L", 1:5)),
                RMSE = runif(10), MAE = runif(10))
as_benchmark_aggr(df, task_id = "tasks", learner_id = "learners")

if (requireNamespaces(c("mlr3", "rpart"))) {
  library(mlr3)
  task = tsks(c("boston_housing", "mtcars"))
  learns = lrns(c("regr.featureless", "regr.rpart"))
  bm = benchmark(benchmark_grid(task, learns, rsmp("cv", folds = 2)))

  # coercion
  as_benchmark_aggr(bm)
}

Description

Internal helper function for documentation.

Usage

requireNamespaces(x)

Arguments

Value

A logical(1), indicating wether all required packages are available.