Package 'mlr3'

Description

Efficient, object-oriented programming on the building blocks of machine learning. Provides 'R6' objects for tasks, learners, resamplings, and measures. The package is geared towards scalability and larger datasets by supporting parallelization and out-of-memory data-backends like databases. While 'mlr3' focuses on the core computational operations, add-on packages provide additional functionality.

Learn mlr3

• Book on mlr3: https://mlr3book.mlr-org.com

• Use cases and examples gallery: https://mlr3gallery.mlr-org.com

• Cheat Sheets: https://github.com/mlr-org/mlr3cheatsheets

mlr3 extensions

• Preprocessing and machine learning pipelines: mlr3pipelines

• Analysis of benchmark experiments: mlr3benchmark

• More classification and regression tasks: mlr3data

• Connector to OpenML: mlr3oml

• Solid selection of good classification and regression learners: mlr3learners

• Even more learners: https://github.com/mlr-org/mlr3extralearners

• Tuning of hyperparameters: mlr3tuning

• Hyperband tuner: mlr3hyperband

• Visualizations for many mlr3 objects: mlr3viz

• Survival analysis and probabilistic regression: mlr3proba

• Cluster analysis: mlr3cluster

• Feature selection filters: mlr3filters

• Feature selection wrappers: mlr3fselect

• Interface to real (out-of-memory) data bases: mlr3db

• Performance measures as plain functions: mlr3measures

• Resampling methods for spatiotemporal data: mlr3spatiotempcv

• Data storage and prediction support for spatial objects: mlr3spatial

Suggested packages

• Parallelization framework: future

• Progress bars: progressr

• Encapsulated evaluation: evaluate, callr (external process)

Package Options

• "mlr3.exec_random": Randomize the order of execution in resample() and benchmark() during parallelization with future. Defaults to TRUE. Note that this does not affect the order of results.

• "mlr3.exec_chunk_size": Number of iterations to perform in a single future::future() during parallelization with future. Defaults to 1.

• "mlr3.exec_chunk_bins": Number of bins to split the iterations into. If set, "mlr3.exec_chunk_size" is ignored.

• "mlr3.debug": If set to TRUE, parallelization via future is disabled to simplify debugging and provide more concise tracebacks. Note that results computed in debug mode use a different seeding mechanism and are not reproducible.

• "mlr3.warn_version_mismatch": Set to FALSE to silence warnings raised during predict if a learner has been trained with a different version version of mlr3.

• "mlr3.prob_as_default": Set to TRUE to set the predict type of classification learners to "prob" by default (if they support it).

• "mlr3.mirai_parallelization": Compute profile to use for parallelization with mirai. Defaults to "mlr3_parallelization".

• "mlr3.mirai_encapsulation": Compute profile to use for encapsulation with mirai. Defaults to "mlr3_encapsulation".

• "mlr3.print_class_ratio_threshold": Maximum number of rows for which class ratios are computed when printing a classification task. For tasks with more rows, only class names are shown. Defaults to 1000000.

Error Classes

• Mlr3Error: The base mlr3 error class.

• Mlr3ErrorConfig: This error signals that the user has misconfigured something. By default, this error is not caught when the learner is encapsulated.

• Mlr3ErrorInput: This error signals that the input to the function is invalid.

• Mlr3ErrorLearner: The base error class for errors related to the learner.

• Mlr3ErrorLearnerTrain: This error signals that the learner failed to train the model.

• Mlr3ErrorLearnerPredict: This error signals that something went wrong during prediction.

• Mlr3TimeoutError: This error signals that the encapsulation during train or predict timed out.

Warning Classes

• Mlr3Warning: The base mlr3 warning class.

• Mlr3WarningConfig: This warning signals that the user has misconfigured something.

• Mlr3WarningInput: This warning signals that the input to the function is invalid.

Author(s)

Maintainer: Marc Becker [email protected] (ORCID)

Authors:

• Michel Lang [email protected] (ORCID)

• Bernd Bischl [email protected] (ORCID)

• Jakob Richter [email protected] (ORCID)

• Patrick Schratz [email protected] (ORCID)

• Martin Binder [email protected]

• Florian Pfisterer [email protected] (ORCID)

• Raphael Sonabend [email protected] (ORCID)

• Sebastian Fischer [email protected] (ORCID)

Other contributors:

• Giuseppe Casalicchio [email protected] (ORCID) [contributor]

• Stefan Coors [email protected] (ORCID) [contributor]

• Quay Au [email protected] (ORCID) [contributor]

• Lennart Schneider [email protected] (ORCID) [contributor]

• Lona Koers [email protected] [contributor]

• John Zobolas [email protected] (ORCID) [contributor]

• Maximilian Mücke [email protected] (ORCID) [contributor]

References

Lang M, Binder M, Richter J, Schratz P, Pfisterer F, Coors S, Au Q, Casalicchio G, Kotthoff L, Bischl B (2019). “mlr3: A modern object-oriented machine learning framework in R.” Journal of Open Source Software. doi:10.21105/joss.01903, https://joss.theoj.org/papers/10.21105/joss.01903.

See Also

Useful links:

• https://mlr3.mlr-org.com

• https://github.com/mlr-org/mlr3

• Report bugs at https://github.com/mlr-org/mlr3/issues

Description

Convert object to a BenchmarkResult.

Usage

as_benchmark_result(x, ...)

## S3 method for class 'BenchmarkResult'
as_benchmark_result(x, ...)

## S3 method for class 'ResampleResult'
as_benchmark_result(x, ...)

Arguments

Value

(BenchmarkResult).

Description

Wraps a DataBackend around data. mlr3 ships with methods for data.frame (converted to a DataBackendDataTable.

Additional methods are implemented in the package mlr3db, e.g. to connect to real DBMS like PostgreSQL (via dbplyr) or DuckDB (via DBI/duckdb).

Usage

as_data_backend(data, primary_key = NULL, ...)

## S3 method for class 'data.frame'
as_data_backend(data, primary_key = NULL, keep_rownames = FALSE, ...)

Arguments

Value

DataBackend.

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-backends

• Package mlr3db to interface out-of-memory data, e.g. SQL servers or duckdb.

Other DataBackend: DataBackend, DataBackendDataTable

Examples

# create a new backend using the penguins data:
as_data_backend(palmerpenguins::penguins)

Description

Convert object to a Learner or a list of Learner.

Usage

as_learner(x, ...)

## S3 method for class 'Learner'
as_learner(x, clone = FALSE, discard_state = FALSE, ...)

as_learners(x, ...)

## Default S3 method:
as_learners(x, ...)

## S3 method for class 'list'
as_learners(x, ...)

Arguments

Value

Learner.

Description

Convert object to a Measure or a list of Measure.

Usage

as_measure(x, task_type = NULL, clone = FALSE, ...)

## S3 method for class ''NULL''
as_measure(x, task_type = NULL, clone = FALSE, ...)

## S3 method for class 'Measure'
as_measure(x, task_type = NULL, clone = FALSE, ...)

as_measures(x, task_type = NULL, clone = FALSE, ...)

## Default S3 method:
as_measures(x, task_type = NULL, clone = FALSE, ...)

## S3 method for class ''NULL''
as_measures(x, task_type = NULL, clone = FALSE, ...)

## S3 method for class 'list'
as_measures(x, task_type = NULL, clone = FALSE, ...)

Arguments

Value

Measure.

Description

Convert object to a Prediction or a list of Prediction.

Usage

as_prediction(x, check = FALSE, ...)

## S3 method for class 'Prediction'
as_prediction(x, check = FALSE, ...)

## S3 method for class 'PredictionDataClassif'
as_prediction(x, check = FALSE, ...)

## S3 method for class 'PredictionDataRegr'
as_prediction(x, check = FALSE, ...)

as_predictions(x, predict_sets = "test", ...)

## S3 method for class 'list'
as_predictions(x, predict_sets = "test", ...)

Arguments

Value

Prediction.

Description

Convert object to a PredictionClassif.

Usage

as_prediction_classif(x, ...)

## S3 method for class 'PredictionClassif'
as_prediction_classif(x, ...)

## S3 method for class 'data.frame'
as_prediction_classif(x, ...)

Arguments

Value

PredictionClassif.

Examples

# create a prediction object
task = tsk("penguins")
learner = lrn("classif.rpart", predict_type = "prob")
learner$train(task)
p = learner$predict(task)

# convert to a data.table
tab = as.data.table(p)

# convert back to a Prediction
as_prediction_classif(tab)

# split data.table into a list of data.tables
tabs = split(tab, tab$truth)

# convert back to list of predictions
preds = lapply(tabs, as_prediction_classif)

# calculate performance in each group
sapply(preds, function(p) p$score())

Description

Convert object to a PredictionData or a list of PredictionData.

Usage

as_prediction_data(x, task, row_ids = task$row_ids, check = TRUE, ...)

## S3 method for class 'Prediction'
as_prediction_data(x, task, row_ids = task$row_ids, check = TRUE, ...)

## S3 method for class 'PredictionData'
as_prediction_data(x, task, row_ids = task$row_ids, check = TRUE, ...)

## S3 method for class 'list'
as_prediction_data(
  x,
  task,
  row_ids = task$row_ids,
  check = TRUE,
  ...,
  train_task
)

Arguments

Value

PredictionData.

Description

Convert object to a PredictionRegr.

Usage

as_prediction_regr(x, ...)

## S3 method for class 'PredictionRegr'
as_prediction_regr(x, ...)

## S3 method for class 'data.frame'
as_prediction_regr(x, ...)

Arguments

Value

PredictionRegr.

Examples

# create a prediction object
task = tsk("mtcars")
learner = lrn("regr.rpart")
learner$train(task)
p = learner$predict(task)

# convert to a data.table
tab = as.data.table(p)

# convert back to a Prediction
as_prediction_regr(tab)

# split data.table into a list of data.tables
tabs = split(tab, cut(tab$truth, 3))

# convert back to list of predictions
preds = lapply(tabs, as_prediction_regr)

# calculate performance in each group
sapply(preds, function(p) p$score())

Description

Convert object to a ResampleResult.

The S3 method for list expects argument x to be a list of Prediction objects and all other relevant objects (Task, Learners, and instantiated Resampling) must be provided, too. A more flexible way to manually create a ResampleResult is implemented in as_result_data().

Usage

as_resample_result(x, ...)

## S3 method for class 'ResampleResult'
as_resample_result(x, ...)

## S3 method for class 'ResultData'
as_resample_result(x, view = NULL, ...)

## S3 method for class 'list'
as_resample_result(x, task, learners, resampling, store_backends = TRUE, ...)

Arguments

Value

(ResampleResult).

Description

Convert object to a Resampling or a list of Resampling. This method e.g. allows to convert an OMLTask of mlr3oml to a Resampling.

Usage

as_resampling(x, ...)

## S3 method for class 'Resampling'
as_resampling(x, clone = FALSE, ...)

as_resamplings(x, ...)

## Default S3 method:
as_resamplings(x, ...)

## S3 method for class 'list'
as_resamplings(x, ...)

Arguments

Description

This function allows to construct or convert to a ResultData object, the result container used by ResampleResult and BenchmarkResult. A ResampleResult or BenchmarkResult can be initialized with the returned object. Note that ResampleResults can be converted to a BenchmarkResult with as_benchmark_result() and multiple BenchmarkResults can be combined to a larger BenchmarkResult with the ⁠$combine()⁠ method of BenchmarkResult.

Usage

as_result_data(
  task,
  learners,
  resampling,
  iterations,
  predictions,
  learner_states = NULL,
  data_extra = NULL,
  store_backends = TRUE
)

Arguments

Value

ResultData object which can be passed to the constructor of ResampleResult.

Examples

task = tsk("penguins")
learner = lrn("classif.rpart")
resampling = rsmp("cv", folds = 2)$instantiate(task)
iterations = seq_len(resampling$iters)

# manually train two learners.
# store learners and predictions
learners = list()
predictions = list()
for (i in iterations) {
  l = learner$clone(deep = TRUE)
  learners[[i]] = l$train(task, row_ids = resampling$train_set(i))
  predictions[[i]] = list(test = l$predict(task, row_ids = resampling$test_set(i)))
}

rdata = as_result_data(task, learners, resampling, iterations, predictions)
ResampleResult$new(rdata)

Description

Convert object to a Task or a list of Task.

The function supports:

• Converting existing Task objects (with optional cloning)

• Converting objects from other packages (e.g., OMLTask from mlr3oml)

• Converting lists of objects to lists of tasks

For constructing tasks from data frames, use the dedicated converters:

• as_task_classif() for classification tasks

• as_task_regr() for regression tasks

• as_task_unsupervised() for unsupervised tasks

Usage

as_task(x, ...)

## S3 method for class 'Task'
as_task(x, clone = FALSE, ...)

as_tasks(x, ...)

## Default S3 method:
as_tasks(x, ...)

## S3 method for class 'list'
as_tasks(x, ...)

Arguments

Description

Convert object to a TaskClassif. This is a S3 generic. mlr3 ships with methods for the following objects:

1. TaskClassif: returns the object as-is, possibly cloned.

2. formula, data.frame(), matrix(), and DataBackend: provides an alternative to the constructor of TaskClassif.

3. TaskRegr: Calls convert_task().

Note that the target column will be converted to a factor(), if possible.

Usage

as_task_classif(x, ...)

## S3 method for class 'TaskClassif'
as_task_classif(x, clone = FALSE, ...)

## S3 method for class 'data.frame'
as_task_classif(
  x,
  target,
  id = deparse1(substitute(x)),
  positive = NULL,
  label = NA_character_,
  ...
)

## S3 method for class 'matrix'
as_task_classif(
  x,
  target,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'DataBackend'
as_task_classif(
  x,
  target,
  id = deparse1(substitute(x)),
  positive = NULL,
  label = NA_character_,
  ...
)

## S3 method for class 'TaskRegr'
as_task_classif(
  x,
  target,
  drop_original_target = FALSE,
  drop_levels = TRUE,
  ...
)

## S3 method for class 'formula'
as_task_classif(
  x,
  data,
  id = deparse1(substitute(data)),
  positive = NULL,
  label = NA_character_,
  ...
)

Arguments

Value

TaskClassif.

Examples

as_task_classif(palmerpenguins::penguins, target = "species")

Description

Convert object to a TaskRegr. This is a S3 generic. mlr3 ships with methods for the following objects:

1. TaskRegr: returns the object as-is, possibly cloned.

2. formula, data.frame(), matrix(), and DataBackend: provides an alternative to the constructor of TaskRegr.

3. TaskClassif: Calls convert_task().

Usage

as_task_regr(x, ...)

## S3 method for class 'TaskRegr'
as_task_regr(x, clone = FALSE, ...)

## S3 method for class 'data.frame'
as_task_regr(
  x,
  target,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'matrix'
as_task_regr(
  x,
  target,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'DataBackend'
as_task_regr(
  x,
  target,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'TaskClassif'
as_task_regr(x, target, drop_original_target = FALSE, drop_levels = TRUE, ...)

## S3 method for class 'formula'
as_task_regr(
  x,
  data,
  id = deparse1(substitute(data)),
  label = NA_character_,
  ...
)

Arguments

Value

TaskRegr.

Examples

as_task_regr(datasets::mtcars, target = "mpg")

Description

Convert object to a TaskUnsupervised or a list of TaskUnsupervised.

Usage

as_task_unsupervised(x, ...)

## S3 method for class 'Task'
as_task_unsupervised(x, clone = FALSE, ...)

## S3 method for class 'data.frame'
as_task_unsupervised(
  x,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

## S3 method for class 'DataBackend'
as_task_unsupervised(
  x,
  id = deparse1(substitute(x)),
  label = NA_character_,
  ...
)

as_tasks_unsupervised(x, ...)

## S3 method for class 'list'
as_tasks_unsupervised(x, clone = FALSE, ...)

## S3 method for class 'Task'
as_tasks_unsupervised(x, clone = FALSE, ...)

Arguments

Description

Assertions for CallbackResample class.

Usage

assert_resample_callback(callback, null_ok = FALSE)

assert_resample_callbacks(callbacks, null_ok = FALSE)

Arguments

Value

CallbackResample | List of CallbackResamples.

Description

Runs a benchmark on arbitrary combinations of tasks (Task), learners (Learner), and resampling strategies (Resampling), possibly in parallel.

For large-scale benchmarking we recommend to use the mlr3batchmark package. This package runs benchmark experiments on high-performance computing clusters and handles failed experiments.

Usage

benchmark(
  design,
  store_models = FALSE,
  store_backends = TRUE,
  encapsulate = NA_character_,
  allow_hotstart = FALSE,
  clone = c("task", "learner", "resampling"),
  unmarshal = TRUE,
  callbacks = NULL
)

Arguments

Value

BenchmarkResult.

Stochasticity

Note that uninstantiated Resamplings are instantiated on the task, making the function stochastic even in case of deterministic learners.

Predict Sets

If you want to compare the performance of a learner on the training with the performance on the test set, you have to configure the Learner to predict on multiple sets by setting the field predict_sets to c("train", "test") (default is "test"). Each set yields a separate Prediction object during resampling. In the next step, you have to configure the measures to operate on the respective Prediction object:

m1 = msr("classif.ce", id = "ce.train", predict_sets = "train")
m2 = msr("classif.ce", id = "ce.test", predict_sets = "test")

The (list of) created measures can finally be passed to ⁠$aggregate()⁠ or ⁠$score()⁠.

Parallelization

This function can be parallelized with the future or mirai package. One job is one resampling iteration. All jobs are send to an apply function from future.apply or mirai::mirai_map() in a single batch. To select a parallel backend, use future::plan(). To use mirai, call mirai::daemons(.compute = "mlr3_parallelization") before calling this function. The future package guarantees reproducible results independent of the parallel backend. The results of mirai will not be the same but can be made reproducible by setting a seed when calling mirai::daemons(). More on parallelization can be found in the book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html

Progress Bars

This function supports progress bars via the package progressr. Simply wrap the function call in progressr::with_progress() to enable them. Alternatively, call progressr::handlers() with global = TRUE to enable progress bars globally. We recommend the progress package as backend which can be enabled with progressr::handlers("progress").

Logging

The mlr3 uses the lgr package for logging. lgr supports multiple log levels which can be queried with getOption("lgr.log_levels").

To suppress output and reduce verbosity, you can lower the log from the default level "info" to "warn":

lgr::get_logger("mlr3")$set_threshold("warn")

To get additional log output for debugging, increase the log level to "debug" or "trace":

lgr::get_logger("mlr3")$set_threshold("debug")

To log to a file or a data base, see the documentation of lgr::lgr-package.

Note

The fitted models are discarded after the predictions have been scored in order to reduce memory consumption. If you need access to the models for later analysis, set store_models to TRUE.

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter3/evaluation_and_benchmarking.html#sec-benchmarking

• Package mlr3viz for some generic visualizations.

• mlr3benchmark for post-hoc analysis of benchmark results.

Other benchmark: BenchmarkResult, benchmark_grid()

Examples

# benchmarking with benchmark_grid()
tasks = lapply(c("penguins", "sonar"), tsk)
learners = lapply(c("classif.featureless", "classif.rpart"), lrn)
resamplings = rsmp("cv", folds = 3)

design = benchmark_grid(tasks, learners, resamplings)
print(design)

set.seed(123)
bmr = benchmark(design)

## Data of all resamplings
head(as.data.table(bmr))

## Aggregated performance values
aggr = bmr$aggregate()
print(aggr)

## Extract predictions of first resampling result
rr = aggr$resample_result[[1]]
as.data.table(rr$prediction())

# Benchmarking with a custom design:
# - fit classif.featureless on penguins with a 3-fold CV
# - fit classif.rpart on sonar using a holdout
tasks = list(tsk("penguins"), tsk("sonar"))
learners = list(lrn("classif.featureless"), lrn("classif.rpart"))
resamplings = list(rsmp("cv", folds = 3), rsmp("holdout"))

design = data.table::data.table(
  task = tasks,
  learner = learners,
  resampling = resamplings
)

## Instantiate resamplings
design$resampling = Map(
  function(task, resampling) resampling$clone()$instantiate(task),
  task = design$task, resampling = design$resampling
)

## Run benchmark
bmr = benchmark(design)
print(bmr)

## Get the training set of the 2nd iteration of the featureless learner on penguins
rr = bmr$aggregate()[learner_id == "classif.featureless"]$resample_result[[1]]
rr$resampling$train_set(2)

Description

Takes a lists of Task, a list of Learner and a list of Resampling to generate a design in an expand.grid() fashion (a.k.a. cross join or Cartesian product).

There are two modes of operation, depending on the flag paired.

• With paired set to FALSE (default), resampling strategies are not allowed to be instantiated, and instead will be instantiated per task internally. The only exception to this rule applies if all tasks have exactly the same row ids, and the resamplings are all instantiated for such tasks. The grid will be generated based on the Cartesian product of tasks, learners, and resamplings. Because the resamplings are instantiated on the tasks, reproducibility requires a seed to be set before calling this function, as this process is stochastic.

• With paired set to TRUE, tasks and resamplings are treated as pairs. This means that you must provide as many tasks as corresponding instantiated resamplings. The grid will be generated based on the Cartesian product of learners and pairs.

Usage

benchmark_grid(
  tasks,
  learners,
  resamplings,
  param_values = NULL,
  paired = FALSE
)

Arguments

Value

(data.table::data.table()) with the cross product of the input vectors.

Errors and Warnings

• Mlr3WarningVaryingPredictTypes: This warning will be thrown if the learners have different predict_types.

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter3/evaluation_and_benchmarking.html#sec-benchmarking

• Package mlr3viz for some generic visualizations.

• mlr3benchmark for post-hoc analysis of benchmark results.

Other benchmark: BenchmarkResult, benchmark()

Examples

tasks = list(tsk("penguins"), tsk("sonar"))
learners = list(lrn("classif.featureless"), lrn("classif.rpart"))
resamplings = list(rsmp("cv"), rsmp("subsampling"))

# Set a seed to ensure reproducibility of the resampling instantiation
set.seed(123)
grid = benchmark_grid(tasks, learners, resamplings)
# the resamplings are now instantiated
head(grid$resampling[[1]]$instance)
print(grid)
## Not run: 
benchmark(grid)

## End(Not run)

# paired
learner = lrn("classif.rpart")
task1 = tsk("penguins")
task2 = tsk("german_credit")
res1 = rsmp("holdout")
res2 = rsmp("holdout")
res1$instantiate(task1)
res2$instantiate(task2)
design = benchmark_grid(list(task1, task2), learner, list(res1, res2), paired = TRUE)
print(design)

# manual construction of the grid with data.table::CJ()
grid = data.table::CJ(
 task = tasks,
  learner = learners,
  resampling = resamplings,
  sorted = FALSE
)

# manual instantiation (not suited for a fair comparison of learners!)
Map(function(task, resampling) {
  resampling$instantiate(task)
}, task = grid$task, resampling = grid$resampling)
## Not run: 
benchmark(grid)

## End(Not run)

Description

This is the result container object returned by benchmark(). A BenchmarkResult consists of the data of multiple ResampleResults. The contents of a BenchmarkResult and ResampleResult are almost identical and the stored ResampleResults can be extracted via the ⁠$resample_result(i)⁠ method, where i is the index of the performed resample experiment. This allows us to investigate the extracted ResampleResult and individual resampling iterations, as well as the predictions and models from each fold.

BenchmarkResults can be visualized via mlr3viz's autoplot() function.

For statistical analysis of benchmark results and more advanced plots, see mlr3benchmark.

S3 Methods

• as.data.table(rr, ..., reassemble_learners = TRUE, convert_predictions = TRUE, predict_sets = "test", task_characteristics = FALSE)
  BenchmarkResult -> data.table::data.table()
  Returns a tabular view of the internal data.

• c(...)
  (BenchmarkResult, ...) -> BenchmarkResult
  Combines multiple objects convertible to BenchmarkResult into a new BenchmarkResult.

Active bindings

Methods

Public methods

• BenchmarkResult$new()

• BenchmarkResult$help()

• BenchmarkResult$format()

• BenchmarkResult$print()

• BenchmarkResult$combine()

• BenchmarkResult$marshal()

• BenchmarkResult$unmarshal()

• BenchmarkResult$score()

• BenchmarkResult$obs_loss()

• BenchmarkResult$aggregate()

• BenchmarkResult$filter()

• BenchmarkResult$resample_result()

• BenchmarkResult$discard()

• BenchmarkResult$set_threshold()

• BenchmarkResult$clone()

Method new()

Creates a new instance of this R6 class.

Usage

BenchmarkResult$new(data = NULL)

Arguments

Method help()

Opens the help page for this object.

Usage

BenchmarkResult$help()

Method format()

Helper for print outputs.

Usage

BenchmarkResult$format(...)

Arguments

Method print()

Printer.

Usage

BenchmarkResult$print()

Method combine()

Fuses a second BenchmarkResult into itself, mutating the BenchmarkResult in-place. If the second BenchmarkResult bmr is NULL, simply returns self. Note that you can alternatively use the combine function c() which calls this method internally.

Usage

BenchmarkResult$combine(bmr)

Arguments

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keep the object in its previous state.

Method marshal()

Marshals all stored models.

Usage

BenchmarkResult$marshal(...)

Arguments

Examples

bmr$marshal()

Method unmarshal()

Unmarshals all stored models.

Usage

BenchmarkResult$unmarshal(...)

Arguments

Examples

bmr$unmarshal()

Method score()

Returns a table with one row for each resampling iteration, including all involved objects: Task, Learner, Resampling, iteration number (integer(1)), and Prediction. If ids is set to TRUE, character column of extracted ids are added to the table for convenient filtering: "task_id", "learner_id", and "resampling_id".

Additionally calculates the provided performance measures and binds the performance scores as extra columns. These columns are named using the id of the respective Measure.

Usage

BenchmarkResult$score(
  measures = NULL,
  ids = TRUE,
  conditions = FALSE,
  predictions = TRUE
)

Arguments

Returns

data.table::data.table().

Examples

bmr$score(msr("classif.acc"))

Method obs_loss()

Calculates the observation-wise loss via the Measure's obs_loss method. Returns a data.table() with columns from the predictions (e.g., row_ids, truth, response, etc.), plus one numeric column for each measure, named with the respective measure id, and a resample_result column. If there is no observation-wise loss function for the measure, the column is filled with NA_real_ values. Note that some measures such as RMSE, do have an ⁠$obs_loss⁠, but they require an additional transformation after aggregation, in this example taking the square-root.

Usage

BenchmarkResult$obs_loss(measures = NULL, predict_sets = "test")

Arguments

Examples

bmr$obs_loss(msr("classif.acc"))

Method aggregate()

Returns a result table where resampling iterations are combined into ResampleResults. A column with the aggregated performance score is added for each Measure, named with the id of the respective measure.

The method for aggregation is controlled by the Measure, e.g. micro aggregation, macro aggregation or custom aggregation. Most measures default to macro aggregation.

Note that the aggregated performances just give a quick impression which approaches work well and which approaches are probably underperforming. However, the aggregates do not account for variance and cannot replace a statistical test. See mlr3viz to get a better impression via boxplots or mlr3benchmark for critical difference plots and significance tests.

For convenience, different flags can be set to extract more information from the returned ResampleResult.

Usage

BenchmarkResult$aggregate(
  measures = NULL,
  ids = TRUE,
  uhashes = FALSE,
  params = FALSE,
  conditions = FALSE
)

Arguments

Returns

data.table::data.table().

Examples

bmr$aggregate()

Method filter()

Subsets the benchmark result. You can either directly provide the row IDs or the uhashes of the resample results to keep, or use the learner_ids, task_ids and resampling_ids arguments to filter for learner, task and resampling IDs. The three options are mutually exclusive.

Usage

BenchmarkResult$filter(
  i = NULL,
  uhashes = NULL,
  learner_ids = NULL,
  task_ids = NULL,
  resampling_ids = NULL
)

Arguments

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keep the object in its previous state.

Examples

design = benchmark_grid(
  tsks(c("iris", "sonar")),
  lrns(c("classif.debug", "classif.featureless")),
  rsmp("holdout")
)
bmr = benchmark(design)
bmr
bmr2 = bmr$clone(deep = TRUE)
bmr2$filter(learner_ids = "classif.featureless")
bmr2

Method resample_result()

Retrieve the i-th ResampleResult, by position, by unique hash uhash or by learner, task and resampling IDs. All three options are mutually exclusive.

Usage

BenchmarkResult$resample_result(
  i = NULL,
  uhash = NULL,
  task_id = NULL,
  learner_id = NULL,
  resampling_id = NULL
)

Arguments

Returns

ResampleResult.

Examples

design = benchmark_grid(
  tsk("iris"),
  lrns(c("classif.debug", "classif.featureless")),
  rsmp("holdout")
)
bmr = benchmark(design)
bmr$resample_result(learner_id = "classif.featureless")
bmr$resample_result(i = 1)
bmr$resample_result(uhash = uhashes(bmr, learner_id = "classif.debug"))

Method discard()

Shrinks the BenchmarkResult by discarding parts of the internally stored data. Note that certain operations might stop work, e.g. extracting importance values from learners or calculating measures requiring the task's data.

Usage

BenchmarkResult$discard(backends = FALSE, models = FALSE)

Arguments

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keep the object in its previous state.

Examples

bmr$discard(models = TRUE)

Method set_threshold()

Sets the threshold for the response prediction of classification learners, given they have output a probability prediction for a binary classification task.

The resample results for which to change the threshold can either be specified directly via uhashes, by selecting the specific iterations (i) or by filtering according to learner, task and resampling IDs.

If none of the three options is specified, the threshold is set for all resample results.

Usage

BenchmarkResult$set_threshold(
  threshold,
  i = NULL,
  uhashes = NULL,
  learner_ids = NULL,
  task_ids = NULL,
  resampling_ids = NULL,
  ties_method = "random"
)

Arguments

Examples

design = benchmark_grid(
  tsk("sonar"),
  lrns(c("classif.debug", "classif.featureless"), predict_type = "prob"),
  rsmp("holdout")
)
bmr = benchmark(design)
bmr$set_threshold(0.8, learner_ids = "classif.featureless")
bmr$set_threshold(0.3, i = 2)
bmr$set_threshold(0.7, uhashes = uhashes(bmr, learner_ids = "classif.featureless"))

Method clone()

The objects of this class are cloneable with this method.

Usage

BenchmarkResult$clone(deep = FALSE)

Arguments

Note

All stored objects are accessed by reference. Do not modify any extracted object without cloning it first.

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter3/evaluation_and_benchmarking.html#sec-benchmarking

• Package mlr3viz for some generic visualizations.

• mlr3benchmark for post-hoc analysis of benchmark results.

Other benchmark: benchmark(), benchmark_grid()

Examples

set.seed(123)
learners = list(
  lrn("classif.featureless", predict_type = "prob"),
  lrn("classif.rpart", predict_type = "prob")
)

design = benchmark_grid(
  tasks = list(tsk("sonar"), tsk("penguins")),
  learners = learners,
  resamplings = rsmp("cv", folds = 3)
)
print(design)

bmr = benchmark(design)
print(bmr)

bmr$tasks
bmr$learners

# first 5 resampling iterations
head(as.data.table(bmr, measures = c("classif.acc", "classif.auc")), 5)

# aggregate results
bmr$aggregate()

# aggregate results with hyperparameters as separate columns
mlr3misc::unnest(bmr$aggregate(params = TRUE), "params")

# extract resample result for classif.rpart
rr = bmr$aggregate()[learner_id == "classif.rpart", resample_result][[1]]
print(rr)

# access the confusion matrix of the first resampling iteration
rr$predictions()[[1]]$confusion

# reduce to subset with task id "sonar"
bmr$filter(task_ids = "sonar")
print(bmr)

## ------------------------------------------------
## Method `BenchmarkResult$marshal`
## ------------------------------------------------

bmr$marshal()

## ------------------------------------------------
## Method `BenchmarkResult$unmarshal`
## ------------------------------------------------

bmr$unmarshal()

## ------------------------------------------------
## Method `BenchmarkResult$score`
## ------------------------------------------------

bmr$score(msr("classif.acc"))

## ------------------------------------------------
## Method `BenchmarkResult$obs_loss`
## ------------------------------------------------

bmr$obs_loss(msr("classif.acc"))

## ------------------------------------------------
## Method `BenchmarkResult$aggregate`
## ------------------------------------------------

bmr$aggregate()

## ------------------------------------------------
## Method `BenchmarkResult$filter`
## ------------------------------------------------

design = benchmark_grid(
  tsks(c("iris", "sonar")),
  lrns(c("classif.debug", "classif.featureless")),
  rsmp("holdout")
)
bmr = benchmark(design)
bmr
bmr2 = bmr$clone(deep = TRUE)
bmr2$filter(learner_ids = "classif.featureless")
bmr2

## ------------------------------------------------
## Method `BenchmarkResult$resample_result`
## ------------------------------------------------

design = benchmark_grid(
  tsk("iris"),
  lrns(c("classif.debug", "classif.featureless")),
  rsmp("holdout")
)
bmr = benchmark(design)
bmr$resample_result(learner_id = "classif.featureless")
bmr$resample_result(i = 1)
bmr$resample_result(uhash = uhashes(bmr, learner_id = "classif.debug"))

## ------------------------------------------------
## Method `BenchmarkResult$discard`
## ------------------------------------------------

bmr$discard(models = TRUE)

## ------------------------------------------------
## Method `BenchmarkResult$set_threshold`
## ------------------------------------------------

design = benchmark_grid(
  tsk("sonar"),
  lrns(c("classif.debug", "classif.featureless"), predict_type = "prob"),
  rsmp("holdout")
)
bmr = benchmark(design)
bmr$set_threshold(0.8, learner_ids = "classif.featureless")
bmr$set_threshold(0.3, i = 2)
bmr$set_threshold(0.7, uhashes = uhashes(bmr, learner_ids = "classif.featureless"))

Description

A regression task to predict the median house value in California.

Contains 9 features and 20640 observations. Target column is "median_house_value".

Format

R6::R6Class inheriting from TaskRegr.

Construction

mlr_tasks$get("california_housing")
tsk("california_housing")

Meta Information

• Task type: “regr”

• Dimensions: 20640x10

• Properties: -

• Has Missings: TRUE

• Target: “median_house_value”

• Features: “households”, “housing_median_age”, “latitude”, “longitude”, “median_income”, “ocean_proximity”, “population”, “total_bedrooms”, “total_rooms”

Source

https://www.kaggle.com/datasets/camnugent/california-housing-prices

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html

• Package mlr3data for more toy tasks.

• Package mlr3oml for downloading tasks from https://www.openml.org.

• Package mlr3viz for some generic visualizations.

• Dictionary of Tasks: mlr_tasks

• as.data.table(mlr_tasks) for a table of available Tasks in the running session (depending on the loaded packages).

• mlr3fselect and mlr3filters for feature selection and feature filtering.

• Extension packages for additional task types:

  • Unsupervised clustering: mlr3cluster

  • Probabilistic supervised regression and survival analysis: https://mlr3proba.mlr-org.com/.

Other Task: Task, TaskClassif, TaskRegr, TaskSupervised, TaskUnsupervised, mlr_tasks, mlr_tasks_breast_cancer, mlr_tasks_german_credit, mlr_tasks_iris, mlr_tasks_mtcars, mlr_tasks_penguins, mlr_tasks_pima, mlr_tasks_sonar, mlr_tasks_spam, mlr_tasks_wine, mlr_tasks_zoo

Description

Function to create a CallbackResample. Predefined callbacks are stored in the dictionary mlr_callbacks and can be retrieved with clbk().

Evaluation callbacks are called at different stages of the resampling process. Each stage is called once per resampling iteration. The stages are prefixed with ⁠on_resample_*⁠. The text in brackets indicates what happens between the stages in the internal workhorse() function and which accesses to the ContextResample (ctx) are typical for the stage.

Start Resampling Iteration on Worker
 - on_resample_begin
   (Split `ctx$task` into training and test set with `ctx$resampling` and `ctx$iteration`)
 - on_resample_before_train
   (Train the learner `ctx$learner` on training data)
 - on_resample_before_predict
   (Predict on predict sets and store prediction data `ctx$pdatas`)
 - on_resample_end
   (Erase model `ctx$learner$model` if requested and return results)
End Resampling Iteration on Worker

The callback can store data in ctx$learner$state or ctx$data_extra. The data in ctx$data_extra is stored in the ResampleResult or BenchmarkResult. See also the section on parameters for more information on the stages.

Usage

callback_resample(
  id,
  label = NA_character_,
  man = NA_character_,
  on_resample_begin = NULL,
  on_resample_before_train = NULL,
  on_resample_before_predict = NULL,
  on_resample_end = NULL
)

Arguments

Details

When implementing a callback, each function must have two arguments named callback and context. A callback can write data to the state (⁠$state⁠), e.g. settings that affect the callback itself. We highly discourage changing the task, learner and resampling objects via the callback.

Examples

learner = lrn("classif.rpart")
task = tsk("pima")
resampling = rsmp("cv", folds = 3)

# save selected features callback
callback = callback_resample("selected_features",
 on_resample_end = function(callback, context) {
    context$learner$state$selected_features = context$learner$selected_features()
  }
)

rr = resample(task, learner, resampling, callbacks = callback)
rr$learners[[1]]$state$selected_features

# holdout task callback
callback = callback_resample("holdout_task",
  on_resample_before_predict = function(callback, context) {
    pred = context$learner$predict(callback$state$task)
    context$data_extra = list(prediction_holdout = pred)
  }
)

task_holdout = tsk("pima")
splits = partition(task, 0.7)
task$filter(splits$train)
task_holdout$filter(splits$test)

callback$state$task = task_holdout

rr = resample(task, learner, resampling, callbacks = callback)
rr$data_extra

Description

Specialized mlr3misc::Callback to customize the behavior of resample() and benchmark() in mlr3. For example, callbacks can be used to extract information from models on the worker or to store intermediate results to disk. The callback_resample() function is used to create instances of this class. Predefined callbacks are stored in the dictionary mlr_callbacks and can be retrieved with clbk(). For more information on callbacks, see the callback_resample() documentation.

Super class

mlr3misc::Callback -> CallbackResample

Public fields

Methods

Public methods

• CallbackResample$clone()

Method clone()

The objects of this class are cloneable with this method.

Usage

CallbackResample$clone(deep = FALSE)

Arguments

Description

A CallbackResample accesses and modifies data during resample() and benchmark() via the ContextResample. See the section on fields for a list of modifiable objects. See callback_resample() for a list of stages that access ContextResample.

Super class

mlr3misc::Context -> ContextResample

Active bindings

Methods

Public methods

• ContextResample$new()

• ContextResample$clone()

Method new()

Creates a new instance of this R6 class.

Usage

ContextResample$new(task, learner, resampling, iteration)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

ContextResample$clone(deep = FALSE)

Arguments

Description

The task's target is replaced by a different column from the data.

Usage

convert_task(
  intask,
  target = NULL,
  new_type = NULL,
  drop_original_target = FALSE,
  drop_levels = TRUE
)

Arguments

Value

Task of requested type.

Description

This is the abstract base class for data backends.

Data backends provide a layer of abstraction for various data storage systems. It is not recommended to work directly with the DataBackend. Instead, all data access is handled transparently via the Task.

This package currently ships with one implementation for backends:

• DataBackendDataTable which stores the data as data.table::data.table().

To connect to out-of-memory database management systems such as SQL servers, see the extension package mlr3db.

Details

The required set of fields and methods to implement a custom DataBackend is listed in the respective sections (see DataBackendDataTable).

Public fields

Active bindings

Methods

Public methods

• DataBackend$new()

• DataBackend$format()

• DataBackend$print()

Method new()

Creates a new instance of this R6 class.

Note: This object is typically constructed via a derived classes, e.g. DataBackendDataTable, or via the S3 method as_data_backend().

Usage

DataBackend$new(data, primary_key)

Arguments

Method format()

Helper for print outputs.

Usage

DataBackend$format(...)

Arguments

Method print()

Printer.

Usage

DataBackend$print()

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-backends

• Package mlr3db to interface out-of-memory data, e.g. SQL servers or duckdb.

Other DataBackend: DataBackendDataTable, as_data_backend()

Examples

data = data.table::data.table(id = 1:5, x = runif(5),
  y = sample(letters[1:3], 5, replace = TRUE))

b = DataBackendDataTable$new(data, primary_key = "id")
print(b)
b$head(2)
b$data(rows = 1:2, cols = "x")
b$distinct(rows = b$rownames, "y")
b$missings(rows = b$rownames, cols = names(data))

Description

DataBackend for data.table which serves as an efficient in-memory data base.

Super class

mlr3::DataBackend -> DataBackendDataTable

Public fields

Active bindings

Methods

Public methods

• DataBackendDataTable$new()

• DataBackendDataTable$data()

• DataBackendDataTable$head()

• DataBackendDataTable$distinct()

• DataBackendDataTable$missings()

Method new()

Creates a new instance of this R6 class.

Note that DataBackendDataTable does not copy the input data, while as_data_backend() calls data.table::copy(). as_data_backend() also takes care about casting to a data.table() and adds a primary key column if necessary.

Usage

DataBackendDataTable$new(data, primary_key)

Arguments

Method data()

Returns a slice of the data. The rows must be addressed as vector of primary key values, columns must be referred to via column names. Queries for rows with no matching row id and queries for columns with no matching column name are silently ignored. Rows are guaranteed to be returned in the same order as rows, columns may be returned in an arbitrary order. Duplicated row ids result in duplicated rows, duplicated column names lead to an exception.

Usage

DataBackendDataTable$data(rows, cols)

Arguments

Method head()

Retrieve the first n rows.

Usage

DataBackendDataTable$head(n = 6L)

Arguments

Returns

data.table::data.table() of the first n rows.

Method distinct()

Returns a named list of vectors of distinct values for each column specified. If na_rm is TRUE, missing values are removed from the returned vectors of distinct values. Non-existing rows and columns are silently ignored.

Usage

DataBackendDataTable$distinct(rows, cols, na_rm = TRUE)

Arguments

Returns

Named list() of distinct values.

Method missings()

Returns the number of missing values per column in the specified slice of data. Non-existing rows and columns are silently ignored.

Usage

DataBackendDataTable$missings(rows, cols)

Arguments

Returns

Total of missing values per column (named numeric()).

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-backends

• Package mlr3db to interface out-of-memory data, e.g. SQL servers or duckdb.

Other DataBackend: DataBackend, as_data_backend()

Examples

data = as.data.table(palmerpenguins::penguins)
data$id = seq_len(nrow(palmerpenguins::penguins))
b = DataBackendDataTable$new(data = data, primary_key = "id")
print(b)
b$head()
b$data(rows = 100:101, cols = "species")

b$nrow
head(b$rownames)

b$ncol
b$colnames

# alternative construction
as_data_backend(palmerpenguins::penguins)

Description

Create a fallback learner for a given learner. The function searches for a suitable fallback learner based on the task type. Additional checks are performed to ensure that the fallback learner supports the predict type.

Usage

default_fallback(learner, ...)

## S3 method for class 'Learner'
default_fallback(learner, ...)

## S3 method for class 'LearnerClassif'
default_fallback(learner, ...)

## S3 method for class 'LearnerRegr'
default_fallback(learner, ...)

Arguments

Value

Learner

Description

Gets the default measures using the information in mlr_reflections$default_measures:

• "classif.ce" for classification ("classif").

• "regr.mse" for regression ("regr").

• Add-on package may register additional default measures for their own task types.

Usage

default_measures(task_type)

Arguments

Value

list of Measure.

Examples

default_measures("classif")
default_measures("regr")

Description

This class stores learners for hot starting training, i.e. resuming or continuing from an already fitted model. We assume that hot starting is only possible if a single hyperparameter (also called the fidelity parameter, usually controlling the complexity or expensiveness) is altered and all other hyperparameters are identical.

The HotstartStack stores trained learners which can be potentially used to hot start a learner. Learner automatically hot start while training if a stack is attached to the ⁠$hotstart_stack⁠ field and the stack contains a suitable learner.

For example, if you want to train a random forest learner with 1000 trees but already have a random forest learner with 500 trees (hot start learner), you can add the hot start learner to the HotstartStack of the expensive learner with 1000 trees. If you now call the train() method (or resample() or benchmark()), a random forest with 500 trees will be fitted and combined with the 500 trees of the hotstart learner, effectively saving you to fit 500 trees.

Hot starting is only supported by learners which have the property "hotstart_forward" or "hotstart_backward". For example, an xgboost model (in mlr3learners) can hot start forward by adding more boosting iterations, and a random forest can go backwards by removing trees. The fidelity parameters are tagged with "hotstart" in learner's parameter set.

Public fields

Methods

Public methods

• HotstartStack$new()

• HotstartStack$add()

• HotstartStack$start_cost()

• HotstartStack$format()

• HotstartStack$print()

• HotstartStack$clone()

Method new()

Creates a new instance of this R6 class.

Usage

HotstartStack$new(learners = NULL, hotstart_threshold = NULL)

Arguments

Method add()

Add learners to hot start stack.

Usage

HotstartStack$add(learners)

Arguments

Returns

self (invisibly).

Method start_cost()

Calculates the cost for each learner of the stack to hot start the target learner.

The following cost values can be returned:

• NA_real_: Learner is unsuitable to hot start target learner.

• -1: Hotstart learner in the stack and target learner are identical.

• 0 Cost for hot starting backwards is always 0.

• ⁠> 0⁠ Cost for hot starting forward.

Usage

HotstartStack$start_cost(learner, task_hash)

Arguments

Method format()

Helper for print outputs.

Usage

HotstartStack$format(...)

Arguments

Method print()

Printer.

Usage

HotstartStack$print(...)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

HotstartStack$clone(deep = FALSE)

Arguments

Examples

# train learner on pima task
task = tsk("pima")
learner = lrn("classif.debug", iter = 1)
learner$train(task)

# initialize stack with previously fitted learner
hot = HotstartStack$new(list(learner))

# retrieve learner with increased fidelity parameter
learner = lrn("classif.debug", iter = 2)

# calculate cost of hot starting
hot$start_cost(learner, task$hash)

# add stack with hot start learner
learner$hotstart_stack = hot

# train automatically uses hot start learner while fitting the model
learner$train(task)

Description

extract_pkgs() extracts required package from various objects, including TaskGenerator, Learner, Measure and objects from extension packages such as mlr3pipelines or mlr3filters. If applied on a list, the function is called recursively on all elements.

install_pkgs() calls extract_pkgs() internally and proceeds with the installation of extracted packages.

Usage

install_pkgs(x, ...)

extract_pkgs(x)

## S3 method for class 'character'
extract_pkgs(x)

## S3 method for class 'R6'
extract_pkgs(x)

## S3 method for class 'list'
extract_pkgs(x)

## S3 method for class 'ResampleResult'
extract_pkgs(x)

## S3 method for class 'BenchmarkResult'
extract_pkgs(x)

Arguments

Details

If a package contains a forward slash ('/'), it is assumed to be a package hosted on GitHub in "<user>/<repo>" format, and the string will be passed to remotes::install_github(). Otherwise, the package name will be passed to remotes::install_cran().

Value

extract_pkgs() returns a character() of package strings, install_pkgs() returns the names of extracted packages invisibly.

Examples

extract_pkgs(lrns(c("regr.rpart", "regr.featureless")))

Description

This is the abstract base class for learner objects like LearnerClassif and LearnerRegr.

Learners are built around the three following key parts:

• Methods ⁠$train()⁠ and ⁠$predict()⁠ which call internal methods or private methods ⁠$.train()⁠/⁠$.predict()⁠).

• A paradox::ParamSet which stores meta-information about available hyperparameters, and also stores hyperparameter settings.

• Meta-information about the requirements and capabilities of the learner.

• The fitted model stored in field ⁠$model⁠, available after calling ⁠$train()⁠.

Predefined learners are stored in the dictionary mlr_learners, e.g. classif.rpart or regr.rpart.

More classification and regression learners are implemented in the add-on package mlr3learners. Learners for survival analysis (or more general, for probabilistic regression) can be found in mlr3proba. Unsupervised cluster algorithms are implemented in mlr3cluster. The dictionary mlr_learners gets automatically populated with the new learners as soon as the respective packages are loaded.

More (experimental) learners can be found in the GitHub repository: https://github.com/mlr-org/mlr3extralearners. A guide on how to extend mlr3 with custom learners can be found in the mlr3book.

To combine the learner with preprocessing operations like factor encoding, mlr3pipelines is recommended. Hyperparameters stored in the param_set can be tuned with mlr3tuning.

Optional Extractors

Specific learner implementations are free to implement additional getters to ease the access of certain parts of the model in the inherited subclasses.

For the following operations, extractors are standardized:

• importance(...): Returns the feature importance score as numeric vector. The higher the score, the more important the variable. The returned vector is named with feature names and sorted in decreasing order. Note that the model might omit features it has not used at all. The learner must be tagged with property "importance". To filter variables using the importance scores, see package mlr3filters.

• selected_features(...): Returns a subset of selected features as character(). The learner must be tagged with property "selected_features".

• oob_error(...): Returns the out-of-bag error of the model as numeric(1). The learner must be tagged with property "oob_error".

• internal_valid_scores: Returns the internal validation score(s) of the model as a named list(). Only available for Learners with the "validation" property. If the learner is not trained yet, this returns NULL.

• internal_tuned_values: Returns the internally tuned hyperparameters of the model as a named list(). Only available for Learners with the "internal_tuning" property. If the learner is not trained yet, this returns NULL.

Weights

Many learners support observation weights, indicated by their property "weights". The weights are stored in the Task where the column role weights_learner needs to be assigned to a single numeric column. If a task has weights and the learner supports them, they are used automatically. If a task has weights but the learner does not support them, an error is thrown by default. Both of these behaviors can be disabled by setting the use_weights field to "ignore". See the description of use_weights for more information.

If the learner is set-up to use weights but the task does not have a designated weight column, samples are considered to have equal weight. When weights are being used, they are passed down to the learner directly; the effect of weights depends on the specific learner. Generally, weights do not need to sum up to 1.

When implementing a Learner that uses weights, the "weights" property should be set. The ⁠$.train()⁠-method should then call the ⁠$.get_weights()⁠-method to retrieve the weights from the task. ⁠$.get_weights()⁠ will automatically discard weights when use_weights is set to "ignore";

Setting Hyperparameters

All information about hyperparameters is stored in the slot param_set which is a paradox::ParamSet. The printer gives an overview about the ids of available hyperparameters, their storage type, lower and upper bounds, possible levels (for factors), default values and assigned values. To set hyperparameters, call the set_values() method on the param_set:

lrn = lrn("classif.rpart")
lrn$param_set$set_values(minsplit = 3, cp = 0.01)

Note that this operation replaces all previously set hyperparameter values. If you only intend to change one specific hyperparameter value and leave the others as-is, you can use the helper function mlr3misc::insert_named():

lrn$param_set$values = mlr3misc::insert_named(lrn$param_set$values, list(cp = 0.001))

If the learner has additional hyperparameters which are not encoded in the ParamSet, you can easily extend the learner. Here, we add a factor hyperparameter with id "foo" and possible levels "a" and "b":

lrn$param_set$add(paradox::ParamFct$new("foo", levels = c("a", "b")))

Implementing Validation

Some Learners, such as XGBoost, other boosting algorithms, or deep learning models (mlr3torch), utilize validation data during the training to prevent overfitting or to log the validation performance. It is possible to configure learners to be able to receive such an independent validation set during training. To do so, one must:

• annotate the learner with the "validation" property

• implement the active binding ⁠$internal_valid_scores⁠ (see section Optional Extractors), as well as the private method ⁠$.extract_internal_valid_scores()⁠ which returns the (final) internal validation scores from the model of the Learner and returns them as a named list() of numeric(1). If the model is not trained yet, this method should return NULL.

• Add the validate parameter, which can be either NULL, a ratio in $(0, 1)$, "test", or "predefined":

  • NULL: no validation

  • ratio: only proportion 1 - ratio of the task is used for training and ratio is used for validation.

  • "test" means that the "test" task is used. Warning: This can lead to biased performance estimation. This option is only available if the learner is being trained via resample(), benchmark() or functions that internally use them, e.g. tune() of mlr3tuning or batchmark() of mlr3batchmark. This is especially useful for hyperparameter tuning, where one might e.g. want to use the same validation data for early stopping and model evaluation.

  • "predefined" means that the task's (manually set) ⁠$internal_valid_task⁠ is used. See the Task documentation for more information.

For an example how to do this, see LearnerClassifDebug. Note that in .train(), the ⁠$internal_valid_task⁠ will only be present if the ⁠$validate⁠ field of the Learner is set to a non-NULL value.

Implementing Internal Tuning

Some learners such as XGBoost or cv.glmnet can internally tune hyperparameters. XGBoost, for example, can tune the number of boosting rounds based on the validation performance. CV Glmnet, on the other hand, can tune the regularization parameter based on an internal cross-validation. Internal tuning can therefore rely on the internal validation data, but does not necessarily do so.

In order to be able to combine this internal hyperparameter tuning with the standard hyperparameter optimization implemented via mlr3tuning, one must:

• annotate the learner with the "internal_tuning" property

• implement the active binding ⁠$internal_tuned_values⁠ (see section Optional Extractors) as well as the private method ⁠$.extract_internal_tuned_values()⁠ which extracts the internally tuned values from the Learner's model and returns them as a named list(). If the model is not trained yet, this method should return NULL.

• Have at least one parameter tagged with "internal_tuning", which requires to also provide a in_tune_fn and disable_tune_fn, and should also include a default aggregation function.

For an example how to do this, see LearnerClassifDebug.

Implementing Marshaling

Some Learners have models that cannot be serialized as they e.g. contain external pointers. In order to still be able to save them, use them with parallelization or callr encapsulation it is necessary to implement how they should be (un)-marshaled. See marshaling for how to do this.

Implementing Out-of-Bag Error

Some Learners can compute the out-of-bag error during training. In order to do this, the learner must:

• annotate the learner with the "oob_error" property

• implement the private method ⁠$.extract_oob_error()⁠ which extracts the out-of-bag error from the Learner's model and returns it as a numeric(1).

Public fields

Active bindings

Methods

Public methods

• Learner$new()

• Learner$format()

• Learner$print()

• Learner$help()

• Learner$train()

• Learner$predict()

• Learner$predict_newdata()

• Learner$reset()

• Learner$base_learner()

• Learner$encapsulate()

• Learner$configure()

• Learner$selected_features()

• Learner$clone()

Method new()

Creates a new instance of this R6 class.

Note that this object is typically constructed via a derived classes, e.g. LearnerClassif or LearnerRegr.

Usage

Learner$new(
  id,
  task_type,
  param_set = ps(),
  predict_types = character(),
  feature_types = character(),
  properties = character(),
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments

Method format()

Helper for print outputs.

Usage

Learner$format(...)

Arguments

Method print()

Printer.

Usage

Learner$print(...)

Arguments

Method help()

Opens the corresponding help page referenced by field ⁠$man⁠.

Usage

Learner$help()

Method train()

Train the learner on a set of observations of the provided task. Mutates the learner by reference, i.e. stores the model alongside other information in field ⁠$state⁠.

Usage

Learner$train(task, row_ids = NULL)

Arguments

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keep the object in its previous state.

Examples

task   = tsk("penguins")
learner = lrn("classif.rpart")
learner$train(task)

Method predict()

Uses the fitted model stored in ⁠$state⁠ to generate predictions for a set of observations from the provided task. This method requires that the learner has been previously trained using ⁠$train()⁠.

Usage

Learner$predict(task, row_ids = NULL)

Arguments

Returns

Prediction object containing the predictions for the specified observations.

Examples

task = tsk("penguins")
learner = lrn("classif.rpart")$train(task)
learner$predict(task)

Method predict_newdata()

Uses the model fitted during ⁠$train()⁠ to create a new Prediction based on the new data in newdata. Object task is the task used during ⁠$train()⁠ and required for conversion of newdata. If the learner's ⁠$train()⁠ method has been called, there is a (size reduced) version of the training task stored in the learner. If the learner has been fitted via resample() or benchmark(), you need to pass the corresponding task stored in the ResampleResult or BenchmarkResult, respectively. Further, auto_convert is used for type-conversions to ensure compatibility of features between ⁠$train()⁠ and ⁠$predict()⁠.

If the stored training task has a weights_measure column, and if newdata contains a column with the same name, that column must be numeric with no missing values and is used as measure weights column. Otherwise, no measure weights are used.

Usage

Learner$predict_newdata(newdata, task = NULL)

Arguments

Returns

Prediction.

Examples

task = tsk("penguins")
learner = lrn("classif.rpart")$train(task)
learner$predict_newdata(task$data(rows = 1:5))

Method reset()

Reset the learner, i.e. un-train by resetting the state.

Usage

Learner$reset()

Returns

Returns the object itself, but modified by reference. You need to explicitly ⁠$clone()⁠ the object beforehand if you want to keep the object in its previous state.

Examples

task = tsk("penguins")
learner = lrn("classif.rpart")$train(task)
learner$reset()

Method base_learner()

Extracts the base learner from nested learner objects like GraphLearner in mlr3pipelines or AutoTuner in mlr3tuning. Returns the Learner itself for regular learners.

Usage

Learner$base_learner(recursive = Inf)

Arguments

Returns

Learner

Method encapsulate()

Sets the encapsulation method and fallback learner for the train and predict steps. There are currently four different methods implemented:

• "none": Just runs the learner in the current session and measures the elapsed time. Does not keep a log, output is printed directly to the console. Works well together with traceback().

• "try": Similar to "none", but catches error. Output is printed to the console and not logged.

• "evaluate": Uses the package evaluate to call the learner, measure time and do the logging.

• "callr": Uses the package callr to call the learner, measure time and do the logging. This encapsulation spawns a separate R session in which the learner is called. While this comes with a considerable overhead, it also guards your session from being torn down by segfaults.

• "mirai": Uses the package mirai to call the learner, measure time and do the logging. This encapsulation calls the function in a mirai on a daemon. The daemon can be pre-started via daemons(1, .compute = "mlr3_encapsulation"), otherwise a new R session will be created for each encapsulated call. If a daemon is already running with compute profile "mlr3_encapsulation", it will be used to execute all calls. Using ⁠mirai"⁠ is similarly safe as callr but much faster if several learners are encapsulated one after the other on the same daemon.

The fallback learner is fitted to create valid predictions in case that either the model fitting or the prediction of the original learner fails. If the training step or the predict step of the original learner fails, the fallback is used to make the predictions. If the original learner only partially fails during predict step (usually in the form of missing to predict some observations or producing some NA predictions), these missing predictions are imputed by the fallback. Note that the fallback is always trained, as we do not know in advance whether prediction will fail. If the training step fails, the ⁠$model⁠ field of the original learner is NULL. The results are reproducible across the different encapsulation methods.

Note that for errors of class Mlr3ErrorConfig, the function always errs and no fallback learner is trained.

Also see the section on error handling in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-error-handling

Usage

Learner$encapsulate(method, fallback = NULL, when = NULL)

Arguments

Returns

self (invisibly).

Examples

learner = lrn("classif.rpart")
fallback = lrn("classif.featureless")
learner$encapsulate("try", fallback = fallback)

Method configure()

Sets parameter values and fields of the learner. All arguments whose names match the name of a parameter of the paradox::ParamSet are set as parameters. All remaining arguments are assumed to be regular fields.

Usage

Learner$configure(..., .values = list())

Arguments

Examples

learner = lrn("classif.rpart")
learner$configure(minsplit = 3, parallel_predict = FALSE)
learner$configure(.values = list(cp = 0.005))

Method selected_features()

Returns the features selected by the model. The field selected_features_impute controls the behavior if the learner does not support feature selection. If set to "error", an error is thrown, otherwise all features are returned.

Usage

Learner$selected_features()

Method clone()

The objects of this class are cloneable with this method.

Usage

Learner$clone(deep = FALSE)

Arguments

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-learners

• Package mlr3learners for a solid collection of essential learners.

• Package mlr3extralearners for more learners.

• Dictionary of Learners: mlr_learners

• as.data.table(mlr_learners) for a table of available Learners in the running session (depending on the loaded packages).

• mlr3pipelines to combine learners with pre- and postprocessing steps.

• Package mlr3viz for some generic visualizations.

• Extension packages for additional task types:

  • mlr3proba for probabilistic supervised regression and survival analysis.

  • mlr3cluster for unsupervised clustering.

• mlr3tuning for tuning of hyperparameters, mlr3tuningspaces for established default tuning spaces.

Other Learner: LearnerClassif, LearnerRegr, mlr_learners, mlr_learners_classif.debug, mlr_learners_classif.featureless, mlr_learners_classif.rpart, mlr_learners_regr.debug, mlr_learners_regr.featureless, mlr_learners_regr.rpart

Examples

## ------------------------------------------------
## Method `Learner$train`
## ------------------------------------------------

task   = tsk("penguins")
learner = lrn("classif.rpart")
learner$train(task)

## ------------------------------------------------
## Method `Learner$predict`
## ------------------------------------------------

task = tsk("penguins")
learner = lrn("classif.rpart")$train(task)
learner$predict(task)

## ------------------------------------------------
## Method `Learner$predict_newdata`
## ------------------------------------------------

task = tsk("penguins")
learner = lrn("classif.rpart")$train(task)
learner$predict_newdata(task$data(rows = 1:5))

## ------------------------------------------------
## Method `Learner$reset`
## ------------------------------------------------

task = tsk("penguins")
learner = lrn("classif.rpart")$train(task)
learner$reset()

## ------------------------------------------------
## Method `Learner$encapsulate`
## ------------------------------------------------

learner = lrn("classif.rpart")
fallback = lrn("classif.featureless")
learner$encapsulate("try", fallback = fallback)

## ------------------------------------------------
## Method `Learner$configure`
## ------------------------------------------------

learner = lrn("classif.rpart")
learner$configure(minsplit = 3, parallel_predict = FALSE)
learner$configure(.values = list(cp = 0.005))

Description

This Learner specializes Learner for classification problems:

• task_type is set to "classif".

• Creates Predictions of class PredictionClassif.

• Possible values for predict_types are:

  • "response": Predicts a class label for each observation in the test set.

  • "prob": Predicts the posterior probability for each class for each observation in the test set.

• Additional learner properties include:

  • "twoclass": The learner works on binary classification problems.

  • "multiclass": The learner works on multiclass classification problems.

Predefined learners can be found in the dictionary mlr_learners. Essential classification learners can be found in this dictionary after loading mlr3learners. Additional learners are implemented in the Github package https://github.com/mlr-org/mlr3extralearners.

Super class

mlr3::Learner -> LearnerClassif

Methods

Public methods

• LearnerClassif$new()

• LearnerClassif$predict_newdata_fast()

• LearnerClassif$clone()

Method new()

Creates a new instance of this R6 class.

Usage

LearnerClassif$new(
  id,
  param_set = ps(),
  predict_types = "response",
  feature_types = character(),
  properties = character(),
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments

Method predict_newdata_fast()

Predicts outcomes for new data in newdata using the model fitted during ⁠$train()⁠. This method is faster than ⁠$predict_newdata()⁠ as it skips assertions, type conversions, encapsulation, and logging.

Unlike ⁠$predict_newdata()⁠, this method does not return a Prediction object. Instead, it returns a list with either a "response" or "prob" element, depending on the prediction type.

Note that state$predict_time and state$log will remain empty after using this method. Some learners may not support this method and may fail when it is called. Prefer ⁠$predict_newdata()⁠ unless performance is critical.

If the model was trained via resample() or benchmark(), you must pass the associated task object stored in the corresponding ResampleResult or BenchmarkResult.

Usage

LearnerClassif$predict_newdata_fast(newdata, task = NULL)

Arguments

Returns

list() with elements "response" or "prob" depending on the predict type.

Method clone()

The objects of this class are cloneable with this method.

Usage

LearnerClassif$clone(deep = FALSE)

Arguments

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-learners

• Package mlr3learners for a solid collection of essential learners.

• Package mlr3extralearners for more learners.

• Dictionary of Learners: mlr_learners

• as.data.table(mlr_learners) for a table of available Learners in the running session (depending on the loaded packages).

• mlr3pipelines to combine learners with pre- and postprocessing steps.

• Package mlr3viz for some generic visualizations.

• Extension packages for additional task types:

  • mlr3proba for probabilistic supervised regression and survival analysis.

  • mlr3cluster for unsupervised clustering.

• mlr3tuning for tuning of hyperparameters, mlr3tuningspaces for established default tuning spaces.

Other Learner: Learner, LearnerRegr, mlr_learners, mlr_learners_classif.debug, mlr_learners_classif.featureless, mlr_learners_classif.rpart, mlr_learners_regr.debug, mlr_learners_regr.featureless, mlr_learners_regr.rpart

Examples

# get all classification learners from mlr_learners:
lrns = mlr_learners$mget(mlr_learners$keys("^classif"))
names(lrns)

# get a specific learner from mlr_learners:
lrn = lrn("classif.rpart")
print(lrn)

# train the learner:
task = tsk("penguins")
lrn$train(task, 1:200)

# predict on new observations:
lrn$predict(task, 201:344)$confusion

Description

This Learner specializes Learner for regression problems:

• task_type is set to "regr".

• Creates Predictions of class PredictionRegr.

• Possible values for predict_types are:

  • "response": Predicts a numeric response for each observation in the test set.

  • "se": Predicts the standard error for each value of response for each observation in the test set.

  • "distr": Probability distribution as VectorDistribution object (requires package distr6, available via repository https://raphaels1.r-universe.dev).

• "quantiles": Predicts quantile estimates for each observation in the test set. Set ⁠$quantiles⁠ to specify the quantiles to predict and ⁠$quantile_response⁠ to specify the response quantile. See the mlr3book on quantile regression for more details.

Predefined learners can be found in the dictionary mlr_learners. Essential regression learners can be found in this dictionary after loading mlr3learners. Additional learners are implemented in the Github package https://github.com/mlr-org/mlr3extralearners.

Super class

mlr3::Learner -> LearnerRegr

Active bindings

Methods

Public methods

• LearnerRegr$new()

• LearnerRegr$predict_newdata_fast()

• LearnerRegr$clone()

Method new()

Creates a new instance of this R6 class.

Usage

LearnerRegr$new(
  id,
  task_type = "regr",
  param_set = ps(),
  predict_types = "response",
  feature_types = character(),
  properties = character(),
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments

Method predict_newdata_fast()

Predicts outcomes for new data in newdata using the model fitted during ⁠$train()⁠. This method is faster than ⁠$predict_newdata()⁠ as it skips assertions, type conversions, encapsulation, and logging.

Unlike ⁠$predict_newdata()⁠, this method does not return a Prediction object. Instead, it returns a list with either a "response" or "prob" element, depending on the prediction type.

Note that state$predict_time and state$log will remain empty after using this method. Some learners may not support this method and may fail when it is called. Prefer ⁠$predict_newdata()⁠ unless performance is critical.

If the model was trained via resample() or benchmark(), you must pass the associated task object stored in the corresponding ResampleResult or BenchmarkResult.

Usage

LearnerRegr$predict_newdata_fast(newdata, task = NULL)

Arguments

Returns

list() with elements "response", "se" or "quantiles" depending on the predict type.

Method clone()

The objects of this class are cloneable with this method.

Usage

LearnerRegr$clone(deep = FALSE)

Arguments

See Also

• Chapter in the mlr3book: https://mlr3book.mlr-org.com/chapters/chapter2/data_and_basic_modeling.html#sec-learners

• Package mlr3learners for a solid collection of essential learners.

• Package mlr3extralearners for more learners.

• Dictionary of Learners: mlr_learners

• as.data.table(mlr_learners) for a table of available Learners in the running session (depending on the loaded packages).

• mlr3pipelines to combine learners with pre- and postprocessing steps.

• Package mlr3viz for some generic visualizations.

• Extension packages for additional task types:

  • mlr3proba for probabilistic supervised regression and survival analysis.

  • mlr3cluster for unsupervised clustering.

• mlr3tuning for tuning of hyperparameters, mlr3tuningspaces for established default tuning spaces.

Other Learner: Learner, LearnerClassif, mlr_learners, mlr_learners_classif.debug, mlr_learners_classif.featureless, mlr_learners_classif.rpart, mlr_learners_regr.debug, mlr_learners_regr.featureless, mlr_learners_regr.rpart

Examples

# get all regression learners from mlr_learners:
lrns = mlr_learners$mget(mlr_learners$keys("^regr"))
names(lrns)

# get a specific learner from mlr_learners:
mlr_learners$get("regr.rpart")
lrn("regr.featureless")

Description

Marshaling is the process of processing the model of a trained Learner so it can be successfully serialized and deserialized. The naming is inspired by the marshal package and we plan to fully migrate to this package once it is on CRAN. The current implementation should therefore be considered as a temporary solution and is likely to change in the future.

The central functions (and the only methods that are used by mlr3 internally) are:

• the S3 generic marshal_model(model, inplace, ...). Which takes in a model and returns it in marshaled form. This means, that the resulting object can be serialized and de-serialized without loss of information. If a model is serializable anyway, nothing has to be implemented and the generic will fall back to the default implementation of marshal_model, which is to return the object as-is. Otherwise, the marshaled object should be a list with named elements marshaled and packages, where the former contains the marshaled object, and the latter the package that contains the packages required to unmarshal. Most importantly, this list should contain the package that contains the unmarshal_model method. The returned object should have the classes of the original object with the suffix "_marshaled" appended and the root class should be set to "marshaled".

• the S3 generic ⁠unmarshal_model(model, inplace ...)⁠. Which takes in the marshaled model and returns it in unmarshaled form. The generic takes care that the packages specified during "marshal" are loaded, and errs if they are not available. Calling this function on a marshaled model should reconstruct the original model, i.e. unmarshal_model(marshal_model(x)) should return x. The default implementation of this generic returns x as-is.

• the function is_marshaled_model(model). This (helper) function returns TRUE if the model inherits from class "marshaled" and FALSE otherwise. Note that it is not guaranteed that is_marshaled_model(marshal_model(x)) returns TRUE. This is because the default marshal_model(x) returns x as-is.

For both marshal_model and unmarshal_model, the inplace argument determines whether in-place marshaling should be performed. This is especially relevant in the context of references semantics. If inplace is FALSE, the original input should not be modified, otherwise this is allowed. Note that the input and output can still share references, even when inplace is FALSE.

Usage

learner_unmarshal(.learner, ...)

learner_marshal(.learner, ...)

learner_marshaled(.learner)

marshal_model(model, inplace = FALSE, ...)

unmarshal_model(model, inplace = FALSE, ...)

is_marshaled_model(model)

Arguments

Implementing Marshaling

In order to implement marshaling for a Learner, you need to overload the marshal_model and unmarshal_model methods for the class of the learner's model and tag the learner with the "marshal" property. To make marshaling accessible in an R6-manner, you should also add the public methods ⁠$marshal()⁠, ⁠$unmarshal()⁠ and the active binding ⁠$marshaled⁠. To make this as convenient as possible, the functions learner_marshal(.learner, ...), learner_unmarshal(.learner, ...) and learner_marshaled(.learner) are provided and can be called from the public methods.

You can verify whether you have correctly implemented marshaling by using the internal test helper expect_marshalable_learner(learner, task). This is also run by expect_learner() if a task is provided.

For a concrete example on how to implement marshaling, see LearnerClassifDebug.

Description

This is the abstract base class for measures like MeasureClassif and MeasureRegr.

Measures are classes tailored around two functions doing the work:

1. A function ⁠$score()⁠ which quantifies the performance on a Prediction object, so a set of predicted observation via a scalar number – usually an aggregate of losses on the contained observations, by comparing the truth and prediction columns in the prediction object.

2. A function ⁠$aggregator()⁠ which combines multiple performance scores returned by ⁠$score()⁠ obtained in different resampling iterations to a scalar performance value associated with the complete resampling – usually by averaging or summing.

In addition to these two functions, meta-information about the performance measure is stored.

Predefined measures are stored in the dictionary mlr_measures, e.g. classif.auc or time_train. Many of the measures in mlr3 are implemented in mlr3measures as ordinary functions.

A guide on how to extend mlr3 with custom measures can be found in the mlr3book.

Inheriting

For some measures (such as confidence intervals from mlr3inferr) it is necessary that a measure returns more than one value. In such cases it is necessary to overwrite the public methods ⁠$aggregate()⁠ and/or ⁠$score()⁠ to return a named numeric() where at least one of its names corresponds to the id of the measure itself.

Weights

Many measures support observation weights, indicated by their property "weights". The weights are stored in the Task where the column role weights_measure needs to be assigned to a single numeric column. The weights are automatically used if found in the task, this can be disabled by setting the field use_weights to "ignore". See the description of use_weights for more information.