Package 'mlr3fselect'

Description

Feature selection package of the 'mlr3' ecosystem. It selects the optimal feature set for any 'mlr3' learner. The package works with several optimization algorithms e.g. Random Search, Recursive Feature Elimination, and Genetic Search. Moreover, it can automatically optimize learners and estimate the performance of optimized feature sets with nested resampling.

Author(s)

Maintainer: Marc Becker [email protected] (ORCID)

Authors:

• Patrick Schratz [email protected] (ORCID)

• Michel Lang [email protected] (ORCID)

• Bernd Bischl [email protected] (ORCID)

• John Zobolas [email protected] (ORCID)

See Also

Useful links:

• https://mlr3fselect.mlr-org.com

• https://github.com/mlr-org/mlr3fselect

• Report bugs at https://github.com/mlr-org/mlr3fselect/issues

Description

The ArchiveAsyncFSelect stores all evaluated feature subsets and performance scores in a rush::Rush database.

Details

The ArchiveAsyncFSelect is a connector to a rush::Rush database.

Data Structure

The table (⁠$data⁠) has the following columns:

• One column for each feature of the search space (⁠$search_space⁠).

• One column for each performance measure (⁠$codomain⁠).

• runtime_learners (numeric(1))
  Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

• timestamp (POSIXct)
  Time stamp when the evaluation was logged into the archive.

Analysis

For analyzing the feature selection results, it is recommended to pass the ArchiveAsyncFSelect to as.data.table(). The returned data table contains the mlr3::ResampleResult for each feature subset evaluation.

S3 Methods

• as.data.table.ArchiveFSelect(x, unnest = "x_domain", exclude_columns = "uhash", measures = NULL)
  Returns a tabular view of all evaluated feature subsets.
  ArchiveAsyncFSelect -> data.table::data.table()
  

  • x (ArchiveAsyncFSelect)

  • unnest (character())
    Transforms list columns to separate columns. Set to NULL if no column should be unnested.

  • exclude_columns (character())
    Exclude columns from table. Set to NULL if no column should be excluded.

  • measures (List of mlr3::Measure)
    Score feature subsets on additional measures.

Super classes

bbotk::Archive -> bbotk::ArchiveAsync -> ArchiveAsyncFSelect

Active bindings

Methods

Public methods

• ArchiveAsyncFSelect$new()

• ArchiveAsyncFSelect$learner()

• ArchiveAsyncFSelect$learners()

• ArchiveAsyncFSelect$predictions()

• ArchiveAsyncFSelect$resample_result()

• ArchiveAsyncFSelect$print()

• ArchiveAsyncFSelect$best()

• ArchiveAsyncFSelect$push_result()

• ArchiveAsyncFSelect$clone()

Method new()

Creates a new instance of this R6 class.

Usage

ArchiveAsyncFSelect$new(
  search_space,
  codomain,
  rush,
  ties_method = "least_features"
)

Arguments

Method learner()

Retrieve mlr3::Learner of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive. Learner does not contain a model. Use ⁠$learners()⁠ to get learners with models.

Usage

ArchiveAsyncFSelect$learner(i = NULL, uhash = NULL)

Arguments

Method learners()

Retrieve list of trained mlr3::Learner objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncFSelect$learners(i = NULL, uhash = NULL)

Arguments

Method predictions()

Retrieve list of mlr3::Prediction objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncFSelect$predictions(i = NULL, uhash = NULL)

Arguments

Method resample_result()

Retrieve mlr3::ResampleResult of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncFSelect$resample_result(i = NULL, uhash = NULL)

Arguments

Method print()

Printer.

Usage

ArchiveAsyncFSelect$print()

Arguments

Method best()

Returns the best scoring feature set(s). For single-crit optimization, the solution that minimizes / maximizes the objective function. For multi-crit optimization, the Pareto set / front.

Usage

ArchiveAsyncFSelect$best(n_select = 1, ties_method = "least_features")

Arguments

Returns

data.table::data.table()

Method push_result()

Push result to the archive.

Usage

ArchiveAsyncFSelect$push_result(key, ys, x_domain, extra = NULL)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

ArchiveAsyncFSelect$clone(deep = FALSE)

Arguments

Description

Freezes the Redis data base of an ArchiveAsyncFSelect to a data.table::data.table(). No further points can be added to the archive but the data can be accessed and analyzed. Useful when the Redis data base is not permanently available. Use the callback mlr3fselect.async_freeze_archive to freeze the archive after the optimization has finished.

S3 Methods

• as.data.table(archive)
  ArchiveAsyncFSelectFrozen -> data.table::data.table()
  Returns a tabular view of all performed function calls of the Objective.

Super classes

bbotk::Archive -> bbotk::ArchiveAsync -> bbotk::ArchiveAsyncFrozen -> ArchiveAsyncFSelectFrozen

Active bindings

Methods

Public methods

• ArchiveAsyncFSelectFrozen$new()

• ArchiveAsyncFSelectFrozen$learner()

• ArchiveAsyncFSelectFrozen$learners()

• ArchiveAsyncFSelectFrozen$predictions()

• ArchiveAsyncFSelectFrozen$resample_result()

• ArchiveAsyncFSelectFrozen$print()

• ArchiveAsyncFSelectFrozen$clone()

Method new()

Creates a new instance of this R6 class.

Usage

ArchiveAsyncFSelectFrozen$new(archive)

Arguments

Method learner()

Retrieve mlr3::Learner of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive. Learner does not contain a model. Use ⁠$learners()⁠ to get learners with models.

Usage

ArchiveAsyncFSelectFrozen$learner(i = NULL, uhash = NULL)

Arguments

Method learners()

Retrieve list of trained mlr3::Learner objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncFSelectFrozen$learners(i = NULL, uhash = NULL)

Arguments

Method predictions()

Retrieve list of mlr3::Prediction objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncFSelectFrozen$predictions(i = NULL, uhash = NULL)

Arguments

Method resample_result()

Retrieve mlr3::ResampleResult of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncFSelectFrozen$resample_result(i = NULL, uhash = NULL)

Arguments

Method print()

Printer.

Usage

ArchiveAsyncFSelectFrozen$print()

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

ArchiveAsyncFSelectFrozen$clone(deep = FALSE)

Arguments

Description

The ArchiveBatchFSelect stores all evaluated feature sets and performance scores.

Details

The ArchiveBatchFSelect is a container around a data.table::data.table(). Each row corresponds to a single evaluation of a feature set. See the section on Data Structure for more information. The archive stores additionally a mlr3::BenchmarkResult (⁠$benchmark_result⁠) that records the resampling experiments. Each experiment corresponds to a single evaluation of a feature set. The table (⁠$data⁠) and the benchmark result (⁠$benchmark_result⁠) are linked by the uhash column. If the archive is passed to as.data.table(), both are joined automatically.

Data structure

The table (⁠$data⁠) has the following columns:

• One column for each feature of the task (⁠$search_space⁠).

• One column for each performance measure (⁠$codomain⁠).

• runtime_learners (numeric(1))
  Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

• timestamp (POSIXct)
  Time stamp when the evaluation was logged into the archive.

• batch_nr (integer(1))
  Feature sets are evaluated in batches. Each batch has a unique batch number.

• uhash (character(1))
  Connects each feature set to the resampling experiment stored in the mlr3::BenchmarkResult.

Analysis

For analyzing the feature selection results, it is recommended to pass the archive to as.data.table(). The returned data table is joined with the benchmark result which adds the mlr3::ResampleResult for each feature set.

The archive provides various getters (e.g. ⁠$learners()⁠) to ease the access. All getters extract by position (i) or unique hash (uhash). For a complete list of all getters see the methods section.

The benchmark result (⁠$benchmark_result⁠) allows to score the feature sets again on a different measure. Alternatively, measures can be supplied to as.data.table().

S3 Methods

• as.data.table.ArchiveBatchFSelect(x, exclude_columns = "uhash", measures = NULL)
  Returns a tabular view of all evaluated feature sets.
  ArchiveBatchFSelect -> data.table::data.table()
  

  • x (ArchiveBatchFSelect)

  • exclude_columns (character())
    Exclude columns from table. Set to NULL if no column should be excluded.

  • measures (list of mlr3::Measure)
    Score feature sets on additional measures.

Super classes

bbotk::Archive -> bbotk::ArchiveBatch -> ArchiveBatchFSelect

Public fields

Active bindings

Methods

Public methods

• ArchiveBatchFSelect$new()

• ArchiveBatchFSelect$add_evals()

• ArchiveBatchFSelect$learner()

• ArchiveBatchFSelect$learners()

• ArchiveBatchFSelect$predictions()

• ArchiveBatchFSelect$resample_result()

• ArchiveBatchFSelect$print()

• ArchiveBatchFSelect$best()

• ArchiveBatchFSelect$clone()

Method new()

Creates a new instance of this R6 class.

Usage

ArchiveBatchFSelect$new(
  search_space,
  codomain,
  check_values = TRUE,
  ties_method = "least_features"
)

Arguments

Method add_evals()

Adds function evaluations to the archive table.

Usage

ArchiveBatchFSelect$add_evals(xdt, xss_trafoed = NULL, ydt)

Arguments

Method learner()

Retrieve mlr3::Learner of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive. Learner does not contain a model. Use ⁠$learners()⁠ to get learners with models.

Usage

ArchiveBatchFSelect$learner(i = NULL, uhash = NULL)

Arguments

Method learners()

Retrieve list of trained mlr3::Learner objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveBatchFSelect$learners(i = NULL, uhash = NULL)

Arguments

Method predictions()

Retrieve list of mlr3::Prediction objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveBatchFSelect$predictions(i = NULL, uhash = NULL)

Arguments

Method resample_result()

Retrieve mlr3::ResampleResult of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveBatchFSelect$resample_result(i = NULL, uhash = NULL)

Arguments

Method print()

Printer.

Usage

ArchiveBatchFSelect$print()

Arguments

Method best()

Returns the best scoring feature sets.

Usage

ArchiveBatchFSelect$best(batch = NULL, ties_method = NULL)

Arguments

Returns

data.table::data.table()

Method clone()

The objects of this class are cloneable with this method.

Usage

ArchiveBatchFSelect$clone(deep = FALSE)

Arguments

Description

Assertions for CallbackAsyncFSelect class.

Usage

assert_async_fselect_callback(callback, null_ok = FALSE)

assert_async_fselect_callbacks(callbacks)

Arguments

Value

[CallbackAsyncFSelect | List of CallbackAsyncFSelects.

Description

The AutoFSelector wraps a mlr3::Learner and augments it with an automatic feature selection. The auto_fselector() function creates an AutoFSelector object.

Usage

auto_fselector(
  fselector,
  learner,
  resampling,
  measure = NULL,
  term_evals = NULL,
  term_time = NULL,
  terminator = NULL,
  store_fselect_instance = TRUE,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL,
  ties_method = "least_features",
  rush = NULL,
  id = NULL
)

Arguments

Details

The AutoFSelector is a mlr3::Learner which wraps another mlr3::Learner and performs the following steps during ⁠$train()⁠:

1. The wrapped (inner) learner is trained on the feature subsets via resampling. The feature selection can be specified by providing a FSelector, a bbotk::Terminator, a mlr3::Resampling and a mlr3::Measure.

2. A final model is fit on the complete training data with the best-found feature subset.

During ⁠$predict()⁠ the AutoFSelector just calls the predict method of the wrapped (inner) learner.

Value

AutoFSelector.

Resources

There are several sections about feature selection in the mlr3book.

• Estimate Model Performance with nested resampling.

The gallery features a collection of case studies and demos about optimization.

Nested Resampling

Nested resampling can be performed by passing an AutoFSelector object to mlr3::resample() or mlr3::benchmark(). To access the inner resampling results, set store_fselect_instance = TRUE and execute mlr3::resample() or mlr3::benchmark() with store_models = TRUE (see examples). The mlr3::Resampling passed to the AutoFSelector is meant to be the inner resampling, operating on the training set of an arbitrary outer resampling. For this reason it is not feasible to pass an instantiated mlr3::Resampling here.

Examples

afs = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

afs$train(tsk("pima"))

Description

The AutoFSelector wraps a mlr3::Learner and augments it with an automatic feature selection. The auto_fselector() function creates an AutoFSelector object.

Details

The AutoFSelector is a mlr3::Learner which wraps another mlr3::Learner and performs the following steps during ⁠$train()⁠:

1. The wrapped (inner) learner is trained on the feature subsets via resampling. The feature selection can be specified by providing a FSelector, a bbotk::Terminator, a mlr3::Resampling and a mlr3::Measure.

2. A final model is fit on the complete training data with the best-found feature subset.

During ⁠$predict()⁠ the AutoFSelector just calls the predict method of the wrapped (inner) learner.

Resources

There are several sections about feature selection in the mlr3book.

• Estimate Model Performance with nested resampling.

The gallery features a collection of case studies and demos about optimization.

Nested Resampling

Nested resampling can be performed by passing an AutoFSelector object to mlr3::resample() or mlr3::benchmark(). To access the inner resampling results, set store_fselect_instance = TRUE and execute mlr3::resample() or mlr3::benchmark() with store_models = TRUE (see examples). The mlr3::Resampling passed to the AutoFSelector is meant to be the inner resampling, operating on the training set of an arbitrary outer resampling. For this reason it is not feasible to pass an instantiated mlr3::Resampling here.

Super class

mlr3::Learner -> AutoFSelector

Public fields

Active bindings

Methods

Public methods

• AutoFSelector$new()

• AutoFSelector$base_learner()

• AutoFSelector$importance()

• AutoFSelector$selected_features()

• AutoFSelector$oob_error()

• AutoFSelector$loglik()

• AutoFSelector$print()

• AutoFSelector$clone()

Method new()

Creates a new instance of this R6 class.

Usage

AutoFSelector$new(
  fselector,
  learner,
  resampling,
  measure = NULL,
  terminator,
  store_fselect_instance = TRUE,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL,
  ties_method = "least_features",
  rush = NULL,
  id = NULL
)

Arguments

Method base_learner()

Extracts the base learner from nested learner objects like GraphLearner in mlr3pipelines. If recursive = 0, the (tuned) learner is returned.

Usage

AutoFSelector$base_learner(recursive = Inf)

Arguments

Returns

mlr3::Learner.

Method importance()

The importance scores of the final model.

Usage

AutoFSelector$importance()

Returns

Named numeric().

Method selected_features()

The selected features of the final model. These features are selected internally by the learner.

Usage

AutoFSelector$selected_features()

Returns

character().

Method oob_error()

The out-of-bag error of the final model.

Usage

AutoFSelector$oob_error()

Returns

numeric(1).

Method loglik()

The log-likelihood of the final model.

Usage

AutoFSelector$loglik()

Returns

logLik. Printer.

Method print()

Usage

AutoFSelector$print()

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

AutoFSelector$clone(deep = FALSE)

Arguments

Examples

# Automatic Feature Selection


# split to train and external set
task = tsk("penguins")
split = partition(task, ratio = 0.8)

# create auto fselector
afs = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

# optimize feature subset and fit final model
afs$train(task, row_ids = split$train)

# predict with final model
afs$predict(task, row_ids = split$test)

# show result
afs$fselect_result

# model slot contains trained learner and fselect instance
afs$model

# shortcut trained learner
afs$learner

# shortcut fselect instance
afs$fselect_instance


# Nested Resampling

afs = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 3)
rr = resample(task, afs, resampling_outer, store_models = TRUE)

# retrieve inner feature selection results.
extract_inner_fselect_results(rr)

# performance scores estimated on the outer resampling
rr$score()

# unbiased performance of the final model trained on the full data set
rr$aggregate()

Description

Function to create a CallbackAsyncFSelect. Predefined callbacks are stored in the dictionary mlr_callbacks and can be retrieved with clbk().

Feature selection callbacks can be called from different stages of the feature selection process. The stages are prefixed with ⁠on_*⁠.

Start Feature Selection
     - on_optimization_begin
    Start Worker
         - on_worker_begin
         Start Optimization on Worker
           - on_optimizer_before_eval
             Start Evaluation
               - on_eval_after_xs
                 Start Resampling Iteration
                   - on_resample_begin
                   - on_resample_before_train
                   - on_resample_before_predict
                   - on_resample_end
                 End Resampling Iteration
               - on_eval_after_resample
               - on_eval_before_archive
             End Evaluation
          - on_optimizer_after_eval
         End Optimization on Worker
         - on_worker_end
    End Worker
     - on_fselect_result_begin
     - on_result_begin
     - on_result_end
     - on_optimization_end
End Feature Selection

See also the section on parameters for more information on the stages. A feature selection callback works with ContextAsyncFSelect.

Usage

callback_async_fselect(
  id,
  label = NA_character_,
  man = NA_character_,
  on_optimization_begin = NULL,
  on_worker_begin = NULL,
  on_optimizer_before_eval = NULL,
  on_eval_after_xs = NULL,
  on_resample_begin = NULL,
  on_resample_before_train = NULL,
  on_resample_before_predict = NULL,
  on_resample_end = NULL,
  on_eval_after_resample = NULL,
  on_eval_before_archive = NULL,
  on_optimizer_after_eval = NULL,
  on_worker_end = NULL,
  on_fselect_result_begin = NULL,
  on_result_begin = NULL,
  on_result_end = NULL,
  on_result = NULL,
  on_optimization_end = NULL
)

Arguments

Details

When implementing a callback, each function must have two arguments named callback and context. A callback can write data to the state (⁠$state⁠), e.g. settings that affect the callback itself. Feature selection callbacks access ContextAsyncFSelect and mlr3::ContextResample.

Description

Function to create a CallbackBatchFSelect. Predefined callbacks are stored in the dictionary mlr_callbacks and can be retrieved with clbk().

Feature selection callbacks can be called from different stages of feature selection. The stages are prefixed with ⁠on_*⁠. The ⁠on_auto_fselector_*⁠ stages are only available when the callback is used in an AutoFSelector.

Start Automatic Feature Selection
  Start Feature Selection
      - on_optimization_begin
      Start FSelect Batch
          - on_optimizer_before_eval
          Start Evaluation
              - on_eval_after_design
              - on_eval_after_benchmark
              - on_eval_before_archive
          End Evaluation
          - on_optimizer_after_eval
      End FSelect Batch
      - on_result
      - on_optimization_end
  End Feature Selection
  - on_auto_fselector_before_final_model
  - on_auto_fselector_after_final_model
End Automatic Feature Selection

See also the section on parameters for more information on the stages. A feature selection callback works with bbotk::ContextBatch and ContextBatchFSelect.

Usage

callback_batch_fselect(
  id,
  label = NA_character_,
  man = NA_character_,
  on_optimization_begin = NULL,
  on_optimizer_before_eval = NULL,
  on_eval_after_design = NULL,
  on_eval_after_benchmark = NULL,
  on_eval_before_archive = NULL,
  on_optimizer_after_eval = NULL,
  on_result = NULL,
  on_optimization_end = NULL,
  on_auto_fselector_before_final_model = NULL,
  on_auto_fselector_after_final_model = NULL
)

Arguments

Details

When implementing a callback, each function must have two arguments named callback and context. A callback can write data to the state (⁠$state⁠), e.g. settings that affect the callback itself. Avoid writing large data the state.

Examples

# Write archive to disk
callback_batch_fselect("mlr3fselect.backup",
  on_optimization_end = function(callback, context) {
    saveRDS(context$instance$archive, "archive.rds")
  }
)

Description

Specialized bbotk::CallbackAsync for asynchronous feature selection. Callbacks allow to customize the behavior of processes in mlr3fselect. The callback_async_fselect() function creates a CallbackAsyncFSelect. Predefined callbacks are stored in the dictionary mlr_callbacks and can be retrieved with clbk(). For more information on feature selection callbacks see callback_async_fselect().

Super classes

mlr3misc::Callback -> bbotk::CallbackAsync -> CallbackAsyncFSelect

Public fields

Methods

Public methods

• CallbackAsyncFSelect$clone()

Method clone()

The objects of this class are cloneable with this method.

Usage

CallbackAsyncFSelect$clone(deep = FALSE)

Arguments

Description

Specialized bbotk::CallbackBatch for feature selection. Callbacks allow customizing the behavior of processes in mlr3fselect. The callback_batch_fselect() function creates a CallbackBatchFSelect. Predefined callbacks are stored in the dictionary mlr_callbacks and can be retrieved with clbk(). For more information on callbacks see callback_batch_fselect().

Super classes

mlr3misc::Callback -> bbotk::CallbackBatch -> CallbackBatchFSelect

Public fields

Methods

Public methods

• CallbackBatchFSelect$clone()

Method clone()

The objects of this class are cloneable with this method.

Usage

CallbackBatchFSelect$clone(deep = FALSE)

Arguments

Examples

# Write archive to disk
callback_batch_fselect("mlr3fselect.backup",
  on_optimization_end = function(callback, context) {
    saveRDS(context$instance$archive, "archive.rds")
  }
)

Description

A CallbackAsyncFSelect accesses and modifies data during the optimization via the ContextAsyncFSelect. See the section on active bindings for a list of modifiable objects. See callback_async_fselect() for a list of stages that access ContextAsyncFSelect.

Details

Changes to ⁠$instance⁠ and ⁠$optimizer⁠ in the stages executed on the workers are not reflected in the main process.

Super classes

mlr3misc::Context -> bbotk::ContextAsync -> ContextAsyncFSelect

Public fields

Active bindings

Methods

Public methods

• ContextAsyncFSelect$clone()

Method clone()

The objects of this class are cloneable with this method.

Usage

ContextAsyncFSelect$clone(deep = FALSE)

Arguments

Description

The ContextBatchFSelect allows CallbackBatchFSelects to access and modify data while a batch of feature sets is evaluated. See the section on active bindings for a list of modifiable objects. See callback_batch_fselect() for a list of stages that access ContextBatchFSelect.

Details

This context is re-created each time a new batch of feature sets is evaluated. Changes to ⁠$objective_fselect⁠, ⁠$design⁠ ⁠$benchmark_result⁠ are discarded after the function is finished. Modification on the data table in ⁠$aggregated_performance⁠ are written to the archive. Any number of columns can be added.

Super classes

mlr3misc::Context -> bbotk::ContextBatch -> ContextBatchFSelect

Public fields

Active bindings

Methods

Public methods

• ContextBatchFSelect$clone()

Method clone()

The objects of this class are cloneable with this method.

Usage

ContextBatchFSelect$clone(deep = FALSE)

Arguments

Description

Ensemble feature selection using multiple learners. The ensemble feature selection method is designed to identify the most predictive features from a given dataset by leveraging multiple machine learning models and resampling techniques. Returns an EnsembleFSResult.

Usage

embedded_ensemble_fselect(
  task,
  learners,
  init_resampling,
  measure,
  store_benchmark_result = TRUE
)

Arguments

Details

The method begins by applying an initial resampling technique specified by the user, to create multiple subsamples from the original dataset (train/test splits). This resampling process helps in generating diverse subsets of data for robust feature selection.

For each subsample (train set) generated in the previous step, the method applies learners that support embedded feature selection. These learners are then scored on their ability to predict on the resampled test sets, storing the selected features during training, for each combination of subsample and learner.

Results are stored in an EnsembleFSResult.

Value

an EnsembleFSResult object.

Source

Meinshausen, Nicolai, Buhlmann, Peter (2010). “Stability Selection.” Journal of the Royal Statistical Society Series B: Statistical Methodology, 72(4), 417–473. ISSN 1369-7412, doi:10.1111/J.1467-9868.2010.00740.X, 0809.2932.

Hedou, Julien, Maric, Ivana, Bellan, Gregoire, Einhaus, Jakob, Gaudilliere, K. D, Ladant, Xavier F, Verdonk, Franck, Stelzer, A. I, Feyaerts, Dorien, Tsai, S. A, Ganio, A. E, Sabayev, Maximilian, Gillard, Joshua, Amar, Jonas, Cambriel, Amelie, Oskotsky, T. T, Roldan, Alennie, Golob, L. J, Sirota, Marina, Bonham, A. T, Sato, Masaki, Diop, Maigane, Durand, Xavier, Angst, S. M, Stevenson, K. D, Aghaeepour, Nima, Montanari, Andrea, Gaudilliere, Brice (2024). “Discovery of sparse, reliable omic biomarkers with Stabl.” Nature Biotechnology 2024, 1–13. ISSN 1546-1696, doi:10.1038/s41587-023-02033-x, https://www.nature.com/articles/s41587-023-02033-x.

Examples

eefsr = embedded_ensemble_fselect(
    task = tsk("sonar"),
    learners = lrns(c("classif.rpart", "classif.featureless")),
    init_resampling = rsmp("subsampling", repeats = 5),
    measure = msr("classif.ce")
  )
  eefsr

Description

The EnsembleFSResult stores the results of ensemble feature selection. It includes methods for evaluating the stability of the feature selection process and for ranking the selected features among others.

Both functions ensemble_fselect() and embedded_ensemble_fselect() return an object of this class.

S3 Methods

• as.data.table.EnsembleFSResult(x, benchmark_result = TRUE)
  Returns a tabular view of the ensemble feature selection.
  EnsembleFSResult -> data.table::data.table()
  

  • x (EnsembleFSResult)

  • benchmark_result (logical(1))
    Whether to add the learner, task and resampling information from the benchmark result.

• c(...)
  (EnsembleFSResult, ...) -> EnsembleFSResult
  Combines multiple EnsembleFSResult objects into a new EnsembleFSResult.

Public fields

Active bindings

Methods

Public methods

• EnsembleFSResult$new()

• EnsembleFSResult$format()

• EnsembleFSResult$print()

• EnsembleFSResult$help()

• EnsembleFSResult$set_active_measure()

• EnsembleFSResult$combine()

• EnsembleFSResult$feature_ranking()

• EnsembleFSResult$stability()

• EnsembleFSResult$pareto_front()

• EnsembleFSResult$knee_points()

• EnsembleFSResult$clone()

Method new()

Creates a new instance of this R6 class.

Usage

EnsembleFSResult$new(
  result,
  features,
  benchmark_result = NULL,
  measure,
  inner_measure = NULL
)

Arguments

Method format()

Helper for print outputs.

Usage

EnsembleFSResult$format(...)

Arguments

Method print()

Printer.

Usage

EnsembleFSResult$print(...)

Arguments

Method help()

Opens the corresponding help page referenced by field ⁠$man⁠.

Usage

EnsembleFSResult$help()

Method set_active_measure()

Use this function to change the active measure.

Usage

EnsembleFSResult$set_active_measure(which = "outer")

Arguments

Method combine()

Combines a second EnsembleFSResult into the current object, modifying it in-place. If the second EnsembleFSResult (efsr) is NULL, the method returns the object unmodified.

Both objects must have the same task features and measure. If the inner_measure differs between the objects or is NULL in either, it will be set to NULL in the combined object. Additionally, the importance column will be removed if it is missing in either object. If both objects contain a benchmark_result, these will be combined. Otherwise, the combined object will have a NULL value for benchmark_result.

This method modifies the object by reference. To preserve the original state, explicitly ⁠$clone()⁠ the object beforehand. Alternatively, you can use the c() function, which internally calls this method.

Usage

EnsembleFSResult$combine(efsr)

Arguments

Returns

Returns the object itself, but modified by reference.

Method feature_ranking()

Calculates the feature ranking via fastVoteR::rank_candidates().

Usage

EnsembleFSResult$feature_ranking(
  method = "av",
  use_weights = TRUE,
  committee_size = NULL,
  shuffle_features = TRUE
)

Arguments

Details

The feature ranking process is built on the following framework: models act as voters, features act as candidates, and voters select certain candidates (features). The primary objective is to compile these selections into a consensus ranked list of features, effectively forming a committee.

For every feature a score is calculated, which depends on the "method" argument. The higher the score, the higher the ranking of the feature. Note that some methods output a feature ranking instead of a score per feature, so we always include Borda's score, which is method-agnostic, i.e. it can be used to compare the feature rankings across different methods.

We shuffle the input candidates/features so that we enforce random tie-breaking. Users should set the same seed for consistent comparison between the different feature ranking methods and for reproducibility.

Returns

A data.table::data.table listing all the features, ordered by decreasing scores (depends on the "method"). Columns are as follows:

• "feature": Feature names.

• "score": Scores assigned to each feature based on the selected method (if applicable).

• "norm_score": Normalized scores (if applicable), scaled to the range $[0,1]$, which can be loosely interpreted as selection probabilities (Meinshausen et al. (2010)).

• "borda_score": Borda scores for method-agnostic comparison, ranging in $[0,1]$, where the top feature receives a score of 1 and the lowest-ranked feature receives a score of 0. This column is always included so that feature ranking methods that output only rankings have also a feature-wise score.

Method stability()

Calculates the stability of the selected features with the stabm package. The results are cached. When the same stability measure is requested again with different arguments, the cache must be reset.

Usage

EnsembleFSResult$stability(
  stability_measure = "jaccard",
  stability_args = NULL,
  global = TRUE,
  reset_cache = FALSE
)

Arguments

Returns

A numeric() value representing the stability of the selected features. Or a numeric() vector with the stability of the selected features for each learner.

Method pareto_front()

This function identifies the Pareto front of the ensemble feature selection process, i.e., the set of points that represent the trade-off between the number of features and performance (e.g. classification error).

Usage

EnsembleFSResult$pareto_front(type = "empirical", max_nfeatures = NULL)

Arguments

Details

Two options are available for the Pareto front:

• "empirical" (default): returns the empirical Pareto front.

• "estimated": the Pareto front points are estimated by fitting a linear model with the inversed of the number of features ($1/x$) as input and the associated performance scores as output.

This method is useful when the Pareto points are sparse and the front assumes a convex shape if better performance corresponds to lower measure values (e.g. classification error), or a concave shape otherwise (e.g. classification accuracy).

When type = "estimated", the estimated Pareto front includes points with the number of features ranging from 1 up to max_nfeatures. If max_nfeatures is not provided, it defaults to the maximum number of features available in the ensemble feature selection result, i.e. the maximum out of all learners and resamplings included.

Returns

A data.table::data.table with columns the number of features and the performance that together form the Pareto front.

Method knee_points()

This function implements various knee point identification (KPI) methods, which select points in the Pareto front, such that an optimal trade-off between performance and number of features is achieved. In most cases, only one such point is returned.

Usage

EnsembleFSResult$knee_points(
  method = "NBI",
  type = "empirical",
  max_nfeatures = NULL
)

Arguments

Details

The available KPI methods are:

• "NBI" (default): The Normal-Boundary Intersection method is a geometry-based method which calculates the perpendicular distance of each point from the line connecting the first and last points of the Pareto front. The knee point is determined as the Pareto point with the maximum distance from this line, see Das (1999).

Returns

A data.table::data.table with the knee point(s) of the Pareto front.

Method clone()

The objects of this class are cloneable with this method.

Usage

EnsembleFSResult$clone(deep = FALSE)

Arguments

References

Das, I (1999). “On characterizing the 'knee' of the Pareto curve based on normal-boundary intersection.” Structural Optimization, 18(1-2), 107–115. ISSN 09344373.

Meinshausen, Nicolai, Buhlmann, Peter (2010). “Stability Selection.” Journal of the Royal Statistical Society Series B: Statistical Methodology, 72(4), 417–473. ISSN 1369-7412, doi:10.1111/J.1467-9868.2010.00740.X, 0809.2932.

Examples

efsr = ensemble_fselect(
    fselector = fs("rfe", n_features = 2, feature_fraction = 0.8),
    task = tsk("sonar"),
    learners = lrns(c("classif.rpart", "classif.featureless")),
    init_resampling = rsmp("subsampling", repeats = 2),
    inner_resampling = rsmp("cv", folds = 3),
    inner_measure = msr("classif.ce"),
    measure = msr("classif.acc"),
    terminator = trm("none")
  )

  # contains the benchmark result
  efsr$benchmark_result

  # contains the selected features for each iteration
  efsr$result

  # returns the stability of the selected features
  efsr$stability(stability_measure = "jaccard")

  # returns a ranking of all features
  head(efsr$feature_ranking())

  # returns the empirical pareto front, i.e. n_features vs measure (error)
  efsr$pareto_front()

  # returns the knee points (optimal trade-off between n_features and performance)
  efsr$knee_points()

  # change to use the inner optimization measure
  efsr$set_active_measure(which = "inner")

  # Pareto front is calculated on the inner measure
  efsr$pareto_front()

Description

Ensemble feature selection using multiple learners. The ensemble feature selection method is designed to identify the most predictive features from a given dataset by leveraging multiple machine learning models and resampling techniques. Returns an EnsembleFSResult.

Usage

ensemble_fselect(
  fselector,
  task,
  learners,
  init_resampling,
  inner_resampling,
  inner_measure,
  measure,
  terminator,
  callbacks = NULL,
  store_benchmark_result = TRUE,
  store_models = FALSE
)

Arguments

Details

The method begins by applying an initial resampling technique specified by the user, to create multiple subsamples from the original dataset (train/test splits). This resampling process helps in generating diverse subsets of data for robust feature selection.

For each subsample (train set) generated in the previous step, the method performs wrapped-based feature selection (auto_fselector) using each provided learner, the given inner resampling method, inner performance measure and optimization algorithm. This process generates 1) the best feature subset and 2) a final trained model using these best features, for each combination of subsample and learner. The final models are then scored on their ability to predict on the resampled test sets.

Results are stored in an EnsembleFSResult.

The result object also includes the performance scores calculated during the inner resampling of the training sets, using models with the best feature subsets. These scores are stored in a column named ⁠{measure_id}_inner⁠.

Value

an EnsembleFSResult object.

Note

The active measure of performance is the one applied to the test sets. This is preferred, as inner resampling scores on the training sets are likely to be overestimated when using the final models. Users can change the active measure by using the set_active_measure() method of the EnsembleFSResult.

Source

Saeys, Yvan, Abeel, Thomas, Van De Peer, Yves (2008). “Robust feature selection using ensemble feature selection techniques.” Machine Learning and Knowledge Discovery in Databases, 5212 LNAI, 313–325. doi:10.1007/978-3-540-87481-2_21.

Abeel, Thomas, Helleputte, Thibault, Van de Peer, Yves, Dupont, Pierre, Saeys, Yvan (2010). “Robust biomarker identification for cancer diagnosis with ensemble feature selection methods.” Bioinformatics, 26, 392–398. ISSN 1367-4803, doi:10.1093/BIOINFORMATICS/BTP630.

Pes, Barbara (2020). “Ensemble feature selection for high-dimensional data: a stability analysis across multiple domains.” Neural Computing and Applications, 32(10), 5951–5973. ISSN 14333058, doi:10.1007/s00521-019-04082-3.

Examples

efsr = ensemble_fselect(
    fselector = fs("random_search"),
    task = tsk("sonar"),
    learners = lrns(c("classif.rpart", "classif.featureless")),
    init_resampling = rsmp("subsampling", repeats = 2),
    inner_resampling = rsmp("cv", folds = 3),
    inner_measure = msr("classif.ce"),
    measure = msr("classif.acc"),
    terminator = trm("evals", n_evals = 10)
  )
  efsr

Description

Extract inner feature selection archives of nested resampling. Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult. The function iterates over the AutoFSelector objects and binds the archives to a data.table::data.table(). AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE.

Usage

extract_inner_fselect_archives(x, exclude_columns = "uhash")

Arguments

Value

data.table::data.table().

Data structure

The returned data table has the following columns:

• experiment (integer(1))
  Index, giving the according row number in the original benchmark grid.

• iteration (integer(1))
  Iteration of the outer resampling.

• One column for each feature of the task.

• One column for each performance measure.

• runtime_learners (numeric(1))
  Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

• timestamp (POSIXct)
  Time stamp when the evaluation was logged into the archive.

• batch_nr (integer(1))
  Feature sets are evaluated in batches. Each batch has a unique batch number.

• resample_result (mlr3::ResampleResult)
  Resample result of the inner resampling.

• task_id (character(1)).

• learner_id (character(1)).

• resampling_id (character(1)).

Examples

# Nested Resampling on Palmer Penguins Data Set

# create auto fselector
at = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("penguins"), at, resampling_outer, store_models = TRUE)

# extract inner archives
extract_inner_fselect_archives(rr)

Description

Extract inner feature selection results of nested resampling. Implemented for mlr3::ResampleResult and mlr3::BenchmarkResult.

Usage

extract_inner_fselect_results(x, fselect_instance, ...)

Arguments

Details

The function iterates over the AutoFSelector objects and binds the feature selection results to a data.table::data.table(). AutoFSelector must be initialized with store_fselect_instance = TRUE and resample() or benchmark() must be called with store_models = TRUE. Optionally, the instance can be added for each iteration.

Value

data.table::data.table().

Data structure

The returned data table has the following columns:

• experiment (integer(1))
  Index, giving the according row number in the original benchmark grid.

• iteration (integer(1))
  Iteration of the outer resampling.

• One column for each feature of the task.

• One column for each performance measure.

• features (character())
  Vector of selected feature set.

• task_id (character(1)).

• learner_id (character(1)).

• resampling_id (character(1)).

Examples

# Nested Resampling on Palmer Penguins Data Set

# create auto fselector
at = auto_fselector(
  fselector = fs("random_search"),
  learner = lrn("classif.rpart"),
  resampling = rsmp ("holdout"),
  measure = msr("classif.ce"),
  term_evals = 4)

resampling_outer = rsmp("cv", folds = 2)
rr = resample(tsk("iris"), at, resampling_outer, store_models = TRUE)

# extract inner results
extract_inner_fselect_results(rr)

Description

Aggregates a mlr3::ResampleResult or mlr3::BenchmarkResult for a single simple measure. Returns the aggregated score for each resample result.

Usage

faggregate(obj, measure, conditions = FALSE)

Arguments

Details

This function is faster than ⁠$aggregate()⁠ because it does not reassemble the resampling results. It only works on simple measures which do not require the task, learner, model or train set to be available.

Value

(data.table::data.table())

Description

Functions to retrieve objects, set parameters and assign to fields in one go. Relies on mlr3misc::dictionary_sugar_get() to extract objects from the respective mlr3misc::Dictionary:

• fs() for a FSelector from mlr_fselectors.

• fss() for a list of FSelectors from mlr_fselectors.

• trm() for a bbotk::Terminator from mlr_terminators.

• trms() for a list of Terminators from mlr_terminators.

Usage

fs(.key, ...)

fss(.keys, ...)

Arguments

Value

R6::R6Class object of the respective type, or a list of R6::R6Class objects for the plural versions.

Examples

# random search fselector with batch size of 5
fs("random_search", batch_size = 5)

# run time terminator with 20 seconds
trm("run_time", secs = 20)

Description

Function to optimize the features of a mlr3::Learner. The function internally creates a FSelectInstanceBatchSingleCrit or FSelectInstanceBatchMultiCrit which describes the feature selection problem. It executes the feature selection with the FSelector (fselector) and returns the result with the feature selection instance (⁠$result⁠). The ArchiveBatchFSelect and ArchiveAsyncFSelect (⁠$archive⁠) stores all evaluated feature subsets and performance scores.

You can find an overview of all feature selectors on our website.

Usage

fselect(
  fselector,
  task,
  learner,
  resampling,
  measures = NULL,
  term_evals = NULL,
  term_time = NULL,
  terminator = NULL,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL,
  ties_method = "least_features",
  rush = NULL
)

Arguments

Details

The mlr3::Task, mlr3::Learner, mlr3::Resampling, mlr3::Measure and bbotk::Terminator are used to construct a FSelectInstanceBatchSingleCrit. If multiple performance mlr3::Measures are supplied, a FSelectInstanceBatchMultiCrit is created. The parameter term_evals and term_time are shortcuts to create a bbotk::Terminator. If both parameters are passed, a bbotk::TerminatorCombo is constructed. For other Terminators, pass one with terminator. If no termination criterion is needed, set term_evals, term_time and terminator to NULL.

Value

FSelectInstanceBatchSingleCrit | FSelectInstanceBatchMultiCrit

Default Measures

If no measure is passed, the default measure is used. The default measure depends on the task type.

Resources

There are several sections about feature selection in the mlr3book.

• Getting started with wrapper feature selection.

• Do a sequential forward selection Palmer Penguins data set.

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

• Run a feature selection with Shadow Variable Search.

Analysis

For analyzing the feature selection results, it is recommended to pass the archive to as.data.table(). The returned data table is joined with the benchmark result which adds the mlr3::ResampleResult for each feature set.

The archive provides various getters (e.g. ⁠$learners()⁠) to ease the access. All getters extract by position (i) or unique hash (uhash). For a complete list of all getters see the methods section.

The benchmark result (⁠$benchmark_result⁠) allows to score the feature sets again on a different measure. Alternatively, measures can be supplied to as.data.table().

Examples

# Feature selection on the Pima Indians data set
task = tsk("pima")

# Load learner
learner = lrn("classif.rpart")

# Run feature selection
instance = fselect(
  fselector = fs("random_search", batch_size = 2),
  task = task,
  learner = learner,
  resampling = rsmp ("holdout"),
  measures = msr("classif.ce"),
  term_evals = 4)

# Subset task to optimized feature set
task$select(instance$result_feature_set)

# Train the learner with optimal feature set on the full data set
learner$train(task)

# Inspect all evaluated feature subsets
as.data.table(instance$archive)

Description

Function to conduct nested resampling.

Usage

fselect_nested(
  fselector,
  task,
  learner,
  inner_resampling,
  outer_resampling,
  measure = NULL,
  term_evals = NULL,
  term_time = NULL,
  terminator = NULL,
  store_fselect_instance = TRUE,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL,
  ties_method = "least_features"
)

Arguments

Value

mlr3::ResampleResult

Examples

# Nested resampling on Palmer Penguins data set
rr = fselect_nested(
  fselector = fs("random_search"),
  task = tsk("penguins"),
  learner = lrn("classif.rpart"),
  inner_resampling = rsmp ("holdout"),
  outer_resampling = rsmp("cv", folds = 2),
  measure = msr("classif.ce"),
  term_evals = 4)

# Performance scores estimated on the outer resampling
rr$score()

# Unbiased performance of the final model trained on the full data set
rr$aggregate()

Description

The FSelectInstanceAsyncMultiCrit specifies a feature selection problem for a FSelectorAsync. The function fsi_async() creates a FSelectInstanceAsyncMultiCrit and the function fselect() creates an instance internally.

Default Measures

If no measure is passed, the default measure is used. The default measure depends on the task type.

Analysis

For analyzing the feature selection results, it is recommended to pass the ArchiveAsyncFSelect to as.data.table(). The returned data table contains the mlr3::ResampleResult for each feature subset evaluation.

Resources

There are several sections about feature selection in the mlr3book.

• Getting started with wrapper feature selection.

• Do a sequential forward selection Palmer Penguins data set.

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

• Run a feature selection with Shadow Variable Search.

Super classes

bbotk::OptimInstance -> bbotk::OptimInstanceAsync -> bbotk::OptimInstanceAsyncMultiCrit -> FSelectInstanceAsyncMultiCrit

Methods

Public methods

• FSelectInstanceAsyncMultiCrit$new()

• FSelectInstanceAsyncMultiCrit$assign_result()

• FSelectInstanceAsyncMultiCrit$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelectInstanceAsyncMultiCrit$new(
  task,
  learner,
  resampling,
  measures,
  terminator,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL,
  rush = NULL
)

Arguments

Method assign_result()

The FSelectorAsync object writes the best found points and estimated performance values here (probably the Pareto set / front). For internal use.

Usage

FSelectInstanceAsyncMultiCrit$assign_result(xdt, ydt, extra = NULL, ...)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectInstanceAsyncMultiCrit$clone(deep = FALSE)

Arguments

Description

The FSelectInstanceAsyncSingleCrit specifies a feature selection problem for a FSelectorAsync. The function fsi_async() creates a FSelectInstanceAsyncSingleCrit and the function fselect() creates an instance internally.

Default Measures

If no measure is passed, the default measure is used. The default measure depends on the task type.

Analysis

For analyzing the feature selection results, it is recommended to pass the ArchiveAsyncFSelect to as.data.table(). The returned data table contains the mlr3::ResampleResult for each feature subset evaluation.

Resources

There are several sections about feature selection in the mlr3book.

• Getting started with wrapper feature selection.

• Do a sequential forward selection Palmer Penguins data set.

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

• Run a feature selection with Shadow Variable Search.

Super classes

bbotk::OptimInstance -> bbotk::OptimInstanceAsync -> bbotk::OptimInstanceAsyncSingleCrit -> FSelectInstanceAsyncSingleCrit

Methods

Public methods

• FSelectInstanceAsyncSingleCrit$new()

• FSelectInstanceAsyncSingleCrit$assign_result()

• FSelectInstanceAsyncSingleCrit$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelectInstanceAsyncSingleCrit$new(
  task,
  learner,
  resampling,
  measure = NULL,
  terminator,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL,
  ties_method = "least_features",
  rush = NULL
)

Arguments

Method assign_result()

The FSelectorAsync object writes the best found point and estimated performance value here. For internal use.

Usage

FSelectInstanceAsyncSingleCrit$assign_result(xdt, y, extra = NULL, ...)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectInstanceAsyncSingleCrit$clone(deep = FALSE)

Arguments

Description

The FSelectInstanceBatchMultiCrit specifies a feature selection problem for a FSelector. The function fsi() creates a FSelectInstanceBatchMultiCrit and the function fselect() creates an instance internally.

Resources

There are several sections about feature selection in the mlr3book.

• Learn about multi-objective optimization.

The gallery features a collection of case studies and demos about optimization.

Analysis

For analyzing the feature selection results, it is recommended to pass the archive to as.data.table(). The returned data table is joined with the benchmark result which adds the mlr3::ResampleResult for each feature set.

The archive provides various getters (e.g. ⁠$learners()⁠) to ease the access. All getters extract by position (i) or unique hash (uhash). For a complete list of all getters see the methods section.

The benchmark result (⁠$benchmark_result⁠) allows to score the feature sets again on a different measure. Alternatively, measures can be supplied to as.data.table().

Super classes

bbotk::OptimInstance -> bbotk::OptimInstanceBatch -> bbotk::OptimInstanceBatchMultiCrit -> FSelectInstanceBatchMultiCrit

Active bindings

Methods

Public methods

• FSelectInstanceBatchMultiCrit$new()

• FSelectInstanceBatchMultiCrit$assign_result()

• FSelectInstanceBatchMultiCrit$print()

• FSelectInstanceBatchMultiCrit$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelectInstanceBatchMultiCrit$new(
  task,
  learner,
  resampling,
  measures,
  terminator,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL
)

Arguments

Method assign_result()

The FSelector object writes the best found feature subsets and estimated performance values here. For internal use.

Usage

FSelectInstanceBatchMultiCrit$assign_result(xdt, ydt, extra = NULL, ...)

Arguments

Method print()

Printer.

Usage

FSelectInstanceBatchMultiCrit$print(...)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectInstanceBatchMultiCrit$clone(deep = FALSE)

Arguments

Examples

# Feature selection on Palmer Penguins data set


task = tsk("penguins")

# Construct feature selection instance
instance = fsi(
  task = task,
  learner = lrn("classif.rpart"),
  resampling = rsmp("cv", folds = 3),
  measures = msrs(c("classif.ce", "time_train")),
  terminator = trm("evals", n_evals = 4)
)

# Choose optimization algorithm
fselector = fs("random_search", batch_size = 2)

# Run feature selection
fselector$optimize(instance)

# Optimal feature sets
instance$result_feature_set

# Inspect all evaluated sets
as.data.table(instance$archive)

Description

The FSelectInstanceBatchSingleCrit specifies a feature selection problem for a FSelector. The function fsi() creates a FSelectInstanceBatchSingleCrit and the function fselect() creates an instance internally.

The instance contains an ObjectiveFSelectBatch object that encodes the black box objective function a FSelector has to optimize. The instance allows the basic operations of querying the objective at design points (⁠$eval_batch()⁠). This operation is usually done by the FSelector. Evaluations of feature subsets are performed in batches by calling mlr3::benchmark() internally. The evaluated feature subsets are stored in the Archive (⁠$archive⁠). Before a batch is evaluated, the bbotk::Terminator is queried for the remaining budget. If the available budget is exhausted, an exception is raised, and no further evaluations can be performed from this point on. The FSelector is also supposed to store its final result, consisting of a selected feature subset and associated estimated performance values, by calling the method instance$assign_result().

Default Measures

If no measure is passed, the default measure is used. The default measure depends on the task type.

Resources

There are several sections about feature selection in the mlr3book.

• Getting started with wrapper feature selection.

• Do a sequential forward selection Palmer Penguins data set.

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

• Run a feature selection with Shadow Variable Search.

Analysis

For analyzing the feature selection results, it is recommended to pass the archive to as.data.table(). The returned data table is joined with the benchmark result which adds the mlr3::ResampleResult for each feature set.

The archive provides various getters (e.g. ⁠$learners()⁠) to ease the access. All getters extract by position (i) or unique hash (uhash). For a complete list of all getters see the methods section.

The benchmark result (⁠$benchmark_result⁠) allows to score the feature sets again on a different measure. Alternatively, measures can be supplied to as.data.table().

Super classes

bbotk::OptimInstance -> bbotk::OptimInstanceBatch -> bbotk::OptimInstanceBatchSingleCrit -> FSelectInstanceBatchSingleCrit

Active bindings

Methods

Public methods

• FSelectInstanceBatchSingleCrit$new()

• FSelectInstanceBatchSingleCrit$assign_result()

• FSelectInstanceBatchSingleCrit$print()

• FSelectInstanceBatchSingleCrit$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelectInstanceBatchSingleCrit$new(
  task,
  learner,
  resampling,
  measure,
  terminator,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL,
  ties_method = "least_features"
)

Arguments

Method assign_result()

The FSelector writes the best found feature subset and estimated performance value here. For internal use.

Usage

FSelectInstanceBatchSingleCrit$assign_result(xdt, y, extra = NULL, ...)

Arguments

Method print()

Printer.

Usage

FSelectInstanceBatchSingleCrit$print(...)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectInstanceBatchSingleCrit$clone(deep = FALSE)

Arguments

Examples

# Feature selection on Palmer Penguins data set


task = tsk("penguins")
learner = lrn("classif.rpart")

# Construct feature selection instance
instance = fsi(
  task = task,
  learner = learner,
  resampling = rsmp("cv", folds = 3),
  measures = msr("classif.ce"),
  terminator = trm("evals", n_evals = 4)
)

# Choose optimization algorithm
fselector = fs("random_search", batch_size = 2)

# Run feature selection
fselector$optimize(instance)

# Subset task to optimal feature set
task$select(instance$result_feature_set)

# Train the learner with optimal feature set on the full data set
learner$train(task)

# Inspect all evaluated sets
as.data.table(instance$archive)

Description

The 'FSelector“ implements the optimization algorithm.

Details

FSelector is an abstract base class that implements the base functionality each fselector must provide.

Resources

There are several sections about feature selection in the mlr3book.

• Learn more about fselectors.

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

• Run a feature selection with Shadow Variable Search.

Public fields

Active bindings

Methods

Public methods

• FSelector$new()

• FSelector$format()

• FSelector$print()

• FSelector$help()

• FSelector$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelector$new(
  id = "fselector",
  param_set,
  properties,
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments

Method format()

Helper for print outputs.

Usage

FSelector$format(...)

Arguments

Returns

(character()).

Method print()

Print method.

Usage

FSelector$print()

Returns

(character()).

Method help()

Opens the corresponding help page referenced by field ⁠$man⁠.

Usage

FSelector$help()

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelector$clone(deep = FALSE)

Arguments

See Also

Other FSelector: mlr_fselectors, mlr_fselectors_design_points, mlr_fselectors_exhaustive_search, mlr_fselectors_genetic_search, mlr_fselectors_random_search, mlr_fselectors_rfe, mlr_fselectors_rfecv, mlr_fselectors_sequential, mlr_fselectors_shadow_variable_search

Description

The FSelectorAsync implements the asynchronous optimization algorithm.

Details

FSelectorAsync is an abstract base class that implements the base functionality each asynchronous fselector must provide.

Resources

There are several sections about feature selection in the mlr3book.

• Learn more about fselectors.

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

• Run a feature selection with Shadow Variable Search.

Super class

mlr3fselect::FSelector -> FSelectorAsync

Methods

Public methods

• FSelectorAsync$optimize()

• FSelectorAsync$clone()

Method optimize()

Performs the feature selection on a FSelectInstanceAsyncSingleCrit or FSelectInstanceAsyncMultiCrit until termination. The single evaluations will be written into the ArchiveAsyncFSelect that resides in the FSelectInstanceAsyncSingleCrit/FSelectInstanceAsyncMultiCrit. The result will be written into the instance object.

Usage

FSelectorAsync$optimize(inst)

Arguments

Returns

data.table::data.table()

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectorAsync$clone(deep = FALSE)

Arguments

Description

The FSelectorBatch implements the optimization algorithm.

Details

FSelectorBatch is an abstract base class that implements the base functionality each fselector must provide. A subclass is implemented in the following way:

• Inherit from FSelectorBatch.

• Specify the private abstract method ⁠$.optimize()⁠ and use it to call into your optimizer.

• You need to call instance$eval_batch() to evaluate design points.

• The batch evaluation is requested at the FSelectInstanceBatchSingleCrit/FSelectInstanceBatchMultiCrit object instance, so each batch is possibly executed in parallel via mlr3::benchmark(), and all evaluations are stored inside of instance$archive.

• Before the batch evaluation, the bbotk::Terminator is checked, and if it is positive, an exception of class "terminated_error" is generated. In the latter case the current batch of evaluations is still stored in instance, but the numeric scores are not sent back to the handling optimizer as it has lost execution control.

• After such an exception was caught we select the best set from instance$archive and return it.

• Note that therefore more points than specified by the bbotk::Terminator may be evaluated, as the Terminator is only checked before a batch evaluation, and not in-between evaluation in a batch. How many more depends on the setting of the batch size.

• Overwrite the private super-method .assign_result() if you want to decide how to estimate the final set in the instance and its estimated performance. The default behavior is: We pick the best resample experiment, regarding the given measure, then assign its set and aggregated performance to the instance.

Private Methods

• .optimize(instance) -> NULL
  Abstract base method. Implement to specify feature selection of your subclass. See technical details sections.

• .assign_result(instance) -> NULL
  Abstract base method. Implement to specify how the final feature subset is selected. See technical details sections.

Resources

There are several sections about feature selection in the mlr3book.

• Learn more about fselectors.

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

• Run a feature selection with Shadow Variable Search.

Super class

mlr3fselect::FSelector -> FSelectorBatch

Methods

Public methods

• FSelectorBatch$new()

• FSelectorBatch$optimize()

• FSelectorBatch$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelectorBatch$new(
  id = "fselector_batch",
  param_set,
  properties,
  packages = character(),
  label = NA_character_,
  man = NA_character_
)

Arguments

Method optimize()

Performs the feature selection on a FSelectInstanceBatchSingleCrit or FSelectInstanceBatchMultiCrit until termination. The single evaluations will be written into the ArchiveBatchFSelect that resides in the FSelectInstanceBatchSingleCrit / FSelectInstanceBatchMultiCrit. The result will be written into the instance object.

Usage

FSelectorBatch$optimize(inst)

Arguments

Returns

data.table::data.table().

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectorBatch$clone(deep = FALSE)

Arguments

Description

Function to construct a FSelectInstanceBatchSingleCrit or FSelectInstanceBatchMultiCrit.

Usage

fsi(
  task,
  learner,
  resampling,
  measures = NULL,
  terminator,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL,
  ties_method = "least_features"
)

Arguments

Resources

There are several sections about feature selection in the mlr3book.

• Getting started with wrapper feature selection.

• Do a sequential forward selection Palmer Penguins data set.

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

• Run a feature selection with Shadow Variable Search.

Default Measures

If no measure is passed, the default measure is used. The default measure depends on the task type.

Examples

# Feature selection on Palmer Penguins data set


task = tsk("penguins")
learner = lrn("classif.rpart")

# Construct feature selection instance
instance = fsi(
  task = task,
  learner = learner,
  resampling = rsmp("cv", folds = 3),
  measures = msr("classif.ce"),
  terminator = trm("evals", n_evals = 4)
)

# Choose optimization algorithm
fselector = fs("random_search", batch_size = 2)

# Run feature selection
fselector$optimize(instance)

# Subset task to optimal feature set
task$select(instance$result_feature_set)

# Train the learner with optimal feature set on the full data set
learner$train(task)

# Inspect all evaluated sets
as.data.table(instance$archive)

Description

Function to construct a FSelectInstanceAsyncSingleCrit or FSelectInstanceAsyncMultiCrit.

Usage

fsi_async(
  task,
  learner,
  resampling,
  measures = NULL,
  terminator,
  store_benchmark_result = TRUE,
  store_models = FALSE,
  check_values = FALSE,
  callbacks = NULL,
  ties_method = "least_features",
  rush = NULL
)

Arguments

Resources

There are several sections about feature selection in the mlr3book.

• Getting started with wrapper feature selection.

• Do a sequential forward selection Palmer Penguins data set.

The gallery features a collection of case studies and demos about optimization.

• Utilize the built-in feature importance of models with Recursive Feature Elimination.

• Run a feature selection with Shadow Variable Search.

Default Measures

If no measure is passed, the default measure is used. The default measure depends on the task type.

Examples

# Feature selection on Palmer Penguins data set


task = tsk("penguins")
learner = lrn("classif.rpart")

# Construct feature selection instance
instance = fsi(
  task = task,
  learner = learner,
  resampling = rsmp("cv", folds = 3),
  measures = msr("classif.ce"),
  terminator = trm("evals", n_evals = 4)
)

# Choose optimization algorithm
fselector = fs("random_search", batch_size = 2)

# Run feature selection
fselector$optimize(instance)

# Subset task to optimal feature set
task$select(instance$result_feature_set)

# Train the learner with optimal feature set on the full data set
learner$train(task)

# Inspect all evaluated sets
as.data.table(instance$archive)

Description

A mlr3misc::Dictionary storing objects of class FSelector. Each fselector has an associated help page, see mlr_fselectors_[id].

For a more convenient way to retrieve and construct fselectors, see fs()/fss().

Format

R6::R6Class object inheriting from mlr3misc::Dictionary.

Methods

See mlr3misc::Dictionary.

S3 methods

• as.data.table(dict, ..., objects = FALSE)
  mlr3misc::Dictionary -> data.table::data.table()
  Returns a data.table::data.table() with fields "key", "label", "properties" and "packages" as columns. If objects is set to TRUE, the constructed objects are returned in the list column named object.

See Also

Sugar functions: fs(), fss()

Other FSelector: FSelector, mlr_fselectors_design_points, mlr_fselectors_exhaustive_search, mlr_fselectors_genetic_search, mlr_fselectors_random_search, mlr_fselectors_rfe, mlr_fselectors_rfecv, mlr_fselectors_sequential, mlr_fselectors_shadow_variable_search

Examples

as.data.table(mlr_fselectors)
mlr_fselectors$get("random_search")
fs("random_search")

Description

Subclass for asynchronous design points feature selection.

Dictionary

This FSelector can be instantiated with the associated sugar function fs():

fs("async_design_points")

Parameters

design

data.table::data.table
Design points to try in search, one per row.

Super classes

mlr3fselect::FSelector -> mlr3fselect::FSelectorAsync -> mlr3fselect::FSelectorAsyncFromOptimizerAsync -> FSelectorAsyncDesignPoints

Methods

Public methods

• FSelectorAsyncDesignPoints$new()

• FSelectorAsyncDesignPoints$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelectorAsyncDesignPoints$new()

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectorAsyncDesignPoints$clone(deep = FALSE)

Arguments

See Also

Other FSelectorAsync: mlr_fselectors_async_exhaustive_search, mlr_fselectors_async_random_search

Description

Feature Selection using the Asynchronous Exhaustive Search Algorithm. Exhaustive Search generates all possible feature sets. The feature sets are evaluated asynchronously.

Details

The feature selection terminates itself when all feature sets are evaluated. It is not necessary to set a termination criterion.

Dictionary

This FSelector can be instantiated with the associated sugar function fs():

fs("async_exhaustive_search")

Control Parameters

max_features

integer(1)
Maximum number of features. By default, number of features in mlr3::Task.

Super classes

mlr3fselect::FSelector -> mlr3fselect::FSelectorAsync -> FSelectorAsyncExhaustiveSearch

Methods

Public methods

• FSelectorAsyncExhaustiveSearch$new()

• FSelectorAsyncExhaustiveSearch$optimize()

• FSelectorAsyncExhaustiveSearch$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelectorAsyncExhaustiveSearch$new()

Method optimize()

Starts the asynchronous optimization.

Usage

FSelectorAsyncExhaustiveSearch$optimize(inst)

Arguments

Returns

data.table::data.table.

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectorAsyncExhaustiveSearch$clone(deep = FALSE)

Arguments

See Also

Other FSelectorAsync: mlr_fselectors_async_design_points, mlr_fselectors_async_random_search

Description

Feature selection using Asynchronous Random Search Algorithm.

Dictionary

This FSelector can be instantiated with the associated sugar function fs():

fs("async_random_search")

Control Parameters

max_features

integer(1)
Maximum number of features. By default, number of features in mlr3::Task.

Super classes

mlr3fselect::FSelector -> mlr3fselect::FSelectorAsync -> FSelectorAsyncRandomSearch

Methods

Public methods

• FSelectorAsyncRandomSearch$new()

• FSelectorAsyncRandomSearch$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelectorAsyncRandomSearch$new()

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectorAsyncRandomSearch$clone(deep = FALSE)

Arguments

Source

Bergstra J, Bengio Y (2012). “Random Search for Hyper-Parameter Optimization.” Journal of Machine Learning Research, 13(10), 281–305. https://jmlr.csail.mit.edu/papers/v13/bergstra12a.html.

See Also

Other FSelectorAsync: mlr_fselectors_async_design_points, mlr_fselectors_async_exhaustive_search

Description

Feature selection using user-defined feature sets.

Details

The feature sets are evaluated in order as given.

The feature selection terminates itself when all feature sets are evaluated. It is not necessary to set a termination criterion.

Dictionary

This FSelector can be instantiated with the associated sugar function fs():

fs("design_points")

Parameters

batch_size

integer(1)
Maximum number of configurations to try in a batch.

design

data.table::data.table
Design points to try in search, one per row.

Super classes

mlr3fselect::FSelector -> mlr3fselect::FSelectorBatch -> mlr3fselect::FSelectorBatchFromOptimizerBatch -> FSelectorBatchDesignPoints

Methods

Public methods

• FSelectorBatchDesignPoints$new()

• FSelectorBatchDesignPoints$clone()

Method new()

Creates a new instance of this R6 class.

Usage

FSelectorBatchDesignPoints$new()

Method clone()

The objects of this class are cloneable with this method.

Usage

FSelectorBatchDesignPoints$clone(deep = FALSE)

Arguments

See Also

Other FSelector: FSelector, mlr_fselectors, mlr_fselectors_exhaustive_search, mlr_fselectors_genetic_search, mlr_fselectors_random_search, mlr_fselectors_rfe, mlr_fselectors_rfecv, mlr_fselectors_sequential, mlr_fselectors_shadow_variable_search

Examples

# Feature Selection


# retrieve task and load learner
task = tsk("pima")
learner = lrn("classif.rpart")

# create design
design = mlr3misc::rowwise_table(
  ~age, ~glucose, ~insulin, ~mass, ~pedigree, ~pregnant, ~pressure, ~triceps,
  TRUE, FALSE,    TRUE,     TRUE,  FALSE,     TRUE,       FALSE,    TRUE,
  TRUE, TRUE,     FALSE,    TRUE,  FALSE,     TRUE,       FALSE,    FALSE,
  TRUE, FALSE,    TRUE,     TRUE,  FALSE,     TRUE,       FALSE,    FALSE,
  TRUE, FALSE,    TRUE,     TRUE,  FALSE,     TRUE,       TRUE,     TRUE
)

# run feature selection on the Pima Indians diabetes data set
instance = fselect(
  fselector = fs("design_points", design = design),
  task = task,
  learner = learner,
  resampling = rsmp("holdout"),
  measure = msr("classif.ce")
)

# best performing feature set
instance$result

# all evaluated feature sets
as.data.table(instance$archive)

# subset the task and fit the final model
task$select(instance$result_feature_set)
learner$train(task)

Description

Feature Selection using the Exhaustive Search Algorithm. Exhaustive Search generates all possible feature sets.

Details

The feature selection terminates itself when all feature sets are evaluated. It is not necessary to set a termination criterion.

Dictionary

This FSelector can be instantiated with the associated sugar function fs():

fs("exhaustive_search")

Control Parameters

max_features

integer(1)
Maximum number of features. By default, number of features in mlr3::Task.

Super classes

mlr3fselect::FSelector -> mlr3fselect::FSelectorBatch -> FSelectorBatchExhaustiveSearch

Methods

Public methods

• FSelectorBatchExhaustiveSearch$new()

• FSelectorBatchExhaustiveSearch$clone()