Creating a new Learner

Defining the Parameter Set of a Learner

The parameter set of a learner is the set of hyperparameters used in model training and predicting, this is given as a paradox::ParamSet. The set consists of a list of hyperparameters, where each has a specific class for the hyperparameter type. For an explanation of parameters and the ParamSet class we refer to the mlr3book, where we show how to create parameter sets to define search spaces for hyperparameter tuning. However, there are some aspects of parameters that are mostly relevant when creating new Learners, so we cover them here. These are:

• tags that organize the parameters, e.g. whether they are used during training or prediction.
• default values which give information about the value that is used by the upstream function when no value is set. Note that this is not necessarily the default of the function signature but what is used internally: e.g. the default of parameter a for a function is NULL, but internally the value for a is set to 1 it is left as NULL by the caller.
• initial values are the initial $values of the parameter set that are set during construction.
• custom checks which allow to specify custom constraints for untyped parameters (ParamUty).

Tags

Each parameter has one or more tags that determine in which method they are used. I.e. parameters that are used during training must be tagged with "train" and those that are used during prediction with "predict". In our case there are only training parameters. Parameters with certain tags, can be retrieved by calling the $get_values() method of a ParamSet with the tag argument.

Furthermore, there are other tags that serve specific purposes:

• The tag "threads" should be used (if applicable) to tag the parameter that determines the number of threads used for the learner’s internal parallelization. Once a learner is constructed, this parameter can be set using set_threads().
• The tag "required" should be used to tag parameters that must be provided for the algorithm to be executable.

Default Values

The default values of each parameter indicate which value the upstream function (in this case rpart::rpart()) uses if no parameter is specified. These have to be retrieved from the package’s documentation. If parameters have no default, they have to be tagged with the "required" tag. For such parameters, we recommended setting the parameter to a value in the initialize method, so that the learner can be executed after creation. Note that setting a parameter value in the $initialize() method does not change the parameter’s default value (as the default denotes what happens when no value is set).

Parameters expect the default values to be of their respective type, e.g. the default of ParamInt must be an integer. Some packages have R expressions as default values which cannot be properly expressed in paradox. In such cases, a practical compromise is to not specify any default value when calling p_int().

Initial Values

In some cases you want to change the parameter values during initialization. You can do this by changing the param_set$values during the learner’s construction. You can see we have done this for regr.rpart_simple where the default for xval is changed to 0. Note however that we still annotate the default of the xval parameter as 10. It is recommended to initialize parameter values for good reasons and document this properly when doing so. When contributing a learner to mlr3extralearners this information should go into the Initial parameter values section.

Custom Checks

In paradox only integers, numerics, factors, and logicals have dedicated parameter types. For all other parameters, the ParamUty class has to be used, which is constructable via the p_uty() function. It has the argument custom_check, which allows specifying custom constraints. This function return TRUE if the input is valid and a character(1) with the error message otherwise.

Custom Parameters

Sometimes you might want to add paramters that are not present in the upstream function. One example is the mtry.ratio parameter of lrn("surv.ranger"). Such parameters should usually not have default values (because we are here not calling an upstream function with a default behaviour) but instead have their value initialized during construction and have the "required" tag. This avoids having to deal with the case where this parameter has no value set. When contributing a learner to mlr3extralearners this information should go into the Custom parameters section.

Train function

The train function takes a Task as input and must return a model, i.e. some R object that allows the learner to make predictions. We want to translate the following call of rpart::rpart() into code that can be used inside the .train() method.

First, we write something down that works completely without mlr3:

data = mtcars
model = rpart::rpart(mpg ~ ., data = mtcars, xval = 0)

We need to pass the formula notation mpg ~ ., the data and the hyperparameters. To get the hyperparameters, we call self$param_set$get_values(tag = "train") and thereby query all parameters that are using during "train". Then, the dataset is extracted from the Task. Because the learner has the property "weights", we insert the weights of the task if there are any. Then we obtain the formula from the task using task$formula() and access the training data using task$data(). Last, we call the upstream function rpart::rpart() with the data and pass all hyperparameters via argument .args using the mlr3misc::invoke() function. The latter is simply an optimized version of do.call() that we use within the mlr3 ecosystem. The return value of this method will be available as the $model slot of the trained learner.

.train = function(task) {
  pv = self$param_set$get_values(tags = "train")
  if ("weights" %in% task$properties) {
    pv$weights = task$weights$weight
  }
  formula = task$formula()
  data = task$data()
  invoke(
    rpart::rpart,
    formula = formula,
    data = data,
    .args = pv
  )
}

At this point, an explanation for the difference between the public method $train() and the private method $.train() is in order. The former essentially calls the latter and additionally performs some checks and tasks that are the same for every learner. The same holds for the predict method.

Predict function

The internal method $.predict() also operates on a Task as well as on the fitted model that has been created by the $train() call previously and has been stored in self$model. The return value must contain the required information to produce a mlr3::Prediction object that is returned when $predict() is called on a learner. We proceed analogously to what we did in the previous section. We start with a version without any mlr3 objects and continue to replace objects until we have reached the desired interface:

# inputs:
task = tsk("mtcars")
self = list(model = rpart::rpart(task$formula(), data = task$data()))
data = mtcars

# response prediction
response = predict(self$model, newdata = data)

Next, we transition from data to a Task again and construct a list with the return type requested by the user. This is stored in the $predict_type slot of a learner class. Because the regr.rpart_simple learner we are implementing has only one predict type, we do not have to take the value of self$predict_type into account. For, e.g., regression learners with predict types "response" and "se" or classification learners with predict types "response" and "prob", the predict method must return the prediction requested by the user.

The final $.predict() method is below, we could omit the pv line as there are no parameters with the "predict" tag but we keep it here to be consistent:

.predict = function(task) {
  pv = self$param_set$get_values(tags = "predict")
  # ensure same column order in train and predict
  newdata = mlr3extralearners:::ordered_features(task, self)
  response = invoke(predict, self$model, newdata = newdata, .args = pv)
  list(response = unname(response))
}

Beware that you cannot rely on the column order of the data returned by task$data() as the order of columns may be different from the order of the columns during $.train(). The newdata line ensures the ordering is the same by calling the same order as in train!

Optional Extractors

Specific learner implementations are free to implement additional getters to ease the access of certain parts of the model in the inherited subclasses. Some of these methods are standardized in mlr3, c.f. the Learner documentation. If it is one of the standardized extractors, some custom code might have to be written for the return value to comply with the mlr3 standards (see the Learner documentation). Because we specified earlier that the learner has the property "importance", we implement the public $importance() method. They access the $model slot of a learner after training and return one of its fields. Because the $importance() method is standardized, it must return the importance scores as a sorted numeric vector, the names being the features.

importance = function() {
  if (is.null(self$model)) {
    stopf("No model stored")
  }
  # importance is only present if there is at least on split
  if (is.null(self$model$variable.importance)) {
    importance = set_names(numeric())
  } else {
    importance = sort(self$model$variable.importance, decreasing = TRUE)
  }
  return(importance)
}

Testing the learner

Once your learner is created, you should write tests to verify its correctness. While you should also do this when implementing a learner only for your own use, this is absolutely required when you want to contribute a new learner to mlr3extralearners. Besides manual tests, the mlr3 package also provides automatic tests to check that a learner satisfies some basic sanity checks and that it implements all the available parameters of the upstream function. The manual page for these functions can be accessed via ?mlr_test_helpers.

For a bare-bone check you can just try to run a simple $train() call.

task = tsk("mtcars")
learner = LearnerRegrRpartSimple$new()
learner$train(task)
p = learner$predict(task)

p$score(msr("regr.mse"))

## regr.mse 
## 9.127693

learner$importance()

##       cyl      disp        hp        wt      qsec        vs      carb      gear 
## 724.18935 721.08062 702.29023 573.72817 442.05737 395.01237  31.36333  15.68167

If it runs without erroring, that’s a very good start!

lapply(list.files(system.file("testthat", package = "mlr3"), pattern = "^helper.*\\.[rR]", full.names = TRUE), source)
# mlr3proba
# lapply(list.files(system.file("testthat", package = "mlr3proba"), pattern = "^helper.*\\.[rR]", full.names = TRUE), source)

library(testthat)
test_that("autotest", {
  learner = LearnerRegrRpartSimple$new()

  # basic learner properties
  expect_learner(learner)

  # you can skip tests using the `exclude` argument
  result = run_autotest(learner)
  expect_true(result, info = result$error)
})

## Test passed

test_that("paramtest", {
  learner = LearnerRegrRpartSimple$new()

  exclude = c("formula", "data", "weights", "subset", "na.action", "method", "model",
    "x", "y", "parms", "control", "cost", "keep_model")
  result = run_paramtest(learner, list(rpart::rpart, rpart::rpart.control), exclude, tag = "train")
  expect_true(result, info = result$error)

  exclude = c(
    "object", # handled internally
    "newdata", # handled internally
    "type", # handled internally
    "na.action", # handled internally
    "model" # not implemented
  )
  result = run_paramtest(learner, rpart:::predict.rpart, exclude, tag = "predict")
  expect_true(result, info = result$error)
})

## Test passed

Contributing to mlr3extralearners

When adding a Learner to mlr3extralearners there are some additional requirements that have to be satisfied:

• Adding the line .extralrns_dict$add("<learner-id>", function() <LearnerName>$new()) to the bottom of the learner file to add the learner to the learners Dictionary.
• Add the learner dependencies to Suggests (and not e.g. to Imports) in the DESCRIPTION file.
• The learner must be properly documented. When using mlr3extralearners::create_learner() this is automatically generated. Otherwise you can use and adapt the documentation of an already implemented learner.
• Conforming to the mlr3 style guide.
• The tests must pass in the CI (also check the CodeFactor results).
• Update the NEWS.md file and describe that you added the learner.
• Add yourself as a contributor