Package 'mlr3hf'

Description

Converts an HFData object into an mlr3 DataBackend, triggering download and parsing of the underlying dataset if this has not already happened.

Usage

## S3 method for class 'HFData'
as_data_backend(data, ...)

Arguments

Value

An mlr3 DataBackend.

Description

Converts an HFData object into an mlr3 Task, either a TaskClassif or a TaskRegr. The target column and task type can be taken from the HFData object or overridden explicitly.

When task_type = "auto", the task type is inferred from the class of the target column: factor and logical columns become classification tasks, numeric and integer columns become regression tasks.

Usage

## S3 method for class 'HFData'
as_task(x, target_names = NULL, task_type = NULL, ...)

Arguments

Value

An mlr3::TaskClassif or mlr3::TaskRegr object.

Description

Downloads and caches a specific file from the canonical (original) version of a dataset repository on the Hugging Face Hub — that is, the dataset as published by its authors, rather than the auto-generated Parquet conversion (see cache_parquet for that). If the file has already been cached locally, it is reused instead of being downloaded again.

Usage

cache_hfhub(
  repo_id,
  file_name,
  local_files_only = FALSE,
  revision = "main",
  ...
)

Arguments

Details

Files are cached in a local directory following the Hugging Face Hub caching convention, keyed by repo_id, revision, and file_name. Subsequent calls with the same arguments will return the cached path without re-downloading, unless the remote file has changed.

Value

Character string giving the local file path where the cached file is stored.

See Also

cache_parquet for caching the auto-converted Parquet version of a dataset instead.

https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache

Examples

cache_hfhub(
  repo_id   = "scikit-learn/iris",
  file_name = "iris.csv"
)

# Use a cached copy only, without attempting a network request
cache_hfhub(
  repo_id          = "scikit-learn/iris",
  file_name        = "iris.csv",
  local_files_only = TRUE
)

Description

Downloads and caches the Parquet files for a dataset hosted on the Hugging Face Hub, using the auto-generated refs/convert/parquet branch. This branch contains a Parquet conversion of the dataset that Hugging Face maintains automatically, which avoids the need to load the dataset via its original (often script-based) loading format.

Usage

cache_parquet(
  repo_id,
  config,
  split = NULL,
  revision = "refs%2Fconvert%2Fparquet",
  ...
)

Arguments

Details

Files are downloaded from the refs/convert/parquet branch maintained by Hugging Face, not from the dataset's default branch. Large datasets may be split across multiple Parquet shards; all matching shards for the requested split are downloaded and cached.

Value

A named list with one entry per cached split. Each entry is a character vector of local file paths to the cached Parquet shard(s) for that split (a single path if the split has one shard, multiple paths if it was sharded across several files).

See Also

https://huggingface.co/docs/datasets-server/en/parquet

Examples

cache_parquet(
  repo_id = "scikit-learn/iris",
  config  = "default"
)

cache_parquet(
  repo_id = "fancyzhx/amazon_polarity",
  config  = "amazon_polarity",
  split   = "train"
)

Description

it helps in downloading the description

Usage

download_desc(repo_id, revision = "main", ..., files_metadata = FALSE)

Arguments

Value

list containing description and downloads of the dataset

Examples

download_desc("scikit-learn/iris")

Description

A single call to ⁠/size⁠ per dataset is enough. First the configs[] array is checked: if the dataset does not have exactly one config, nrows/ncols are both NA. If it does have exactly one config, nrows is computed by summing num_rows across all of that config's entries in the splits[] array (rather than trusting the config-level aggregate directly), and ncols is read from num_columns on one of those splits (column count doesn't vary across splits of the same config).

Usage

fetch_dataset_meta(ids, max_concurrent = 10, timeout = 30)

Arguments

Details

If a dataset has zero or more than one config, both nrows and ncols are set to NA – there's no single unambiguous row/column count to report in that case.

Value

A data.table with columns: id, nrows, ncols.

Description

An R6::R6Class that wraps a dataset hosted on the Hugging Face Hub and exposes it as an mlr3 compatible data backend / task. Data is loaded lazily: no download or parsing happens until fields that require access to the underlying data (e.g. data, nrow, ncol, colnames) are accessed for the first time. Results are then cached for the lifetime of the object.

A dataset can be identified either by a config (one of the dataset's predefined configurations, loaded via its parquet export) or by a single file_name within the repository. Exactly one of the two must be supplied before data can be retrieved.

All active bindings are read-only; attempting to assign to them raises an error. See the Fields section below for details on each one.

Note on clone(): like every R6 object, HFData objects have an inherited clone(deep = FALSE) method for copying the object (deep controls whether nested R6 fields are also cloned). It is not re-documented here since it isn't defined in this class's source, but behaves exactly as described in R6::R6Class.

Active bindings

Methods

Public methods

• HFData$new()

• HFData$print()

• HFData$clone()

Method new()

Create a new HFData object.

Usage

HFData$new(
  repo_id,
  config = NULL,
  file_name = NULL,
  split = NULL,
  target = NULL,
  primary_key = NULL,
  task_type = c("auto", "classif", "regr"),
  ...
)

Arguments

Returns

A new HFData object.

Method print()

Print a short summary of the HFData object, including its dimensions, target column, and repository storage size.

Usage

HFData$print()

Returns

self, invisibly (called for its side effect).

Method clone()

The objects of this class are cloneable with this method.

Usage

HFData$clone(deep = FALSE)

Arguments

Description

Creates an HFData object for a dataset hosted on the Hugging Face Hub. The returned object can be converted to an mlr3 task using mlr3::as_task() or accessed through its active bindings.

Usage

hfdt(
  repo_id,
  config = NULL,
  file_name = NULL,
  target = NULL,
  primary_key = NULL,
  split = NULL,
  revision = NULL,
  task_type = c("auto", "classif", "regr"),
  ...
)

Arguments

Value

An HFData object.

Description

Query datasets hosted on https://huggingface.co/datasets via the Hugging Face Hub API, returning metadata such as dataset id, gated status, and download counts.

Usage

list_datasets(
  num_of_dataset = 100,
  chunk_size = 100,
  search = NULL,
  author = NULL,
  fetch_meta = TRUE,
  max_concurrent = 10,
  ...
)

Arguments

Value

(data.table()) of results with columns id, nrows, and ncols or an empty data.table() if no datasets are found.

References

https://huggingface.co/docs/hub/api

Examples

## Not run: 
dat <- list_datasets(num_of_dataset = 50)
dat <- list_datasets(num_of_dataset = 500, chunk_size = 100)
dat <- list_datasets(author = "a4n9i")

## End(Not run)

Description

Provides integration between Hugging Face datasets and the mlr3 ecosystem. Datasets can be downloaded from the Hugging Face Hub and converted directly into mlr3 Tasks.

mlr3 Integration

The package registers the "hf" task in mlr3::mlr_tasks.

task = tsk(
  "hf",
  repo_id = "scikit-learn/iris",
  config = "default",
  target = "species"
)

Alternatively, users can create an HFData object and convert it using as_task().

Caching

Downloaded datasets are cached locally to avoid repeated downloads.

Documentation

See the package vignettes for examples of loading datasets, converting them to mlr3 tasks, and working with large datasets.

Author(s)

Maintainer: Sebastian Fischer [email protected]

Authors:

• Toby Hocking [email protected]

• Anjani Nandan [email protected]

See Also

Useful links:

• https://mlr-org.github.io/mlr3hf/

• https://github.com/mlr-org/mlr3hf

• Report bugs at https://github.com/mlr-org/mlr3hf/issues