scandeval.task_utils.multiple_choice_classification

source module scandeval.task_utils.multiple_choice_classification

Utility functions related to the multiple-choice classification task group.

Classes

• MultipleChoiceClassificationTrainer — Trainer subclass for question answering tasks.

Functions

• prepare_examples — Prepare the features.

• postprocess_predictions_and_labels — Postprocess the predictions and labels.

source class MultipleChoiceClassificationTrainer()

Bases : Trainer

Trainer subclass for question answering tasks.

Methods

• evaluate — Evaluate the model on the given dataset.

source method MultipleChoiceClassificationTrainer.evaluate(eval_dataset: Dataset | None = None, ignore_keys: list[str] | None = None, metric_key_prefix: str = 'eval') → dict[str, float] | None

Evaluate the model on the given dataset.

Parameters

• eval_dataset : Dataset | None —

  The dataset to evaluate on. If None, then use the stored evaluation dataset.

• ignore_keys : list[str] | None —

  The keys to ignore when computing the metrics.

• metric_key_prefix : str —

  The prefix to use for the metric keys.

Returns

• dict[str, float] | None — The metrics computed on the evaluation dataset.

source prepare_examples(examples: BatchEncoding, tokenizer: PreTrainedTokenizer) → BatchEncoding

Prepare the features.

Parameters

• examples : BatchEncoding —

  The examples to prepare.

• tokenizer : PreTrainedTokenizer —

  The tokenizer to use to prepare the examples.

Returns

• BatchEncoding — The prepared examples.

source postprocess_predictions_and_labels(predictions: np.ndarray, dataset: Dataset) → tuple[Predictions, Labels]

Postprocess the predictions and labels.

Parameters

• predictions : np.ndarray —

  The model predictions, of shape (num_examples, 2).

• dataset : Dataset —

  The dataset containing the examples.

Returns

• tuple[Predictions, Labels] — The postprocessed predictions and labels.