scandeval.task_utils.text_to_text

source module scandeval.task_utils.text_to_text

Utility functions related to the text-to-text task group.

Functions

• compute_metrics — Compute the metrics needed for evaluation.

• extract_labels_from_generation — Extract the predicted labels from the generated output.

source compute_metrics(model_outputs_and_labels: tuple[Predictions, Labels], dataset_config: DatasetConfig, benchmark_config: BenchmarkConfig) → dict[str, float]

Compute the metrics needed for evaluation.

Parameters

• model_outputs_and_labels : tuple[Predictions, Labels] —

  The first sequence contains the model outputs and the second sequence contains the true labels.

• dataset_config : DatasetConfig —

  The configuration of the dataset.

• benchmark_config : BenchmarkConfig —

  The configuration of the benchmark.

Returns

• dict[str, float] — A dictionary with the names of the metrics as keys and the metric values as values.

Raises

• InvalidBenchmark

source extract_labels_from_generation(input_batch: dict[str, list], model_output: GenerativeModelOutput) → list[t.Any]

Extract the predicted labels from the generated output.

Parameters

• input_batch : dict[str, list] —

  The input batch, where the keys are the feature names and the values are lists with the feature values.

• model_output : GenerativeModelOutput —

  The raw generated output of the model.

Returns

• list[t.Any] — The predicted labels.