scandeval.enums

source module scandeval.enums

Enums used in the project.

Classes

• AutoStrEnum — StrEnum where auto() returns the field name in lower case.

• Device — The compute device to use for the evaluation.

• InferenceBackend — The backend used for model inference.

• ModelType — The type of a model.

• GenerativeType — The type of a generative model.

• DataType — The data type of the model weights.

• BatchingPreference — The preference for batching.

• TaskGroup — The overall task group of a task.

source enum AutoStrEnum()

Bases : str, Enum

StrEnum where auto() returns the field name in lower case.

source enum Device()

Bases : AutoStrEnum

The compute device to use for the evaluation.

Attributes

• CPU —

  CPU device.

• MPS —

  MPS GPU, used in M-series MacBooks.

• CUDA —

  CUDA GPU, used with NVIDIA GPUs.

source enum InferenceBackend()

Bases : AutoStrEnum

The backend used for model inference.

Attributes

• TRANSFORMERS —

  Hugging Face transformers library.

• VLLM —

  VLLM library.

• LITELLM —

  LiteLLM library.

• NONE —

  No inference backend used (e.g., for human evaluation).

source enum ModelType()

Bases : AutoStrEnum

The type of a model.

Attributes

• ENCODER —

  An encoder (i.e., BERT-style) model.

• GENERATIVE —

  A generative model. Can be either decoder or encoder-decoder (aka seq2seq).

• HUMAN —

  Human evaluator.

source enum GenerativeType()

Bases : AutoStrEnum

The type of a generative model.

Attributes

• BASE —

  A base (i.e., pretrained) generative model.

• INSTRUCTION_TUNED —

  An instruction-tuned generative model.

• REASONING —

  A generative reasoning model.

source enum DataType()

Bases : AutoStrEnum

The data type of the model weights.

Attributes

• FP32 —

  32-bit floating point.

• FP16 —

  16-bit floating point.

• BF16 —

  16-bit bfloat.

source enum BatchingPreference()

Bases : AutoStrEnum

The preference for batching.

Attributes

• NO_PREFERENCE —

  No preference for batching.

• SINGLE_SAMPLE —

  Single sample batching.

• ALL_AT_ONCE —

  All samples at once batching.

source enum TaskGroup()

Bases : AutoStrEnum

The overall task group of a task.

Attributes

• SEQUENCE_CLASSIFICATION —

  Classification of documents.

• MULTIPLE_CHOICE_CLASSIFICATION —

  Classification of documents with multiple-choice options.

• TOKEN_CLASSIFICATION —

  Token-level classification.

• QUESTION_ANSWERING —

  Extractive question answering.

• TEXT_TO_TEXT —

  Text-to-text generation.

• SPEED —

  Speed benchmark.