scandeval.exceptions

source module scandeval.exceptions

Exceptions to used by other functions.

Classes

• InvalidBenchmark — The (model, dataset) combination cannot be benchmarked.

• InvalidModel — The model cannot be benchmarked on any datasets.

• HuggingFaceHubDown — The Hugging Face Hub seems to be down.

• NoInternetConnection — There seems to be no internet connection.

• NaNValueInModelOutput — There is a NaN value in the model output.

• FlashAttentionNotInstalled — The flash-attn package has not been installed.

• NeedsExtraInstalled — The evaluation requires extra to be installed.

• NeedsManualDependency — The evaluation requires a dependency to be manually installed.

• NeedsAdditionalArgument — The evaluation requires additional arguments to the scandeval command.

• NeedsEnvironmentVariable — The evaluation requires an environment variable to be set.

source class InvalidBenchmark()

Bases : Exception

The (model, dataset) combination cannot be benchmarked.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class InvalidModel()

Bases : Exception

The model cannot be benchmarked on any datasets.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class HuggingFaceHubDown()

Bases : Exception

The Hugging Face Hub seems to be down.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class NoInternetConnection()

Bases : Exception

There seems to be no internet connection.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class NaNValueInModelOutput()

Bases : Exception

There is a NaN value in the model output.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class FlashAttentionNotInstalled()

Bases : Exception

The flash-attn package has not been installed.

Initialize the exception.

Parameters

• message : str —

  The message to display.

source class NeedsExtraInstalled()

Bases : InvalidModel

The evaluation requires extra to be installed.

Initialize the exception.

Parameters

• extra : str —

  The extra that needs to be installed.

source class NeedsManualDependency()

Bases : InvalidModel

The evaluation requires a dependency to be manually installed.

Initialize the exception.

Parameters

• package : str —

  The package that needs to be manually installed.

source class NeedsAdditionalArgument(script_argument: str, run_with_cli: bool)

Bases : InvalidModel

The evaluation requires additional arguments to the scandeval command.

Initialize the exception.

Parameters

• cli_argument : str —

  The argument that needs to be passed to the scandeval command.

• script_argument : str —

  The argument that needs to be passed to the Benchmarker class.

• run_with_cli : bool —

  Whether the benchmark is being run with the CLI.

source class NeedsEnvironmentVariable()

Bases : InvalidModel

The evaluation requires an environment variable to be set.

Initialize the exception.

Parameters

• env_var : str —

  The environment variable that needs to be set.