scandeval.cli

Command-line interface for benchmarking.

Functions

benchmark — Benchmark pretrained language models on language tasks.

source benchmark(model: tuple[str], dataset: tuple[str], language: tuple[str], model_language: tuple[str], dataset_language: tuple[str], raise_errors: bool, task: tuple[str], batch_size: str, progress_bar: bool, save_results: bool, cache_dir: str, api_key: str | None, force: bool, verbose: bool, device: str | None, trust_remote_code: bool, use_flash_attention: bool | None, clear_model_cache: bool, evaluate_test_split: bool, few_shot: bool, num_iterations: int, api_base: str | None, api_version: str | None, debug: bool, only_allow_safetensors: bool) → None

Benchmark pretrained language models on language tasks.