scandeval.speed_benchmark

source module scandeval.speed_benchmark

Benchmarking model inference speed.

Functions

• benchmark_speed — Benchmark model inference speed.

• benchmark_speed_single_iteration — Run a single iteration of the speed benchmark.

source benchmark_speed(model: BenchmarkModule, benchmark_config: BenchmarkConfig) → list[dict[str, float]]

Benchmark model inference speed.

Parameters

• model : BenchmarkModule —

  Model to use.

• benchmark_config : BenchmarkConfig —

  Configuration for the benchmark.

Returns

• list[dict[str, float]] — Dictionary of scores.

source benchmark_speed_single_iteration(model: BenchmarkModule, itr_idx: int) → dict[str, float]

Run a single iteration of the speed benchmark.

Parameters

• model : BenchmarkModule —

  The model to use in the benchmark.

• itr_idx : int —

  The index of the iteration.

Returns

• dict[str, float] — A dictionary containing the scores for the current iteration.

Raises

• InvalidBenchmark

• ValueError