vertebrae.benchmark

Benchmark runner.

Classes

Benchmark

Run one or more extractors against a labeled dataset.

Module Contents

class vertebrae.benchmark.Benchmark(dataset, extractors=None, scoring_config=None, stability_config=None, label_view_config=None, target_view_config=None, separatix_config=None, cache_config=None, compression_config=None, compression_configs=None, embedding_config=None, memory_config=None, execution=None, execution_config=None, segmentation_config=None, structured_aligners=None, metrics=None, primary_metric=None, resource_profiling_config=None)

Run one or more extractors against a labeled dataset.

Parameters:

• dataset (Any) – Dataset object with inputs and labels.

• extractors (Optional[Iterable[Any]]) – Optional iterable of extractors to evaluate.

• scoring_config (Optional[Union[vertebrae.config.OverlapScoringConfig, vertebrae.config.ContinuousOverlapScoringConfig]]) – OverlapIndex scoring configuration.

• stability_config (Optional[vertebrae.config.StabilityConfig]) – Stability-analysis configuration.

• cache_config (Optional[vertebrae.config.CacheConfig]) – Embedding cache configuration.

• embedding_config (Optional[vertebrae.config.EmbeddingConfig]) – Embedding batching and streaming configuration.

• execution (Optional[Any]) – Optional local, Ray, or Dask execution backend. When omitted, the benchmark uses the direct in-process path.

• execution_config (Optional[vertebrae.config.ExecutionConfig]) – Artifact-backed execution and sharding settings used only when an explicit backend is provided.

• label_view_config (Optional[vertebrae.config.LabelViewConfig])

• target_view_config (Optional[vertebrae.config.TargetViewConfig])

• separatix_config (Optional[vertebrae.config.SeparatixConfig])

• compression_config (Optional[vertebrae.config.EmbeddingCompressionConfig])

• compression_configs (Optional[Iterable[vertebrae.config.EmbeddingCompressionConfig]])

• memory_config (Optional[vertebrae.config.MemoryConfig])

• segmentation_config (Optional[vertebrae.config.SegmentationConfig])

• structured_aligners (Optional[dict[str, Any]])

• metrics (Optional[Iterable[Any]])

• primary_metric (Optional[str])

• resource_profiling_config (Optional[vertebrae.config.ResourceProfilingConfig])

add_extractor(extractor)

Add an extractor to this benchmark.

Parameters:

extractor (Any) – Feature extractor implementing the vertebrae protocol.

Returns:

This benchmark instance for fluent chaining.

Return type:

Benchmark

run()

Run feature extraction, scoring, diagnostics, and reporting aggregation.

Returns:

Aggregated benchmark result.

Raises:

ValueError – If no extractors are configured or dataset validation fails.

Return type:

vertebrae.results.BenchmarkResult