scoring

All OverlapIndex scoring in vertebrae goes through one internal adapter: OverlapIndexScorer.

vertebrae can also run a default Separatix complexity diagnostic through the internal SeparatixScorer adapter. Separatix does not affect extractor ranking; it adds classifier-complexity guidance on top of the overlap result when enabled. When a dataset declares groups, vertebrae forwards them to Separatix so supervised evaluation and structural evidence respect those independence units. A grouped diagnostic that lacks sufficient cross-group class support is recorded as skipped; vertebrae never retries it with a row-level split.

Fixed metric backend

vertebrae depends on the external OverlapIndex package and does not reimplement it. Currently, the backend is fixed internally to MiniBatchKMeans:

OverlapIndex(
    model_type="MiniBatchKMeans",
    kmeans_k=resolved_k,
    kmeans_kwargs=resolved_kmeans_kwargs,
    offline_chunk_size=offline_chunk_size,
)

This backend choice is not exposed as a public user option.

OverlapScoringConfig

Use OverlapScoringConfig to tune the supported MiniBatchKMeans-facing settings for classification and multi-label datasets:

from vertebrae import OverlapScoringConfig

config = OverlapScoringConfig(
    k="auto",
    min_k=10,
    max_k=50,
    min_samples_per_cluster=5,
    kmeans_kwargs={"batch_size": 256},
    offline_chunk_size=10_000,
    normalize_embeddings=True,
    exclude_classes=None,
)

Classification and multi-label OverlapIndex scores are bounded to [0, 1]: 1.0 indicates perfect class separation and 0.0 indicates perfect class overlap.

Key fields:

  • k: an integer, a class-to-k mapping, or "auto".

  • min_k and max_k: bounds used during automatic per-class resolution.

  • min_samples_per_cluster: guardrail that prevents too many prototypes for a small class.

  • kmeans_kwargs: extra keyword arguments forwarded to MiniBatchKMeans.

  • offline_chunk_size: chunk size passed to OverlapIndex.fit_offline(...).

  • normalize_embeddings: enables L2 row normalization before scoring.

  • max_dense_bytes: fallback limit for downstream dense-only diagnostics, including Separatix when its own limit is not set. Sparse overlap scoring does not densify the full embedding matrix.

  • exclude_classes: class id or ids retained during fitting and detailed diagnostics but omitted from global macro and weighted aggregation.

Typed labels are translated through the dataset’s semantic label catalog before the OverlapIndex adapter is called. k mappings and exclude_classes may still use the original typed values; per-class diagnostics are restored to stable keys and readable, type-disambiguated report labels afterward.

Do not reconstruct the global score by averaging per-class scores: reporting-only excluded classes remain present there. OverlapScoreResult records both macro_score and weighted_score, plus the effective aggregation classes.

ContinuousOverlapScoringConfig

Explicit regression datasets use ContinuousOverlapIndex internally through the same OverlapIndexScorer adapter. Configure that path with ContinuousOverlapScoringConfig:

from vertebrae import ContinuousOverlapScoringConfig

config = ContinuousOverlapScoringConfig(
    k=8,
    target_cover="auto",
    target_distance="auto",
    n_null_permutations=20,
    aggregation="support_weighted",
)

Regression scoring keeps the MiniBatchKMeans backend fixed internally and reports OverlapScoreResult.score as the primary continuous overlap score. The score is always bounded to [0, 1]: 1.0 indicates no observed harmful continuous-target overlap, while 0.0 is the permutation-equivalent null endpoint. The unbounded loss comparison remains available through OverlapScoreResult.loss_ratio.

Automatic k resolution

The default k="auto" mode resolves a separate k for each class using class size, min_k, max_k, and min_samples_per_cluster. For multi-label targets, class size means per-label occurrence count. If a class or label is too small to support the requested or inferred k, the scorer reduces it and records a warning.

That warning is carried into the structured result and final reports so users can see when a weakly represented class forced a smaller prototype count.

Scoring inputs

The scorer accepts numeric embedding matrices with single-label, multi-label, or explicit regression targets.

  • Dense inputs are scored directly.

  • SciPy sparse matrices and sparse arrays are validated and normalized to CSR while retaining sparse storage through OverlapIndex.

  • Single-label targets are encoded as one-dimensional semantic keys before being passed to OverlapIndex, then translated back through the label catalog for reports.

  • Multi-label targets, including sparse binary indicators supplied by the user, are normalized to a CSR 0/1 indicator before calling OverlapIndex.

  • Regression targets are passed to ContinuousOverlapIndex as finite numeric 1D or 2D targets.

  • When normalize_embeddings=True, embeddings are L2-normalized row-wise before scoring.

This keeps extractor and caching layers free to preserve sparse artifacts longer, while still meeting the MiniBatchKMeans backend requirements.

Returned diagnostics

OverlapIndexScorer.score(...) returns an OverlapScoreResult with:

  • score

  • macro_score

  • per_class_scores

  • pairwise_scores

  • sparse_adjacency

  • class_counts

  • k_per_class

  • warnings

  • metadata

The metadata payload records backend details such as normalization, chunk size, seed, KMeans kwargs, and whether the original input was sparse. Regression results also preserve continuous null-calibration metadata plus prototype-level summaries, support, adjacency, and loss diagnostics.

Stability analysis

Repeated scoring is handled by run_stability_analysis(...), which uses OverlapIndexScorer under the hood. The default stability mode is prototype stability:

  • embeddings and labels stay fixed,

  • MiniBatchKMeans seeds change across repeats,

  • summaries report mean, standard deviation, min, max, and percentile interval.

Dense and scipy sparse embeddings are both preserved across repeats and subsample slices. Sparse matrices remain CSR through every repeated OverlapIndex call.

Subsample stability is also available when requested through StabilityConfig. It is target-aware by definition:

  • single-label targets are sampled per class;

  • multi-label targets are sampled to preserve every active label;

  • regression targets use target-preserving samples containing at least one non-constant target.

Categorical repeats retain at least two occurrences of every original class or active label, while regression repeats retain at least three rows. The requested fraction is applied with floor(count * subsample_fraction). If those minimums cannot be met, validation fails before any repeat is scored and recommends either a larger fraction or prototype mode; the subset is never silently expanded.

from vertebrae import OverlapScoringConfig, StabilityConfig

scoring = OverlapScoringConfig()
stability = StabilityConfig(mode="prototype", repeats=20, interval_level=0.95)
subsample_stability = StabilityConfig(
    mode="subsample",
    repeats=20,
    subsample_fraction=0.8,
    random_state=42,
)

Scoring seeds and sampling seeds use independent deterministic streams. Subsample results record sampling_seeds, effective_sample_counts, and effective_subsample_fractions so the sampling plan can be reproduced and audited. StabilityConfig.stratified has been removed: categorical subsampling cannot opt out of target preservation. Serialized configurations containing that field are intentionally unsupported.

vertebrae reports these as stability summaries and stability intervals. They are not formal confidence intervals unless a different statistical protocol is added explicitly in a future release.

Separatix diagnostics

Use SeparatixConfig to control the optional Separatix complexity diagnostic stage:

from vertebrae import SeparatixConfig

config = SeparatixConfig(
    enabled=True,
    overlap_threshold=0.80,
    random_state=42,
    densify_policy="warn_and_sample",
)

Current behavior:

  • Separatix runs on the same evaluated embedding variant that overlap scores.

  • It runs after compression and after the main overlap score is available.

  • By default it only runs when the overlap gate passes.

  • Classification and multi-label datasets use overlap_threshold.

  • Regression datasets use regression_overlap_threshold.

  • Multi-label targets are passed to Separatix as CSR 0/1 indicator matrices with target_mode="multilabel".

  • Regression targets are passed with target_mode="regression".

  • Optional Separatix MLP probes can be enabled with mlp_probes=True.

  • Linear/nonlinear probe comparisons use the declared metric direction: accuracy/F1 families favor larger values, while MAE/RMSE favor smaller values. Reported improvement and favored-family fields follow that same direction.

  • The full Separatix report is preserved in JSON outputs, while Markdown reports show a compact recommendation, confidence, decision path, key scores, probe evidence, evaluation context, and skips.

Separatix follows the same normalization convention as overlap scoring when normalize_embeddings=True. Sparse inputs remain sparse at the vertebrae boundary. densify_policy controls unavoidable dense-only diagnostics:

  • "warn_and_sample" (default) samples within the effective whole-MiB budget derived from max_dense_bytes and records warnings;

  • "skip" records the unavailable diagnostic without densifying;

  • "fail" raises when a required dense operation cannot fit.

SeparatixResult.preprocessing, densification_events, skipped_diagnostics, and warnings expose the sparse and memory audit directly; the complete upstream report remains available through SeparatixResult.report.

Separatix is the only downstream-complexity and probe-style diagnostic path in vertebrae. Ranking remains based on overlap scores; Separatix probe fields are reported only when the Separatix diagnostic runs and includes them.

SeparatixResult.probe_summary normalizes the existing probe evidence without fitting another model. It records the best probe, a target-appropriate primary metric when Separatix declares one, the complete reported metric map, comparable linear/nonlinear evidence, evaluation and sampling context, grouping counts, and probe-specific skip reasons. Single-label fallback summaries use Separatix’s balanced-accuracy baseline contract. Multi-label and regression summaries never invent an accuracy value or select an undeclared primary metric.

These fields are descriptive diagnostics. They do not participate in ranking, aggregate validity, or vertebrae’s benchmark recommendations. Recommendation confidence and probe-comparison confidence are reported separately because they describe different evidence.

Custom embedding metrics

Benchmark can score the same full embedding batch with one or more custom metrics. Every metric must return one finite aggregate score; that score is the only value eligible for ranking. Metrics may also return JSON-safe diagnostics, warnings, and metadata. Vertebrae preserves sparse embeddings at this boundary; a custom metric that densifies them is responsible for enforcing its own memory limit.

Classification labels and independence groups cross the metric boundary as marked semantic-key strings in both local and artifact-backed runs. This makes custom-metric behavior identical for typed classes such as integers, booleans, dates, UUIDs, and Decimals without importing user label classes on workers. Use the label_catalog in target_metadata for original typed provenance and display text; do not perform numeric arithmetic directly on categorical labels. Regression targets remain numeric.

Retrieval and matching

RetrievalBenchmark evaluates frozen query embeddings against a declared gallery and explicit relevance grades. It is a separate, exact ranking protocol: it does not fit a head, build an ANN index, mine negatives, or replace OverlapIndex in ordinary labeled benchmarks. Use it for semantic search, image-text matching, entity lookup, and other candidate-ranking use cases.

from vertebrae import RetrievalBenchmark, RetrievalConfig, RetrievalDataset, DatasetIdentity
from vertebrae.extractors import PrecomputedExtractor

dataset = RetrievalDataset.from_embeddings(
    query_embeddings,
    gallery_embeddings,
    relevance=[("query-1", "document-9", 2.0)],
    query_ids=["query-1"],
    gallery_ids=["document-9"],
    identity=DatasetIdentity.declared("example-dataset", "1"),
)
result = RetrievalBenchmark(
    dataset,
    [PrecomputedExtractor("candidate")],
    retrieval_config=RetrievalConfig(primary_metric="ndcg@10"),
).run()

Relevance may be a NumPy/scipy query-by-gallery matrix or sparse (query_id, gallery_id, grade) records. Any other iterable, including a nested Python list, is interpreted as records; use RetrievalDataset.from_relevance_matrix(...) for an explicit nested-list matrix. Sparse matrices are read from their nonzeros without densification, and every query must retain at least one eligible positive after exclusions. Grades above zero are relevant for binary metrics; NDCG uses the original finite nonnegative grades and stable log-domain gain ratios. Results include NDCG, precision, recall, hit rate, MRR, mAP, and positive-versus-negative similarity diagnostics. Cosine similarity is the default, with dot product and squared L2 available explicitly.

For a normal single-label Benchmark, LabelRetrievalMetric provides opt-in leave-one-out same-label retrieval after embedding compression. It is not inferred for multi-label or regression targets because those relevance semantics must be declared.

from vertebrae import Benchmark, CallableMetric, OverlapMetric, OverlapScoringConfig

def label_aware_metric(embeddings, labels, *, target_metadata=None, groups=None, seed=None):
    return {
        "score": 0.87,
        "diagnostics": {"criterion": "my benchmark rule"},
    }

benchmark = Benchmark(
    dataset,
    [extractor],
    metrics=[
        OverlapMetric(config=OverlapScoringConfig(k="auto")),
        CallableMetric("my_metric", label_aware_metric),
    ],
    primary_metric="my_metric",
)

OverlapIndex always runs and is available through ExtractorResult.overlap and ExtractorResult.metrics["overlap"]. Stability and Separatix continue to use this built-in overlap metric. For distributed or CLI scoring, use an importable callable path such as my_project.metrics:label_aware_metric with vertebrae score --metric ....

Practical guidance

  • Keep normalize_embeddings=True unless your embedding space already encodes a deliberate scale that you want to preserve.

  • Watch k_per_class and warnings when classes are imbalanced.

  • Use prototype stability first; it isolates clustering sensitivity without mixing in sampling noise.

  • Treat overlap scores as representation diagnostics for a specific dataset and protocol, not universal model-quality claims.