zero-shot semantic alignment

ZeroShotBenchmark evaluates a frozen text-aligned backbone by comparing each sample embedding with fixed text prompt prototypes. It is training-free: no classifier, prompt search, calibration, threshold fitting, or model head is learned.

This is a capability-specific protocol, not a universal representation-quality score. It answers whether the model’s shared embedding space can address this dataset’s declared classes with the supplied language. Every result also reports OverlapIndex on the same sample embeddings as separate contextual evidence; the scores are never combined.

Fixed prompts and class prototypes

Prompts are explicit protocol input. Use fully rendered strings when classes need domain-specific language, or explicitly provide templates:

from vertebrae import BenchmarkDataset, ZeroShotBenchmark, ZeroShotDataset, DatasetIdentity

dataset = BenchmarkDataset.from_arrays(images, labels, modality="image", identity=DatasetIdentity.declared("example-dataset", "1"))
protocol = ZeroShotDataset.from_templates(
    dataset,
    templates=("a photo of {label}", "a close-up of {label}"),
)
result = ZeroShotBenchmark(
    protocol,
    [clip_extractor],
    sample_branch="image_branch",
    text_branch="text_branch",
).run()

The source dataset must have single-label targets. Every observed class requires one or more non-empty prompts. Prompt order is preserved in the protocol fingerprint; duplicate prompts are rejected. Template strings are parsed with string.Formatter and may reference only the label field; positional, conversion, format-spec, and unknown fields are rejected. Template IDs must be nonempty and the protocol must have exact class-by-template coverage before scoring. For a prompt ensemble, vertebrae L2-normalizes each prompt embedding, averages all prompts for a class with equal weight, then normalizes the class prototype. It never chooses the best prompt after observing labels.

The protocol fingerprint hashes the complete ordered prompt declaration. Changing any prompt, template ID, label order, or protocol metadata produces a distinct prompt artifact and cache identity, including changes beyond a fixed list length.

Class labels retain their semantic values in the in-memory protocol. Strings and numbers work directly, as do UUIDs, enums, dates/times, Decimal, Fraction, and dataclass values. Arbitrary custom objects must be converted to a stable value first; vertebrae rejects them while constructing the protocol instead of failing later during artifact planning.

Artifact protocols use the versioned vertebrae.semantic-label/v1 encoding. Ordinary string labels remain readable, while non-string labels receive collision-resistant keys and an ordered catalog containing their exact typed identity and display value. This keeps values such as integer 1 and string "1", or a UUID and its textual form, distinct through distributed scoring and report reconstruction. User-defined enum and dataclass classes do not need to be importable on scoring workers.

The complete protocol recipe is strict JSON and includes sample-label order, typed sample IDs, prompts, template IDs, metadata, and the label catalog. Artifact scoring recomputes its fingerprint and checks duplicated manifest fields against the recipe; corrupt or stale protocol payloads are rejected before embeddings are scored. Missing or unknown label-encoding schemas are rejected.

sample_branch and text_branch are explicit because only extractors that declare independently encodable, shared-space branches can support zero-shot evaluation. OpenCLIP, SigLIP, compatible Hugging Face multimodal extractors, and custom CallableRetrievalExtractor adapters are appropriate. Ordinary pooled backbones are not implicitly treated as zero-shot-capable.

When candidates use different endpoint names, declare the pair beside each extractor instead of forcing a global convention:

from vertebrae import ZeroShotCandidate

result = ZeroShotBenchmark(
    protocol,
    [
        ZeroShotCandidate(openclip, "image_branch", "text_branch"),
        ZeroShotCandidate(custom_adapter, "query", "gallery"),
    ],
).run()

The global-branch form remains supported. OpenCLIP’s ordinary single-output default remains image-only, but its native text_branch is always independently available to retrieval and zero-shot evaluation.

Metrics and interpretation

The default rank is Top-1 accuracy; macro_f1 and balanced_accuracy are first-class configurable primary metrics. Reports always include all three, valid Top-K accuracy values, per-class precision/recall/F1, a confusion matrix, prompt coherence, correct-class margins, and deterministic tie warnings.

ZeroShotConfig.similarity="squared_l2" ranks prototypes by negative squared Euclidean distance, so larger scores correspond to closer prototypes just as larger cosine and dot-product scores correspond to better matches.

Cosine scoring requires nonzero sample rows and nonzero prompt/prototype rows. Sample zero vectors remain valid for dot-product and squared-L2 scoring; prompt groups must always be nonempty and produce a nonzero prototype.

zero-shot	overlap	interpretation
strong	strong	Text-addressable and well-structured frozen representation.
strong	weak	Native prompt alignment works, but do not infer broad transfer quality.
weak	strong	Promising transfer structure with weak text/prompt alignment.
weak	weak	Weak fit for this declared dataset and fixed protocol.

Poor zero-shot performance can reflect taxonomy wording, domain terminology, prompt coverage, or a weak text branch. It does not prove that sample embeddings are poor features for a separately trained downstream model.

Memory-bounded exact scoring

ZeroShotConfig.sample_batch_size controls how many samples are compared with the class prototypes at once; its default is 128. Scoring remains exact and preserves declared-class-order tie breaking, but it never needs to allocate the complete sample-by-class score matrix.

ZeroShotConfig.max_dense_bytes limits the estimated float64 scorer working set, including dense sample and prompt endpoints, prototypes, result accumulators, and one configured score block. If that block does not fit, lower sample_batch_size, compress the embeddings, or raise the explicit limit. Dense and sparse inputs use the same admission check before conversion or scoring.

ZeroShotBenchmark(memory_config=...) separately bounds local sample and prompt embedding materialization. Streaming batches spill to temporary local staging when allowed; otherwise the run fails as soon as the next batch would exceed the resident budget instead of accumulating the complete endpoint first.

Compression and artifact workflows

Compression variants are supported. Vertebrae fits the label-free compressor on sample embeddings and applies the same fitted transform to prompt embeddings before building prototypes. Sample embedding cache keys use the source dataset identity, so changing only prompts can reuse sample embeddings when the source dataset and extractor have a stable cache identity and raw caching is enabled. An unsafe or explicitly disabled raw cache also makes every paired compressed variant ineligible for reuse.

For CallableRetrievalExtractor, top-level importable callables are recorded in its recipe and cache key. Lambdas or nested callables must supply cache_identity to use the local cache or canonical artifact plans; otherwise local caching is skipped with a warning rather than guessing an unsafe function identity. Functions defined in a script’s __main__ module are likewise nonportable and need cache_identity.

Artifact-backed workflows mirror retrieval: use plan-zero-shot, embed-zero-shot-shard, merge-zero-shot-embeddings, and write-zero-shot-protocol; optionally apply compress-zero-shot; then call score-zero-shot with the merged sample key, prompt key, and protocol key. Each endpoint’s planned shard count is capped at its row count, so execute embed-zero-shot-shard --plan-json PLAN --side ... --shard-index ... from the plan rather than assuming samples and prompts have the same count. Prompt text, ordered labels, and template IDs are preserved in JSON and artifact metadata, while compact Markdown shows prompt counts only.

Use zero-shot-from-artifacts --score-key SCORE ... --json-output REPORT.json --markdown-output REPORT.md to reconstruct a ranked report from one or more score artifacts sharing the same protocol. The report retains zero-shot alignment and OverlapIndex as separate metrics.

Sample and prompt endpoints must be either both raw or produced by the same paired compression artifact. Vertebrae records the fitted compression-pair identity and rejects mixed endpoints, even when their dimensions match. Score artifact keys include the zero-shot and overlap configurations, so different similarity or ranking settings do not overwrite one another; reports combine only matching evaluation settings.

When compression is skipped because a requested dimension is not smaller than the embedding dimension, result names retain that requested dimension so variants remain distinct while metadata reports the actual output dimension.