vertebrae.datasets.base

Dataset abstraction for benchmark inputs.

Classes

TargetView

Declarative target view aligned to an existing dataset sample axis.

UnitAnnotation

Declarative per-parent unit targets and provenance.

BenchmarkDataset

A labeled dataset prepared for feature extraction or scoring.

EmbeddingUnitDataset

Generic embedding dataset for structured units such as boxes or tokens.

Module Contents

class vertebrae.datasets.base.TargetView[source]

Declarative target view aligned to an existing dataset sample axis.

class vertebrae.datasets.base.UnitAnnotation[source]

Declarative per-parent unit targets and provenance.

class vertebrae.datasets.base.BenchmarkDataset[source]

A labeled dataset prepared for feature extraction or scoring.

X[source]

Input samples, tabular frame, image paths, or embedding matrix.

y[source]

Single-label, multi-label, or explicit regression targets.

modality[source]

Dataset modality such as “text”, “tabular”, or “embeddings”.

input_col[source]

Source dataframe input column or columns.

label_col[source]

Source dataframe label column.

metadata[source]

User and construction metadata.

classmethod from_arrays(X, y, modality, *, identity, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create a dataset from array-like inputs and labels.

Parameters:

  • X (Any) – Input samples or feature matrix.

  • y (Any) – Class labels.

  • modality (str) – Dataset modality.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated benchmark dataset.

Return type:

BenchmarkDataset

classmethod from_dataframe(df, input_col, label_col, modality, *, identity, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create a dataset from a pandas DataFrame.

Parameters:

  • df (Any) – DataFrame containing inputs and labels.

  • input_col (Union[str, list[str]]) – Input column name or list of tabular feature columns.

  • label_col (Union[str, list[str]]) – Label column name.

  • modality (str) – Dataset modality.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated benchmark dataset.

Raises:

ValueError – If requested columns are missing.

Return type:

BenchmarkDataset

classmethod from_image_paths(paths, labels, *, identity, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create an image dataset from filesystem paths.

Parameters:

  • paths (Any) – Image file paths.

  • labels (Any) – Class labels.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated image dataset.

Return type:

BenchmarkDataset

classmethod from_audio_paths(paths, labels, *, identity, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create an audio dataset from filesystem paths.

Parameters:

  • paths (Any) – Audio file paths.

  • labels (Any) – Class labels.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated audio dataset.

Return type:

BenchmarkDataset

classmethod from_audio_arrays(audio, labels, sampling_rate, *, identity, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create an audio dataset from waveform arrays.

Parameters:

  • audio (Any) – Sequence of 1D or 2D waveform arrays.

  • labels (Any) – Class labels.

  • sampling_rate (int) – Shared sampling rate for every sample.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated audio dataset.

Return type:

BenchmarkDataset

classmethod from_video_paths(paths, labels, *, identity, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create a video dataset from filesystem paths.

Parameters:

  • paths (Any) – Video file paths.

  • labels (Any) – Class labels.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated video dataset.

Return type:

BenchmarkDataset

classmethod from_video_arrays(frames, labels, *, identity, frame_rate=None, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create a video dataset from predecoded frame arrays.

Parameters:

  • frames (Any) – Sequence of per-sample clip arrays, typically (time, height, width, channels).

  • labels (Any) – Class labels.

  • frame_rate (Optional[Any]) – Optional shared frame rate or sequence aligned to frames.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated video dataset.

Return type:

BenchmarkDataset

classmethod from_time_series(series, labels, *, identity, observed_mask=None, time_features=None, timestamps=None, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create a time-series dataset from aligned sequence inputs.

Parameters:

  • series (Any) – Sequence array with shape (n_samples, time) or (n_samples, time, channels).

  • labels (Any) – Class labels.

  • observed_mask (Any) – Optional boolean or numeric mask aligned to series.

  • time_features (Any) – Optional numeric time features aligned to series.

  • timestamps (Any) – Optional timestamp annotations preserved for reporting.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated time-series dataset.

Return type:

BenchmarkDataset

classmethod from_multimodal(inputs, labels, modalities, *, identity, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create a dataset from aligned multi-modal sample fields.

Parameters:

  • inputs (Dict[str, Any]) – Mapping from field name to aligned per-sample values.

  • labels (Any) – Class labels.

  • modalities (Dict[str, str]) – Mapping from field name to modality string.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated multi-modal dataset.

Return type:

BenchmarkDataset

classmethod from_graphs(graphs, labels, *, identity, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create a graph dataset from aligned graph objects.

Parameters:
Return type:

BenchmarkDataset

classmethod from_embeddings(embeddings, labels, *, identity, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create a dataset from precomputed dense or sparse embeddings.

Parameters:

  • embeddings (Any) – Dense array or scipy sparse embedding matrix.

  • labels (Any) – Class labels.

  • metadata (Optional[Dict[str, Any]]) – Optional metadata to preserve.

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Returns:

Validated embedding dataset.

Return type:

BenchmarkDataset

classmethod from_segmentation_embeddings(embeddings, labels, image_ids, *, identity, metadata=None)[source]

Create a grouped token dataset from precomputed segmentation features.

Parameters:
Return type:

BenchmarkDataset

classmethod from_embedding_units(embeddings, labels, unit_ids, *, identity, parent_ids=None, unit_type='unit', positions=None, spans=None, coordinates=None, provenance=None, metadata=None, label_names=None, target_type='auto', target_names=None, target_views=None)[source]

Create a generic grouped unit dataset from precomputed embeddings.

Parameters:

  • embeddings (Any)

  • labels (Any)

  • unit_ids (Any)

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • parent_ids (Any)

  • unit_type (str)

  • positions (Any)

  • spans (Any)

  • coordinates (Any)

  • provenance (Any)

  • metadata (Optional[Dict[str, Any]])

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

  • target_views (Optional[Iterable[TargetView]])

Return type:

EmbeddingUnitDataset

classmethod from_node_embeddings(embeddings, labels, *, identity, node_ids=None, edge_index=None, metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create an embedding dataset for labeled graph nodes.

This is an embedding-efficacy view of a graph: every row is one node embedding and labels or regression targets remain aligned row-wise.

Parameters:

  • embeddings (Any)

  • labels (Any)

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • node_ids (Any)

  • edge_index (Any)

  • metadata (Optional[Dict[str, Any]])

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Return type:

BenchmarkDataset

classmethod from_entity_embeddings(embeddings, labels, *, identity, entity_ids=None, entity_type='entity', metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create an embedding dataset for labeled entities.

Use this for user, item, query, document, or other entity embeddings when the evaluation target is attached to the entity row itself.

Parameters:

  • embeddings (Any)

  • labels (Any)

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • entity_ids (Any)

  • entity_type (str)

  • metadata (Optional[Dict[str, Any]])

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Return type:

BenchmarkDataset

classmethod from_edge_embeddings(*, identity, edge_embeddings=None, labels=None, edge_index=None, node_embeddings=None, node_ids=None, composition='hadamard', metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create an embedding dataset for labeled graph edges.

Pass either precomputed edge_embeddings, or pass node_embeddings plus edge_index to compose one embedding row per edge.

Parameters:

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • edge_embeddings (Any)

  • labels (Any)

  • edge_index (Any)

  • node_embeddings (Any)

  • node_ids (Any)

  • composition (str)

  • metadata (Optional[Dict[str, Any]])

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Return type:

BenchmarkDataset

classmethod from_pair_embeddings(*, identity, pair_embeddings=None, labels=None, pairs=None, entity_embeddings=None, entity_ids=None, composition='abs_diff', metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create an embedding dataset for labeled entity pairs.

Parameters:

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • pair_embeddings (Any)

  • labels (Any)

  • pairs (Any)

  • entity_embeddings (Any)

  • entity_ids (Any)

  • composition (str)

  • metadata (Optional[Dict[str, Any]])

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Return type:

BenchmarkDataset

classmethod from_triplet_embeddings(*, identity, triplet_embeddings=None, labels=None, triplets=None, entity_embeddings=None, entity_ids=None, composition='abs_diff', metadata=None, label_names=None, target_type='auto', target_names=None)[source]

Create an embedding dataset for supervised triplet-derived rows.

Parameters:

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • triplet_embeddings (Any)

  • labels (Any)

  • triplets (Any)

  • entity_embeddings (Any)

  • entity_ids (Any)

  • composition (str)

  • metadata (Optional[Dict[str, Any]])

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Return type:

BenchmarkDataset

validate()[source]

Validate dataset shape and labels.

Raises:

ValueError – If sample counts mismatch, labels are missing, too few classes are present, or a class has fewer than two samples.

Return type:

None

class_counts()[source]

Count samples per class.

Returns:

Mapping from original label values to sample counts.

Return type:

Dict[Any, int]

with_label_hierarchy(label_paths, level_names=None)[source]

Return a dataset annotated with hierarchical label paths.

Parameters:

  • label_paths (Any)

  • level_names (Optional[Iterable[Any]])

Return type:

BenchmarkDataset

label_view(level, name=None)[source]

Project hierarchical labels to a single requested level.

Parameters:

  • level (Any)

  • name (Optional[str])

Return type:

BenchmarkDataset

active_label_view()[source]

Return metadata describing the active dataset label view.

Return type:

Dict[str, Any]

with_target_views(target_views)[source]

Return a dataset annotated with aligned named target views.

Parameters:

target_views (Iterable[TargetView])

Return type:

BenchmarkDataset

target_view(name)[source]

Materialize one named target view as an ordinary benchmark dataset.

Parameters:

name (str)

Return type:

BenchmarkDataset

target_view_names()[source]

Return the registered target view names in insertion order.

Return type:

list[str]

active_target_view()[source]

Return metadata describing the active dataset target view.

Return type:

Dict[str, Any]

with_unit_annotations(annotations, unit_type='unit', task_family=None)[source]

Return a dataset with aligned per-parent structured unit annotations.

Parameters:

  • annotations (Iterable[UnitAnnotation])

  • unit_type (str)

  • task_family (Optional[str])

Return type:

BenchmarkDataset

unit_annotations()[source]

Return aligned structured unit annotations when configured.

Return type:

Optional[list[Dict[str, Any]]]

with_groups(groups, name='group')[source]

Return a dataset with aligned independence-group identifiers.

Parameters:

  • groups (Any)

  • name (str)

Return type:

BenchmarkDataset

groups()[source]

Return aligned independence groups when configured.

Return type:

Optional[numpy.ndarray]

iter_batches(batch_size, shard=None)[source]

Yield deterministic sample batches for embedding.

Parameters:
Yields:

Sample batches with original dataset indices and sliced inputs.

Raises:

ValueError – If batch_size is less than one.

Return type:

Iterator[vertebrae.execution.jobs.SampleBatch]

stratified_subsample_indices(rate, random_state=42, min_samples_per_class=2)[source]

Select target-aware sample indices without replacement.

Parameters:

  • rate (float) – Requested fraction of samples to keep. Must be in (0, 1].

  • random_state (int) – Random seed for reproducible selection.

  • min_samples_per_class (int) – Minimum retained samples per class when possible.

Returns:

Sorted original sample indices for the stratified subset.

Raises:

ValueError – If rate is outside (0, 1].

Return type:

numpy.ndarray

subset(indices, metadata=None)[source]

Create a dataset subset by original sample indices.

Parameters:

  • indices (Any) – Sample indices to retain.

  • metadata (Optional[Dict[str, Any]]) – Additional metadata to merge into the subset.

Returns:

Validated dataset containing only the selected samples.

Return type:

BenchmarkDataset

summary()[source]

Summarize the dataset for result metadata and reports.

Returns:

JSON-compatible dataset summary.

Return type:

Dict[str, Any]

identity_key()[source]

Resolve and memoize the dataset’s explicit identity key.

Treat the dataset and its identity-bearing values as immutable after this method is called.

Return type:

str

class vertebrae.datasets.base.EmbeddingUnitDataset[source]

Bases: BenchmarkDataset

Generic embedding dataset for structured units such as boxes or tokens.

classmethod from_units(embeddings, labels, unit_ids, *, identity, parent_ids=None, unit_type='unit', positions=None, spans=None, coordinates=None, provenance=None, metadata=None, label_names=None, target_type='auto', target_names=None, target_views=None)[source]

Create a structured unit dataset from aligned embedding rows.

Parameters:

  • embeddings (Any)

  • labels (Any)

  • unit_ids (Any)

  • identity (vertebrae.datasets.identity.DatasetIdentity)

  • parent_ids (Any)

  • unit_type (str)

  • positions (Any)

  • spans (Any)

  • coordinates (Any)

  • provenance (Any)

  • metadata (Optional[Dict[str, Any]])

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

  • target_views (Optional[Iterable[TargetView]])

Return type:

EmbeddingUnitDataset