vertebrae.cache.s3_store

S3-backed artifact store.

Classes

S3ArtifactStore

Store artifacts in S3-compatible object storage.

Module Contents

class vertebrae.cache.s3_store.S3ArtifactStore(bucket, prefix='', endpoint_url=None, profile_name=None, region_name=None)[source]

Store artifacts in S3-compatible object storage.

Parameters:

  • bucket (str) – S3 bucket name.

  • prefix (str) – Optional object key prefix.

  • endpoint_url (Optional[str]) – Optional S3-compatible endpoint URL.

  • profile_name (Optional[str]) – Optional boto3 profile name.

  • region_name (Optional[str]) – Optional AWS region name.

classmethod from_uri(uri, **options)[source]

Build an S3 store from a s3://bucket/prefix URI.

Parameters:

  • uri (str)

  • options (Any)

Return type:

S3ArtifactStore

config()[source]

Return a serializable config for reconstructing this store.

Return type:

vertebrae.cache.artifact_store.ArtifactStoreConfig

exists(key)[source]

Return whether an embedding artifact exists for key.

Parameters:

key (str)

Return type:

bool

put_array(key, arr)[source]

Store a dense or sparse embedding matrix.

Parameters:

  • key (str)

  • arr (Any)

Return type:

str

put_array_batches(key, batches, n_samples, require_complete=True)[source]

Store embeddings from deterministic batches.

Parameters:

  • key (str)

  • batches (Iterable[Tuple[numpy.ndarray, Any]])

  • n_samples (int)

  • require_complete (bool)

Return type:

str

put_artifact(key, arr, metadata, *, metadata_finalizer=None)[source]

Commit an array and immutable metadata with a last-written manifest.

Parameters:
Return type:

str

put_artifact_batches(key, batches, n_samples, metadata, require_complete=True, *, metadata_finalizer=None)[source]

Commit batched arrays and immutable metadata as one generation.

Parameters:
Return type:

str

get_artifact(key)[source]

Load a validated array/metadata pair, retrying one manifest switch.

Parameters:

key (str)

Return type:

tuple[Any, dict]

get_array(key)[source]

Load a dense or sparse embedding matrix.

Parameters:

key (str)

Return type:

Any

stat_array(key)[source]

Return object size using S3 metadata without downloading it.

Parameters:

key (str)

Return type:

vertebrae.cache.artifact_store.ArtifactStat

put_labels(key, labels, *, label_names=None, target_type='auto', target_names=None)[source]

Store labels as JSON.

Parameters:

  • key (str)

  • labels (Any)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Return type:

str

put_labels_artifact(key, labels, metadata, *, label_names=None, target_type='auto', target_names=None)[source]

Commit labels and decoding metadata under one manifest.

Parameters:

  • key (str)

  • labels (Any)

  • metadata (dict)

  • label_names (Optional[Iterable[Any]])

  • target_type (str)

  • target_names (Optional[Iterable[str]])

Return type:

str

get_labels_artifact(key)[source]

Load one validated labels/metadata generation with switch retries.

Parameters:

key (str)

Return type:

tuple[numpy.ndarray, dict]

get_labels(key)[source]

Load labels from JSON.

Parameters:

key (str)

Return type:

numpy.ndarray

put_json(key, obj)[source]

Store JSON metadata for an artifact key.

Parameters:

  • key (str)

  • obj (dict)

Return type:

str

get_json(key)[source]

Load JSON metadata for an artifact key.

Parameters:

key (str)

Return type:

dict

delete_prefix(prefix)[source]

Delete every S3 object beneath an artifact key prefix.

Parameters:

prefix (str)

Return type:

None