vertebrae.cache.gcs_store

GCS-backed artifact store.

Classes

GCSArtifactStore

Store artifacts in Google Cloud Storage.

Module Contents

class vertebrae.cache.gcs_store.GCSArtifactStore(bucket, prefix='', project=None, emulator_host=None)

Store artifacts in Google Cloud Storage.

Parameters:

• bucket (str) – GCS bucket name.

• prefix (str) – Optional object key prefix.

• project (Optional[str]) – Optional GCP project override.

• emulator_host (Optional[str])

classmethod from_uri(uri, **options)

Build a GCS store from a gs://bucket/prefix URI.

Parameters:

• uri (str)

• options (Any)

Return type:

GCSArtifactStore

config()

Return a serializable config for reconstructing this store.

Return type:

vertebrae.cache.artifact_store.ArtifactStoreConfig

exists(key)

Return whether an embedding artifact exists for key.

Parameters:

key (str)

Return type:

bool

put_array(key, arr)

Store a dense or sparse embedding matrix.

Parameters:

• key (str)

• arr (Any)

Return type:

str

put_array_batches(key, batches, n_samples, require_complete=True)

Store embeddings from deterministic batches.

Parameters:

• key (str)

• batches (Iterable[Tuple[numpy.ndarray, Any]])

• n_samples (int)

• require_complete (bool)

Return type:

str

put_artifact(key, arr, metadata, *, metadata_finalizer=None)

Commit an array and immutable metadata with a last-written manifest.

Parameters:

• key (str)

• arr (Any)

• metadata (dict)

• metadata_finalizer (Optional[Callable[[dict, vertebrae.cache.artifact_store.ArrayArtifactManifest, vertebrae.cache.artifact_store.ArtifactStat], dict]])

Return type:

str

put_artifact_batches(key, batches, n_samples, metadata, require_complete=True, *, metadata_finalizer=None)

Commit batched arrays and immutable metadata as one generation.

Parameters:

• key (str)

• batches (Iterable[Tuple[numpy.ndarray, Any]])

• n_samples (int)

• metadata (dict)

• require_complete (bool)

• metadata_finalizer (Optional[Callable[[dict, vertebrae.cache.artifact_store.ArrayArtifactManifest, vertebrae.cache.artifact_store.ArtifactStat], dict]])

Return type:

str

get_artifact(key)

Load a validated array/metadata pair, retrying one manifest switch.

Parameters:

key (str)

Return type:

tuple[Any, dict]

get_array(key)

Load a dense or sparse embedding matrix.

Parameters:

key (str)

Return type:

Any

stat_array(key)

Return blob size using GCS metadata without downloading it.

Parameters:

key (str)

Return type:

vertebrae.cache.artifact_store.ArtifactStat

put_labels(key, labels, *, label_names=None, target_type='auto', target_names=None)

Store labels as JSON.

Parameters:

• key (str)

• labels (Any)

• label_names (Optional[Iterable[Any]])

• target_type (str)

• target_names (Optional[Iterable[str]])

Return type:

str

put_labels_artifact(key, labels, metadata, *, label_names=None, target_type='auto', target_names=None)

Commit labels and decoding metadata under one manifest.

Parameters:

• key (str)

• labels (Any)

• metadata (dict)

• label_names (Optional[Iterable[Any]])

• target_type (str)

• target_names (Optional[Iterable[str]])

Return type:

str

get_labels_artifact(key)

Load one validated labels/metadata generation with switch retries.

Parameters:

key (str)

Return type:

tuple[numpy.ndarray, dict]

get_labels(key)

Load labels from JSON.

Parameters:

key (str)

Return type:

numpy.ndarray

put_json(key, obj)

Store JSON metadata for an artifact key.

Parameters:

• key (str)

• obj (dict)

Return type:

str

get_json(key)

Load JSON metadata for an artifact key.

Parameters:

key (str)

Return type:

dict

delete_prefix(prefix)

Delete every GCS blob beneath an artifact key prefix.

Parameters:

prefix (str)

Return type:

None