vertebrae.extractors.huggingface_multimodal

Optional Hugging Face multi-modal embedding extractor.

Classes

HFMultimodalExtractor

Hugging Face multi-modal backbone extractor with named outputs.

Module Contents

class vertebrae.extractors.huggingface_multimodal.HFMultimodalExtractor(name, model_id, input_modalities, outputs, processor_id=None, input_map=None, input_fn=None, output_fn=None, structured_outputs=None, batch_size=16, image_mode='auto', alpha_mode='drop', device=None, revision=None, trust_remote_code=False, processor_kwargs=None, model_kwargs=None, checkpoint_paths=None, cache_identity=None)[source]

Hugging Face multi-modal backbone extractor with named outputs.

Parameters:

  • name (str)

  • model_id (str)

  • input_modalities (Dict[str, str])

  • outputs (List[Dict[str, Any]])

  • processor_id (Optional[str])

  • input_map (Optional[Dict[str, str]])

  • input_fn (Optional[Callable[[Any], Dict[str, Any]]])

  • output_fn (Optional[Callable[[Any], Any]])

  • structured_outputs (Optional[List[Dict[str, Any]]])

  • batch_size (int)

  • image_mode (str)

  • alpha_mode (str)

  • device (Optional[str])

  • revision (Optional[str])

  • trust_remote_code (bool)

  • processor_kwargs (Optional[Dict[str, Any]])

  • model_kwargs (Optional[Dict[str, Any]])

  • checkpoint_paths (Optional[Iterable[str]])

  • cache_identity (Optional[str])

encode_retrieval(X, *, branch, modality)[source]

Encode one independent branch when the wrapped model exposes it explicitly.

Parameters:

  • X (Any)

  • branch (str)

  • modality (str)

Return type:

numpy.ndarray