Package com.apple.foundationdb.async.common

Class StorageHelpers

java.lang.Object
com.apple.foundationdb.async.common.StorageHelpers
public final class StorageHelpers extends Object
Static helpers for the storage layer shared by the vector index implementations: (de)serializing vectors to and from their Tuple and byte representations, and appending to, consuming, aggregating, and clearing the set of sampled vectors held in a subspace. Not instantiable.

  • Method Details

    • consumeSampledVectors

      @Nonnull public static CompletableFuture<List<AggregatedVector>> consumeSampledVectors(@Nonnull Transaction transaction, @Nonnull Subspace prefixSubspace, int numMaxVectors, @Nonnull OnKeyValueReadListener onReadListener)
      Reads up to numMaxVectors sampled vectors from the given subspace and removes them as it goes, draining that portion of the sample buffer. Each consumed key is read on a snapshot, registered as a read conflict, cleared, and reported to the listener, so the transaction conflicts only on the keys actually consumed rather than on the whole range.
      Parameters:
      transaction - the transaction to read and clear within
      prefixSubspace - the subspace holding the sampled vectors
      numMaxVectors - the maximum number of sampled vectors to consume
      onReadListener - the listener notified of each key/value read
      Returns:
      a future completing with the consumed sampled vectors

    • aggregateVectors

      @Nullable public static AggregatedVector aggregateVectors(@Nonnull Iterable<AggregatedVector> vectors)
      Combines a collection of AggregatedVectors into a single aggregate by summing their partial vectors and counts.
      Parameters:
      vectors - the partial aggregates to combine
      Returns:
      the combined aggregate, or null if vectors contributes no elements

    • appendSampledVector

      public static void appendSampledVector(@Nonnull Transaction transaction, @Nonnull SplittableRandom random, boolean deterministicRandomness, @Nonnull Subspace prefixSubspace, int partialCount, @Nonnull Transformed<RealVector> vector, @Nonnull OnKeyValueWriteListener onWriteListener)
      Appends a single sampled vector to the sample buffer in the given subspace, under a key derived from its partial count and a fresh (optionally deterministic) random id.
      Parameters:
      transaction - the transaction to write within
      random - the random source for the key's id when deterministicRandomness is set
      deterministicRandomness - whether to derive the key's id deterministically rather than randomly
      prefixSubspace - the subspace holding the sampled vectors
      partialCount - the number of vectors this sample aggregates
      vector - the sampled (partial) vector to store
      onWriteListener - the listener notified of the written key/value

    • deleteAllSampledVectors

      public static void deleteAllSampledVectors(@Nonnull Transaction transaction, @Nonnull Subspace prefixSubspace, @Nonnull OnKeyValueWriteListener onWriteListener)
      Clears all sampled vectors held in the given subspace.
      Parameters:
      transaction - the transaction to clear within
      prefixSubspace - the subspace holding the sampled vectors
      onWriteListener - the listener notified of the cleared range

    • tupleFromVector

      @Nonnull public static Tuple tupleFromVector(@Nonnull Transformed<RealVector> vector)
      Converts a transformed vector into a tuple.
      Parameters:
      vector - a transformed vector
      Returns:
      a new, non-null Tuple instance representing the contents of the underlying vector.

    • tupleFromVector

      @Nonnull public static Tuple tupleFromVector(@Nonnull RealVector vector)
      Converts a RealVector into a Tuple.

      This method first serializes the given vector into a byte array using the RealVector.getRawData() getter method. It then creates a Tuple from the resulting byte array.

      Parameters:
      vector - the RealVector to convert. Cannot be null.
      Returns:
      a new, non-null Tuple instance representing the contents of the vector.

    • bytesFromVector

      @Nonnull public static byte[] bytesFromVector(@Nonnull Transformed<RealVector> transformedVector)
      Returns the raw byte representation of a transformed vector's underlying data, as stored in the database.
      Parameters:
      transformedVector - the transformed vector whose underlying raw bytes are returned
      Returns:
      the raw bytes of the underlying vector

    • vectorFromTuple

      @Nonnull public static RealVector vectorFromTuple(@Nonnull VectorEncodingConfig config, @Nonnull Tuple vectorTuple)
      Creates a RealVector from a given Tuple.

      This method assumes the vector data is stored as a byte array at the first. position (index 0) of the tuple. It extracts this byte array and then delegates to the vectorFromBytes(VectorEncodingConfig, byte[]) method for the actual conversion.

      Parameters:
      config - an HNSW configuration
      vectorTuple - the tuple containing the vector data as a byte array at index 0. Must not be null.
      Returns:
      a new RealVector instance created from the tuple's data. This method never returns null.

    • vectorFromBytes

      @Nonnull public static RealVector vectorFromBytes(@Nonnull VectorEncodingConfig config, @Nonnull byte[] vectorBytes)
      Creates a RealVector from a byte array.

      This method reads the leading vector-type byte and, based on it, delegates to the appropriate decoder: EncodedRealVector for RaBitQ-encoded vectors, otherwise RealVector.fromBytes(byte[]).

      Parameters:
      config - a vector-encoding configuration
      vectorBytes - the non-null byte array to convert.
      Returns:
      a new RealVector instance created from the byte array.