Package com.apple.foundationdb.async.guardiann

Class Guardiann

java.lang.Object
com.apple.foundationdb.async.guardiann.Guardiann
@API(EXPERIMENTAL) public class Guardiann extends Object
A transactional approximate nearest neighbor (ANN) vector structure built on top of FoundationDB. Guardiann organizes vectors into clusters, each represented by a centroid stored in an HNSW graph for fast centroid lookup, and maintains per-cluster metadata (vector counts, running statistics, replication state) to support incremental maintenance.

Architecture

Vectors are partitioned into clusters. Each cluster stores a set of primary vector references (the authoritative copies) and replicated references (copies from neighboring clusters kept for improved search recall). Cluster centroids are indexed in an HNSW graph so that the nearest clusters to a query vector can be found efficiently.

Operations

Maintenance

Structural maintenance is performed lazily via deferred tasks that are executed piggy-backed on insert and delete operations:

  • Split/Merge — when a cluster grows beyond Config.primaryClusterMax() or shrinks below Config.primaryClusterMin(), a SplitMergeTask repartitions the affected clusters using bounded k-means and updates the HNSW centroid index.
  • Reassign — a ReassignTask recomputes vector assignments and replication for a cluster, fixing underreplicated vectors and cleaning up excess replicas.
  • Collapse — a CollapseTask detects large groups of identical vectors and replaces them with a single collapsed representative to reduce storage.
  • Bounce — a BounceTask coordinates execution order between dependent tasks, ensuring prerequisite tasks complete before follow-up tasks are created.

Usage

This class is the primary entry point for interacting with the Guardiann vector structure. It delegates to specialized operation classes (Insert, Search, Delete) which implement the respective algorithms. All operations are asynchronous and designed for use within FoundationDB transactions.

See Also:

  • Constructor Details

    • Guardiann

      public Guardiann(@Nonnull Subspace subspace, @Nonnull Executor executor, @Nonnull Config config, @Nonnull OnWriteListener onWriteListener, @Nonnull OnReadListener onReadListener)
      Constructs a new Guardiann vector structure instance.

      Initializes the structure with the necessary components for storage, execution, configuration, and event handling. All parameters are mandatory and must not be null.

      Parameters:
      subspace - the Subspace where the vector structure data is stored
      executor - the Executor service to use for concurrent operations
      config - the Config object containing algorithm and storage parameters
      onWriteListener - a listener to be notified of write events
      onReadListener - a listener to be notified of read events

  • Method Details

    • newConfigBuilder

      public static Config.ConfigBuilder newConfigBuilder()
      Start building a Config.
      Returns:
      a new Config that can be altered and then built for use with a Guardiann
      See Also:

    • defaultConfig

      @Nonnull public static Config defaultConfig(int numDimensions)
      Returns a default Config.
      Parameters:
      numDimensions - number of dimensions
      Returns:
      a new default Config.
      See Also:

    • getLocator

      @Nonnull public Locator getLocator()

    • getSubspace

      @Nonnull public Subspace getSubspace()
      Gets the subspace associated with this object.
      Returns:
      the non-null subspace

    • getExecutor

      @Nonnull public Executor getExecutor()
      Get the executor used by this object.
      Returns:
      executor used when running asynchronous tasks

    • getConfig

      @Nonnull public Config getConfig()
      Get this object's configuration.
      Returns:
      the configuration

    • getOnWriteListener

      @Nonnull public OnWriteListener getOnWriteListener()
      Get the on-write listener.
      Returns:
      the on-write listener

    • getOnReadListener

      @Nonnull public OnReadListener getOnReadListener()
      Get the on-read listener.
      Returns:
      the on-read listener

    • kNearestNeighborsSearch

      @Nonnull public CompletableFuture<List<? extends ResultEntry>> kNearestNeighborsSearch(@Nonnull ReadTransaction readTransaction, int k, @Nonnull SearchConfig searchConfig, boolean includeVectors, @Nonnull RealVector queryVector)
      Performs a search for the k-nearest neighbors for a given query vector.
      Parameters:
      readTransaction - the transaction to use for reading from the database
      k - the number of nearest neighbors to return
      searchConfig - the performance/recall tuning knobs for this search (see SearchConfig)
      includeVectors - indicator if the caller would like the search to also include vectors in the result set
      queryVector - the vector to find the nearest neighbors of
      Returns:
      a CompletableFuture that will complete with a list of the k nearest neighbors, sorted by distance in ascending order.

    • insert

      @Nonnull public CompletableFuture<Void> insert(@Nonnull Transaction transaction, @Nonnull Tuple newPrimaryKey, @Nonnull RealVector newVector, @Nullable Tuple additionalValues, boolean maintainInTransaction)
      Inserts a new vector with its associated primary key into the vector structure.

      Finds the nearest cluster(s) for the new vector by querying the HNSW centroid index, then writes a primary vector reference to the nearest cluster and replicated references to neighboring clusters based on replication priority scoring. Updates cluster metadata and enqueues deferred maintenance tasks (split, merge, reassign) if cluster size invariants are violated.

      Parameters:
      transaction - the Transaction context for all database operations
      newPrimaryKey - the unique Tuple primary key for the new vector being inserted
      newVector - the RealVector data to be inserted
      additionalValues - additional values to be associated with the new vector/record, or null
      maintainInTransaction - when true, drain one deferred maintenance task inside this writing transaction (and skip hard-cap back-pressure); when false tasks accumulate for a background merge
      Returns:
      a CompletableFuture that completes when the insertion operation is finished

    • delete

      @Nonnull public CompletableFuture<Void> delete(@Nonnull Transaction transaction, @Nonnull Tuple primaryKey, @Nonnull RealVector vector, boolean maintainInTransaction)
      Deletes a vector identified by its primary key from the Guardiann index.

      Removes the vector's references from all clusters it belongs to (primary and replicated), updates cluster metadata, and enqueues maintenance tasks if needed. The caller must provide the vector data to locate which clusters contain the vector.

      Parameters:
      transaction - the Transaction context for all database operations
      primaryKey - the unique Tuple primary key of the vector to delete
      vector - the RealVector data of the vector being deleted
      maintainInTransaction - when true, drain one deferred maintenance task inside this writing transaction; when false tasks accumulate for a background merge
      Returns:
      a CompletableFuture that completes when the deletion is finished

    • executeDeferredTasks

      @Nonnull public CompletableFuture<Integer> executeDeferredTasks(@Nonnull Transaction transaction, int numTasks, long deadlineMillis)
      Drains up to numTasks of this structure's queued deferred maintenance tasks, running them inline within transaction. Stops draining before the next task once System.currentTimeMillis() reaches deadlineMillis (at least one task still runs when the queue is non-empty). Lets a background merge bound how long it drains within a single transaction by wall-clock time rather than only by task count. Pass Long.MAX_VALUE for no time bound.
      Parameters:
      transaction - the Transaction to fetch and run the tasks within
      numTasks - the maximum number of queued tasks to run
      deadlineMillis - an absolute wall-clock deadline (epoch millis) after which no further task is started
      Returns:
      a CompletableFuture of the number of tasks actually executed (0 if the structure holds none, or has never been initialized)