Building the Record Layer¶

This page explains how to build the FoundationDB Record Layer and set up a development environment for working on it. It covers building from source, configuring your IDE for development, and running the tests against a local cluster, as well as some advanced workflows. If you intend to contribute to the FoundationDB Record Layer, please also consult our contribution guidelines. If anything in this guide does not work smoothly for you, feel free to create a new issue or open a thread in the FoundationDB Forums.

Building the project¶

To build the Record Layer you only need an installed Java Development Kit as a prerequisite. No separate Gradle installation is needed, as the bundled Gradle wrapper supplies the correct version. A local FoundationDB cluster is not strictly needed for building, but you will need one if you intend to run the tests.

Install a current JDK distribution if you don’t already have one. Our CI/CD builds use Eclipse Temurin, but any distribution should work.
Optionally, create your own fork of the FoundationDB/fdb-record-layer repository. This is only needed if you intend to contribute. All changes are made by creating pull requests from your fork into the main repository; see our contribution guidelines.
Clone the repository (or your own fork of it) to your development machine.
Make sure that Gradle detects your JDK and that it meets the requirements. You can check which JDKs Gradle has found on your machine with:
```
./gradlew javaToolchains
```
If your JDK is unsuitable, the build will fail with an error explaining that no matching toolchain could be found. See Gradle’s Java toolchain support for more information on this process.
Kick off a build. The following command compiles every module and runs the static-analysis checks, SpotBugs and Checkstyle. Because the -x test flag excludes the test tasks, this does not require a running FoundationDB cluster. Expect the command to take several minutes on a first run.
```
./gradlew build -x test
```

Our automated builds are run via GitHub Actions. You can study the workflow definitions under the .github/workflows/ directory to learn how they invoke Gradle precisely.

Configuring IntelliJ IDEA¶

IntelliJ IDEA is the recommended environment for developing the FoundationDB Record Layer, although it is not strictly required. Note that we do not use Gradle’s IDEA plugin, whose IDE file generation is deprecated. Instead, we rely on IntelliJ’s built-in Gradle integration to generate and compile the project and to run the tests. When you open the project, IntelliJ performs a Gradle sync to derive a model of the project.

Install IntelliJ IDEA and launch it. Any recent version should work, though the menu paths mentioned below may differ slightly between releases.
In the “Welcome to IntelliJ IDEA” window, click Open and open the project’s root directory. If you do not see the welcome window, File › Open… can do the same.
Click the Reload All Gradle Projects button on the toolbar of the Gradle tool window. You can open this tool window from View › Tool Windows › Gradle.
Compile everything via Build › Build Project.

We recommend setting the following IntelliJ preferences. They are not strictly required, but useful.

Under Editor › General › Auto Import, enable “Optimize imports on the fly” for the current project.
Under Advanced Settings › Gradle, enable “Download sources” so that dependency sources are available for navigation and debugging.

If you make changes to other IntelliJ settings, they may save over the .xml files that are checked into Git under the .idea/ directory. These changes are meant to be shared by simply committing the updated files. However, other settings—such as the recommendations above—are saved in the user-specific workspace.xml file or other .xml files that we intentionally added to .gitignore. Such user-specific settings must be configured manually by each developer for consistency.

Running the tests¶

If you are interested in running the tests, you need to install and run a local FoundationDB cluster.

Install FoundationDB on your development machine. To find the latest published binaries, refer to the release list of the foundationdb GitHub repository.
Make sure FoundationDB is running. If you’ve installed an official package, the fdbserver process will start automatically. You can confirm that the cluster is available by running:
```
fdbcli --exec 'status minimal'
```
This should print “The database is available”. For platform-specific installation and startup instructions, see FoundationDB’s getting-started guides for macOS and Linux.
Run a test to verify that you have a working configuration. In IntelliJ, pick a test that depends on FoundationDB and hit the green “play” arrow in the gutter next to it—FDBRecordStoreCrudTest.writeRead() is a good, fast one. The first run may take a while (perhaps a minute) while the project builds and the test harness warms up; subsequent runs are faster. You can also run tests from the command line, as described below.
The full suite of checks and tests is run by the check and test Gradle tasks, as follows. In addition to the tests, this includes the static analysis checks, SpotBugs and Checkstyle. The -PspotbugsEnableHtmlReport flag additionally emits a SpotBugs report in HTML format.
```
./gradlew -PspotbugsEnableHtmlReport check test
```

By default, the tests run against the FoundationDB cluster named by the standard cluster file, which the official installer writes to the platform default location (/etc/foundationdb/fdb.cluster on Linux, /usr/local/etc/foundationdb/fdb.cluster on macOS), or wherever FDB_CLUSTER_FILE points if that variable is set. To override this behavior, you can create an fdb-environment.yaml file in the root of your working directory. It lets you point the tests at a non-default cluster, or even run them against multiple clusters. The file specifies the directory containing the FoundationDB C API library (libfdb_c.dylib on macOS, libfdb_c.so on Linux) and a list of cluster files. If you provide more than one, most tests will choose randomly among them. Here is an example:

libraryPath: /opt/fdb/lib
clusterFiles:
  - /opt/fdb/clusters/fdb-one.cluster
  - /opt/fdb/clusters/fdb-two.cluster

Gradle test tasks¶

The build defines several test tasks for different purposes. You can run any of them by passing the task name to the wrapper. For example:

./gradlew performanceTest

Here is an overview of the available tasks:

test

Runs the majority of our tests. This is the go-to task and the idiomatic way of running tests in Gradle.

destructiveTest

Runs the tests annotated with @Tag(Tags.WipesFDB), and only those. These tests work with global state in FoundationDB and need to wipe the entire database while running (or in @Before* and @After* methods), which is why they are segregated into their own task.

performanceTest

Runs the tests annotated with @Tag(Tags.Performance). Such tests gather performance numbers rather than check correctness. We don’t run these on any regular cadence or automatically, but once written they’re valuable to keep around and re-run whenever you touch the associated production code.

:fdb-extensions:scalarFallbackTest

This task exists only in the fdb-extensions module. It re-runs the vector math tests in that module with the scalar backend forced (via -Dfdb.vector.simd=scalar, no --add-modules). The module ships two interchangeable backends: a SIMD backend based on the jdk.incubator.vector API, and a scalar fallback. The standard test task exercises only the SIMD one; the scalarFallbackTest task covers the fallback. It is wired into check, so a normal build will run both. The selection of tests is affected by the following tags:

@Tag(Tags.RequiresSIMD) — for tests that assert SIMD-specific behavior. These run only in the standard test task (which excludes nothing extra; whereas the scalarFallbackTest task doesn’t include this tag).
@Tag(Tags.RequiresScalar) — for tests that assert scalar-specific behavior, such as bit-exact determinism that SIMD lane-reordering would break. The standard test task excludes it, and scalarFallbackTest includes it, so it runs only under scalar.
@Tag(Tags.DualScalarSIMD) — for parity/correctness tests that must hold under both backends. These run twice, under SIMD in test and under scalar in scalarFallbackTest. This is how SIMD-vs-scalar parity is actually exercised.

Running the `@YamlTest` integration tests¶

A large part of the SQL surface of the Record Layer is covered by YAML integration tests. These are declarative test cases written in .yamsql files and driven by the @YamlTest harness. They can be run under a variety of connection and version configurations. For example, there is a configuration that uses the maxRows: 1 option to force every query to page through a continuation after each individual row, which we use to exercise the continuation handling. Other configurations run the tests against an external server rather than an in-process connection. The following tasks each select a different configuration:

quickTest: Runs the tests in just the embedded configuration. This is most useful while developing new features and tests. You can iterate quickly against the embedded configuration alone, without repeatedly running a failing test under all the other configurations.
rpcTest: Like quickTest, but runs the tests through the JDBC driver and the gRPC protocol layer instead of a direct EmbeddedRelationalConnection. The server still runs in the same JVM as the test, but client and server talk over gRPC’s in-process transport (io.grpc.inprocess), which connects them through in-memory queues instead of network sockets. This exercises the request/response marshalling of the RPC path—which the embedded connection bypasses—without the cost or flakiness of a networked server. It will be useful when you are working on an aspect of the RPC protocol.
singleVersionTest: Runs the tests against multiple external servers that are all on the same version, rather than alternating between an external server and the current embedded connection as it normally would.
mixedModeTest: Runs the tests with each test alternating between the current version and some older version. This helps ensure that we remain backward-compatible for data and continuations. By default, it runs against the 10 most recent versions, but you can make it run against a single version with ./gradlew mixedModeTest -Ptests.mixedModeVersion=${version}.

The mentioned mixed-mode tests are run during the Nightly build and also as part of the Release workflow. A pull request build only runs the standard test and destructiveTest tasks against the current version by default, but you can opt into the extra mixed-mode validation by adding the Run mixed-mode label.

Reproducing a nightly test run¶

We have a Nightly GitHub action that we run every night. It includes tests that are exceptionally slow or exhibit randomness (ideally controlled via RandomizedTestUtils). If you wish to mimic such a test run locally, you’ll need to attach the following properties, though you don’t always need all of them. You can add these to a standard ./gradlew test run. For example, the following invocation mirrors what’s in our nightly.yml configuration:

./gradlew test -Ptests.nightly -Ptests.includeRandom -Ptests.iterations=2 -PspotbugsEnableHtmlReport

The relevant properties are:

tests.nightly, which does the following:
- It runs tests that are annotated with @SuperSlow, under a raised timeout of 20 minutes.
- It runs additional test parameters that are passed to the TestConfigurationUtils.onlyNightly() utility method, which are normally filtered out.
- It sets Gradle’s ignoreFailures flag, so that a failing test will not fail the build. Rather than aborting at the first failure, every test task will run to completion across all submodules, and any failures are recorded in the test reports. The rationale is that these tests are more likely to be flaky, so we want to run them all and inspect the reports rather than stop early.
tests.includeRandom, which includes tests tagged with @Tag(Tags.Random). It also sets the system locale to a randomly chosen one, which should help catch any code that inadvertently depends on the default locale being en-US.
tests.iterations=2, which sets how many random seeds to generate for tests that use RandomizedTestUtils. It defaults to 0, and has no effect unless tests.includeRandom is also set.

A note on the randomness controls: The fdb-record-layer-lucene module uses the lucene-test-framework module from the Apache Lucene project to further exercise our custom file format and directory. Because that framework runs on JUnit 4 (via the JUnit Vintage engine) rather than the JUnit Jupiter API used by the rest of our tests, it does not honor the mentioned randomness properties directly. Instead, it has its own equivalents, -Ptests.luceneIncludeRandom and -Ptests.luceneIterations, which the build translates into Lucene’s native tests.seed and tests.iters controls, so that randomness is enabled and seeded consistently across both test frameworks.

Publishing a local snapshot build¶

If you want to consume a local build in another project that depends on the FoundationDB Record Layer, you can publish it to your local Maven repository as follows:

./gradlew publishToMavenLocal -PpublishBuild=true

The published .jar files will be stored in the ~/.m2/repository/org/foundationdb/fdb-record-layer-core/n.n-SNAPSHOT/ directory.

Creating a patch branch¶

Patch branches allow us to ship a fix on top of an older release without having to pull in everything that has since landed on the main branch. Due to our branch protection rules on GitHub, such branches cannot be created directly. Instead, project maintainers can trigger a workflow that creates a patch branch on demand.

Manually trigger the Create Release Branch GitHub Action, specifying the base version in the format x.y.z. The action will check out the corresponding x.y.z.0 tag and push a new branch named x.y.z-release.
Once the branch exists, open a pull request for the patch against it. Remember to change the base branch to the patch branch before creating the PR, since GitHub’s “New pull request” page sets it to main by default.
From there, run the Release workflow against the patch branch. This bumps its version number and updates its release notes automatically, the same way it does for a regular release off of main.
Remember to cherry-pick or merge your changes further up once they land, since changes on a patch branch are not automatically merged back into main, and you’ll normally want them to be included in future releases off of main as well. If you merge the whole branch rather than cherry-picking individual commits, expect conflicts in gradle.properties and docs/sphinx/source/ReleaseNotes.md, since the two branches bump their version numbers and release notes independently. Make sure that the version on main is not accidentally changed, and that the release notes file contains all release notes from both branches afterward.