For testing CockroachDB we commonly use two separate frameworks:
for unit tests (one bit of functionality at a time), we use the TestServer/TestCluster framework.
for integration tests (using the entire CockroachDB server product as a customer would), we use the roachtest framework.
Similarities and Differences between the two frameworks
TestServer/TestCluster | Roachtest | |
|---|---|---|
Can be used to start a 1-node cluster or multi-node cluster | yes to both | |
Can be configured to use a custom external configuration, i.e. a configuration that a customer may change in production. For example: TLS certificate directory, special cluster settings, etc. | yes to both | |
Can be configured to a custom internal configuration, i.e. a testing knob that injects a custom dependency to an internal component or a special behavior not available in release builds. | yes | no |
Startup time until test cluster is ready for use by test | less than 300ms | more than 30s (100x slower) on first run, due to the need to build and/or download a cockroach binary after the first run, more than 3s (10x slower) |
Suitable for unit tests | yes | limited:
|
Suitable for integration tests | no - does not exercise external configuration code paths. | yes |
Can use a release | no | yes |
Can simulate mixed-version clusters and exercise mixed-version testing | limited (see discussion below) | yes |
When to use TestServer vs when to use Roachtest
One should opt to use TestServer/TestCluster when implementing unit tests that need a fully functioning SQL layer or KV layer.
Unit tests are valuable to ensure that a single component or function operates as designed, regardless of how it was integrated from other components.
Tests that do not need a full SQL layer and KV layer can exercise their components in a more focused manner without instantiating a full server via TestServer.
One would otherwise use Roachtest when implementing integration tests:
Integration tests are valuable to exercise a product journey as experienced by an end-user.
Integration tests are also the only way to reliably exercise external configuration mechanisms.
Currently, mixed-version testing is only easy to perform when using roachtest. So even when one wants to implement a “unit test”, as soon as one needs to also include mixed-version testing it is easier to achieve this using roachtest.
But not the only way. See discussion below.
Mixed-version testing
For mixed-version testing we currently have two frameworks:
for integration tests, there is a native and extensive mixed-version test runner inside roachtest. Refer to the roachtest docs for details.
for unit tests, we only have some limited support for mixed-version testing in the SQL logic test framework. It is achieved by downloading different versions of CockroachDB under the hood. Because of this, these mixed-version SQL logic tests can only really be run in CI.
It is still an open question as to how to perform mixed-version unit tests generally while keeping the test speed reasonable and high velocity on development cycles.