For testing CockroachDB we commonly use two separate frameworks:
for unit tests (one bit of functionality at a time), we use the TestServer/TestCluster framework.
for integration tests (using the entire CockroachDB server product as a customer would), we use the roachtest framework.
As a special case, integration tests that exercise the interactive (non-automatable) behavior of cockroachdb do not currently use roachtest.. Likewise, integration tests that exercise all types of client drivers do not use roachtest either. See the separate page on acceptance tests for details.
Similarities and Differences between the two frameworks
TestServer/TestCluster | Roachtest | |
|---|---|---|
Can be used to start a 1-node cluster or multi-node cluster | yes to both | |
Can be configured to use a custom external configuration, i.e. a configuration that a customer may change in production. For example: TLS certificate directory, special cluster settings, etc. | yes to both | |
Can be configured to a custom internal configuration, i.e. a testing knob that injects a custom dependency to an internal component or a special behavior not available in release builds. | yes | no |
Startup time until test cluster is ready for use by test | less than 300ms | more than 30s (100x slower) on first run, due to the need to build and/or download a cockroach binary after the first run, more than 3s (10x slower) |
Suitable for unit tests | yes | limited:
|
Suitable for integration tests for automatable product behavior and with standard SQL client drivers. | no - does not exercise external configuration code paths. | yes |
Suitable for integration tests for non-automatable product behavior and/or with non-standard SQL client drivers. | no - see separate page on acceptance testing | |
Can use a release | no | yes |
Can simulate mixed-version clusters and exercise mixed-version testing | limited (see discussion below) | yes |
When to use TestServer vs when to use Roachtest
One should opt to use TestServer/TestCluster when implementing unit tests that need a fully functioning SQL layer or KV layer.
Unit tests are valuable to ensure that a single component or function operates as designed, regardless of how it was integrated from other components.
Tests that do not need a full SQL layer and KV layer can exercise their components in a more focused manner without instantiating a full server via TestServer.
One would otherwise use Roachtest when implementing integration tests:
Integration tests are valuable to exercise a product journey as experienced by an end-user.
Integration tests are also the only way to reliably exercise external configuration mechanisms.
Currently, mixed-version testing is only easy to perform when using roachtest. So even when one wants to implement a “unit test”, as soon as one needs to also include mixed-version testing it is easier to achieve this using roachtest.
But not the only way. See discussion below.
Mixed-version testing
For mixed-version testing we currently have two frameworks:
for integration tests, there is a native and extensive mixed-version test runner inside roachtest. Refer to the roachtest docs for details.
for unit tests, we only have some limited support for mixed-version testing in the SQL logic test framework. It is achieved by downloading different versions of CockroachDB under the hood. Because of this, these mixed-version SQL logic tests can only really be run in CI.
It is still an open question as to how to perform mixed-version unit tests generally while keeping the test speed reasonable and high velocity on development cycles.