For testing CockroachDB we commonly use two separate frameworks:
...
Similarities and Differences between the two frameworks
TestServer/TestCluster | Roachtest | |
|---|---|---|
Can be used to start a 1-node cluster or multi-node cluster | yes to both | |
Can be configured to use a custom external configuration, i.e. a configuration that a customer may change in production. For example: TLS certificate directory, special cluster settings, etc. | yes to both | |
Can be configured to a custom internal configuration, i.e. a testing knob that injects a custom dependency to an internal component or a special behavior not available in release builds. | yes | no |
Startup time until test cluster is ready for use by test | less than 300ms | more than 30s (100x slower) on first run, due to the need to build and/or download a cockroach binary after the first run, more than 3s (10x slower) |
Suitable for unit tests | yes | limited:
|
Suitable for integration tests for automatable product behavior and with standard SQL client drivers. | no - does not exercise external configuration code paths. | yes |
Suitable for integration tests for non-automatable product behavior and/or with non-standard SQL client drivers. | no - see separate page on acceptance testing | |
Address Space Isolation | no | yes |
Can use a release | no | yes |
Can simulate mixed-version clusters and exercise mixed-version testing | limited (see discussion below) | yes |
NOTE: TestServer/TestCluster shares the same address space (of the test binary) amongst server instances. This may result in surprising behavior when server settings are shared; e.g., https://github.com/cockroachdb/cockroach/issues/112395
When to use TestServer vs when to use Roachtest
...
for integration tests, there is a native and extensive mixed-version test runner inside roachtest. Refer to the roachtest docs for details.
for unit tests, we only have some limited support for mixed-version testing in the SQL logic test framework. It is achieved by downloading different versions of CockroachDB under the hood. Because of this, these mixed-version SQL logic tests can only really be run in CI.
It is still an open question as to how to perform mixed-version unit tests generally while keeping the test speed reasonable and high velocity on development cycles.