For testing CockroachDB we commonly use two separate frameworks:
for unit tests (one bit of functionality at a time), we use the TestServer/TestCluster framework.
for integration tests (using the entire CockroachDB server product as a customer would), we use the roachtest framework.
As a special case, integration tests that exercise the interactive (non-automatable) behavior of cockroachdb do not currently use roachtest.. Likewise, integration tests that exercise all types of client drivers do not use roachtest either. See the separate page on interactive acceptance tests for details.
Similarities and Differences between the two frameworks
TestServer/TestCluster | Roachtest | |
|---|---|---|
Can be used to start a 1-node cluster or multi-node cluster | yes to both | |
Can be configured to use a custom external configuration, i.e. a configuration that a customer may change in production. For example: TLS certificate directory, special cluster settings, etc. | yes to both | |
Can be configured to a custom internal configuration, i.e. a testing knob that injects a custom dependency to an internal component or a special behavior not available in release builds. | yes | no |
Startup time until test cluster is ready for use by test | less than 300ms | more than 30s (100x slower) on first run, due to the need to build and/or download a cockroach binary after the first run, more than 3s (10x slower) |
Suitable for unit tests | yes | limited:
|
Suitable for integration tests for automatable product behavior and with standard SQL client drivers. | no - does not exercise external configuration code paths. | yes |
Suitable for integration tests for non-automatable product behavior and/or with non-standard SQL client drivers. | no - see separate page on acceptance testing | |
Can use a release | no | yes |
Can simulate mixed-version clusters and exercise mixed-version testing | limited (see discussion below) | yes |
...
for integration tests, there is a native and extensive mixed-version test runner inside roachtest. Refer to the roachtest docs for details.
for unit tests, we only have some limited support for mixed-version testing in the SQL logic test framework. It is achieved by downloading different versions of CockroachDB under the hood. Because of this, these mixed-version SQL logic tests can only really be run in CI.
It is still an open question as to how to perform mixed-version unit tests generally while keeping the test speed reasonable and high velocity on development cycles.