For testing CockroachDB we commonly use two separate frameworks:
for unit tests (one bit of functionality at a time), we use the TestServer/TestCluster framework.
for integration tests (using the entire CockroachDB server product as a customer would), we use the roachtest framework.
Similarities and Differences between the two frameworks
TestServer/TestCluster | Roachtest | |
|---|---|---|
Can be used to start a 1-node cluster or multi-node cluster | yes to both | |
Can be configured to use a custom external configuration, i.e. a configuration that a customer may change in production. For example: TLS certificate directory, special cluster settings, etc. | yes to both | |
Can be configured to a custom internal configuration, i.e. a testing knob that injects a custom dependency to an internal component or a special behavior not available in release builds. | yes | no |
Startup time until test cluster is ready for use by test | less than 300ms | more than 30s (100x slower) on first run, due to the need to build and/or download a cockroach binary after the first run, more than 3s (10x slower) |
Suitable for unit tests | yes | limited:
|
Suitable for integration tests | no - does not exercise external configuration code paths. | yes |
Can use a release | no | yes |
Can simulate mixed-version clusters and exercise mixed-version testing | limited (see discussion below) | yes |
Mixed-version testing in unit tests
For mixed-version testing we currently have two frameworks:
for integration tests, there is a native and extensive mixed-version test runner inside roachtest. Refer to the roachtest docs for details.
for unit tests, we only have some limited support for mixed-version testing in the SQL logic test framework. It is achieved by downloading different versions of CockroachDB under the hood. Because of this, these mixed-version SQL logic tests can only really be run in CI.
It is still an open question as to how to perform mixed-version unit tests generally while keeping the test speed reasonable and high velocity on development cycles.