Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

For testing CockroachDB we commonly use two separate frameworks:

  • for unit tests (one bit of functionality at a time), we use the TestServer/TestCluster framework.

  • for integration tests (using the entire CockroachDB server product as a customer would), we use the roachtest framework.

    • As a special case, integration tests that exercise the interactive (non-automatable) behavior of cockroachdb do not currently use roachtest.. Likewise, integration tests that exercise all types of client drivers do not use roachtest either. See the separate page on acceptance tests for details.

Similarities and Differences between the two frameworks

TestServer/TestCluster

Roachtest

Can be used to start a 1-node cluster or multi-node cluster

yes to both

Can be configured to use a custom external configuration, i.e. a configuration that a customer may change in production.

For example: TLS certificate directory, special cluster settings, etc.

yes to both

Can be configured to a custom internal configuration, i.e. a testing knob that injects a custom dependency to an internal component or a special behavior not available in release builds.

yes

no

Startup time until test cluster is ready for use by test

less than 300ms

more than 30s (100x slower) on first run, due to the need to build and/or download a cockroach binary

after the first run, more than 3s (10x slower)

Suitable for unit tests

yes

limited:

  • slow startup hinders fast development cycles

  • exercises the entire cockroach binary so difficult to isolate individual components.

Suitable for integration tests for automatable product behavior and with standard SQL client drivers.

no - does not exercise external configuration code paths.

yes

Suitable for integration tests for non-automatable product behavior and/or with non-standard SQL client drivers.

no - see separate page on acceptance testing

Can use a release cockroach binary

no

yes

Can simulate mixed-version clusters and exercise mixed-version testing

limited (see discussion below)

yes

When to use TestServer vs when to use Roachtest

  • One should opt to use TestServer/TestCluster when implementing unit tests that need a fully functioning SQL layer or KV layer.

    • Unit tests are valuable to ensure that a single component or function operates as designed, regardless of how it was integrated from other components.

    • Tests that do not need a full SQL layer and KV layer can exercise their components in a more focused manner without instantiating a full server via TestServer.

  • One would otherwise use Roachtest when implementing integration tests:

    • Integration tests are valuable to exercise a product journey as experienced by an end-user.

    • Integration tests are also the only way to reliably exercise external configuration mechanisms.

  • Currently, mixed-version testing is only easy to perform when using roachtest. So even when one wants to implement a “unit test”, as soon as one needs to also include mixed-version testing it is easier to achieve this using roachtest.

    • But not the only way. See discussion below.

Mixed-version testing

For mixed-version testing we currently have two frameworks:

  • for integration tests, there is a native and extensive mixed-version test runner inside roachtest. Refer to the roachtest docs for details.

  • for unit tests, we only have some limited support for mixed-version testing in the SQL logic test framework. It is achieved by downloading different versions of CockroachDB under the hood. Because of this, these mixed-version SQL logic tests can only really be run in CI.

  • It is still an open question as to how to perform mixed-version unit tests generally while keeping the test speed reasonable and high velocity on development cycles.

  • No labels