Overview

TestServer and TestCluster are two frameworks we built to create Go unit tests for CockroachDB:

TestServer simulates a single CockroachDB node.
TestCluster simulates a multi-node CockroachDB cluster. Each node is simulated using one TestServer.

By default, TestServer uses in-RAM storage (not persisted) to make tests faster..

Here are the main differences between a regular CockroachDB node (e.g. one started via cockroach start) and one simulated via TestServer:

Component

Behavior in cockroach start

Behavior in TestServer

SQL, HTTP and KV layers

identical

Coordination glue around SQL/HTTP/KV to create an overall running server (topLevelServer in code)

identical

Server configuration

Via CLI flags, defaults useful for production deployments.

Via TestServerArgs struct; defaults useful for testing.

For example, KV storage is in-RAM by default (no persistence) to make tests faster. There is also a predefined TLS configuration with a self-signed CA.

Of note, cockroach demo uses TestServer under the hood so all the properties and limitations of TestServer apply to cockroach demo as well.

Introduction to TestServer and TestCluster in Go unit tests

Tests mostly use the following programming pattern:

func TestSomething(t *testing.T) {
  defer leaktest.AfterTest()()  // verify no goroutine leaks
  defer log.Scope(t).Close(t)   // capture test logs intelligently

  ctx := context.Background()
  srv := serverutils.StartServerOnly(t, base.TestServerArgs{})  // initialize and start a TestServer
  defer srv.Stopper().Stop(ctx) // ensure the TestServer gets cleaned up at the end of the test
  
  ts := srv.ApplicationLayer()  // see below for an explanation
  
  // ... use ts in test code ...

Alternatively, the following is also possible:

    srv, db, kvDB := serverutils.StartServer(t, base.TestServerArgs{})

as it is equivalent to:

    srv := serverutils.StartServerOnly(t, base.TestServerArgs{})
    db := srv.ApplicationLayer().SQLConn(t, "")  // access to SQL
    kvDB := srv.ApplicationLayer().DB()          // access to KV

When a test needs to exercise a cluster of 2 or more nodes connected together, it can use TestCluster:

func TestSomething(t *testing.T) {
  defer leaktest.AfterTest()()  // verify no goroutine leaks
  defer log.Scope(t).Close(t)   // capture test logs intelligently

  ctx := context.Background()
  const numNodes = 3
  tc := serverutils.StartCluster(t, numNodes, base.TestClusterArgs{})  // initialize and start a TestCluster
  defer tc.Stopper().Stop(ctx) // ensure the TestCluster gets cleaned up at the end of the test
  
  ts0 := tc.Server(0).ApplicationLayer()  // see below for an explanation
  ts1 := tc.Server(1).ApplicationLayer()  // see below for an explanation  
  
  // ... use ts0 and ts1 in test code ...

In a nutshell, the result of StartCluster (TestClusterInterface) has a Server(nodeIdx) method which returns a different TestServer for each node in the simulated cluster.

Note: there is no benefit to using StartCluster with just 1 node. Prefer StartServer in that case.

High-level TestServer API

Let’s inspect the interface of Serverutils.StartServerOnly():

func StartServerOnly(t TestFataler, params base.TestServerArgs) TestServerInterface

type TestServerInterface interface {
  TestServerController

  // ApplicationLayer returns the interface to the application layer that is
  // exercised by the test.
  ApplicationLayer() ApplicationLayerInterface

  // SystemLayer returns the interface to the application layer
  // of the system tenant.
  SystemLayer() ApplicationLayerInterface

  // StorageLayer returns the interface to the storage layer.
  StorageLayer() StorageLayerInterface
  ...
}

// TestServerController defines the control interface for a test server.
type TestServerController interface {
   ...
  // Stopper returns the stopper used by the server.
  Stopper() *stop.Stopper
  ...
}

// ApplicationLayerInterface defines accessors to the application
// layer of a test server.
type ApplicationLayerInterface interface {
  ...
  // SQLConn returns a handle to the server's SQL interface, opened
  // with the 'root' user.
  // The connection is closed automatically when the server is stopped.
  SQLConn(t TestFataler, dbName 𝓢) *gosql.DB
  ...
}

// StorageLayerInterface defines accessors to the storage layer of a
// test server. See ApplicationLayerInterface for the relevant
// application-level APIs.
type StorageLayerInterface interface {
  ...
  // LookupRange looks up the range descriptor which contains key.
  LookupRange(key roachpb.Key) (roachpb.RangeDescriptor, ⊙)
  ...
}