We use Go packages to delimit units of functionality.
We generally like smaller packages with a tight scope, but not so small that components that are always updated together are pulled away from each other by package boundaries.
Here are example reasons to create new packages:
New functionality is being added that is self-contained and can be described independently.
Protobuf definitions are being added to a package that did not use protobufs before.
An existing package has grown too large; for example:
it takes too long to run its unit tests.
there are so many components that newcomers become confused; it is generally harder to learn.
making changes to one place require verification to many other places, because of insufficient encapsulation.
another team has expressed interest in reusing a sub-part of the package but they cannot import the entire package.
New functionality requires a separate license (see below for details).
A portion of code uses custom build rules.
A piece of code is identified as a valuable standalone library that can be reused in other projects.
(In that case, we also sometimes elevate it to its own repository.)
Packages that correspond to sub-components of an existing package can be created as sub-directories.
However, we still prefer package names to be unique across the entire project, because otherwise goimports may auto-import the wrong package.
New packages for protobuf definitions should be placed in a xxxxpb sub-directory of the parent where they apply. For example, pkg/server/serverpb, pkg/sql/sessiondatapb. Protobuf definitions common to multiple components should be located as a sub-package of the nearest common parent package.
pkg/ccl sub-directories: package that define Enterprise features (see explanation below).
pkg/cmdsub-directories: packages that define a standalone program.
pkg/util sub-directories: “utility” packages that arguably can have value across multiple programs (even within the crdb repository).
In case of doubt, ask for recommendations from the nearest TL.
To name a package, we use the following guidelines:
All lower-case. No capitals or underscores.
Choose a name that does not need to be renamed using named imports at most call sites (unambiguous in the most frequent use contexts).
As discussed above, we still prefer package names to be unique across the entire project.
A good trick to make this work is to take the name of a parent package and add a suffix or prefix.
For example, server/serverpb, kv/kvserver, util/contextutil, util/testutil, etc.
Short and succinct. Remember that the name is identified in full at every call site.
Not plural. For example, net/url, not net/urls.
Not "common", "util", "shared", or "lib". These are bad, uninformative names.
See also Package Names and Style guideline for Go packages.
Sometimes after choosing a package layout, the Go compiler complains that cyclical dependencies are not allowed.
We can override this in two ways: either using interfaces or using dependency injection. There is also a trick for using helper packages in unit tests.
Suppose you want to define two packages a and b, and define a.Foo() to call b.Bar() and b.Bar() to call a.Foo().
Using interfaces, you would change the code in a as follows:
package a
import "b/bapi"
func Foo(b bapi.B) { b.Bar() } |
Then in package b
package b
import "a"
type b struct{}
// implements the bapi.B interface.
func (x b) Bar() { a.Foo(x) }
func Bar() { var x b; x.Bar() } |
And sub-directory bapi
package bapi
type B interface {
Bar()
} |
When using this model, ensure that package a has a mock implementation of the other package’s API so that its unit tests can run.
Suppose you want to define two packages a and b, and define a.Foo() to call b.Bar() and b.Bar() to call a.Foo().
In package a, for example, define var BarFn = func() { ... placeholder ...}
In package b, implement func init() { a.BarFn = Bar }
Note: We are OK with dep injection for the sake of making the code clearer as per the guidelines above, under the following conditions:
The portion of the cycle that can be imported separately (e.g. package a in the example above) must pass its unit tests independently from the cycle. This implies that the package must define valid behavior even when the injection slots are not populated.
There is a substantiated argument that the interface-based approach described above is impossible or impractical.
Often we want to use a helper package in a unit test, but just in the unit test, and that import creates a dependency cycle.
For this, Go already has a native functionality: test files (*_test.go) inside a package named X can declare another package X_test.
X_test can import another package that then recursively depends on X, and this way there is no cycle.
For example, in CockroachDB we often use testserver in tests. testserver depends on pkg/server, which depends on pkg/sql. So all the tests in the sql and server package that want a test server cannot be part of the sql or server packages; at the top of their source file we see package sql_test and package server_test.
CockroachDB contains code under two separate licenses:
BSL-licensed code defines the “core” product functionality available to all users, even those who do not have a paid contract with CRL. The BSL can be described as “delayed open source”: it automatically applies the Apache License to the code after a delay.
CCL-license code defines the “Enterprise” product functionality. It is not covered by the Apache License, even after a delay. In that way, it remains “proprietary”.
We strongly wish to retain the ability to build cockroach binaries that do not contain CCL-licensed code, for several reasons:
This ability provides us proof, by construction, that the CCL code is not necessary for the core functionality.
The command to build a CCL-free binary is make buildoss .
Certain users are not able to use proprietary software due to their organization’s policy and we want to provide them the ability to run CockroachDB nonetheless.
Certain system distributions only include open source software.
This means that none of the BSL code can import (using the Go import statement) CCL packages. So all the CCL functionality is injected into the BSL code using dependency injection.
(The converse is possible: CCL code can import BSL code, and this happens often.)