Error concepts

Programmers do not usually like thinking about errors. When learning how to program, initially, programming assignments are silent about error handling, or at best dismissive. For many applications, the “best practice” for error handling is “exit the program as soon as an error is encountered”.

In contrast, in a distributed system, especially in CockroachDB, error definitions and error handling are a critical aspect of product quality.

Here are some of the important things that we care about:

• Errors should not cause running servers (e.g. a database node) to terminate immediately. Customers would consider this an unacceptable defect. Correct and deliberate error handling is a core part of product quality and stability.

• Users will read the text of error messages, however users cannot be assumed to understand the source code. If an error message is confusing, the users will ask confused questions to our tech support. If an error message is misguiding, the users will ask the wrong questions to our tech support. And so on. Error messages should be clear and accurate and avoid referring to source code internals.

• Any error visible to one user will likely be visible to dozens, if not thousands of users eventually. We want our users to understand what they should do about an error on their own, so they do not need to reach out to technical support. For this, we want our error messages to be self-explanatory and include hint annotations. We also make specific error codes (e.g. SQLSTATE) part of our public, documented API for using CockroachDB.

• Errors are part of the API and thus error situations should be exercised in unit tests.

• Error make their way to log files and crash reports and can contain user-provided data. We care to separate customer confidential data from non-confidential data in log files and crash reports, and so we need to distinguish sensitive data inside error objects too.

Basic usage in Go

See the sub-page “Error handling basics”, which is also included in the overall Go style guide.

In a nutshell:

• We prefer the use of the CockroachDB errors library at github.com/cockroachdb/errors. This is a superset of Go's own errors and pkg/errors.

• use errors.Wrap to add context to an error

• handle type assertion failures gracefully with an error, instead of letting Go generate a panic

• avoid panics generally, unless in an init function or in a package that uses a disciplined panic-based error handling protocol (and converts panics to errors)

Errors and stability

Here are the general rules about how errors are allowed to impact the lifecycle of a network service:

Large strings inside error payloads

Be careful not to include arbitrarily large strings inside error payloads.

This can cause excessive memory consumption (even a server crash) and incomplete/truncated crash reports.

• A copy of the SQL syntax input by the SQL client is usually OK.

• Placeholder values or the body of COPY statements can be more tricky.

• Be especially careful with data loaded from storage.

• Be careful of data generated from SQL built-in functions or subqueries.

When in doubt, only include a prefix up to a maximum length. Use a special character (e.g. unicode ellipsis “…” ) to indicate that truncation happened.

Errors and performance

We work under the assumption that errors are important, but yet are uncommon.

There are two sides of this “uncommon“ coin:

• Error handling does not need to be optimized for performance. For example, we tolerate a moderate amount of string processing and heap allocations to construct error objects.

• Error objects should not be constructed on the common path. Only construct errors when needed.

For example:

func myFunc(x int) (result, error) {
   maybeErr := errors.New("hello")
   if x > 10 {
      return nil, maybeErr
   }
   return result, nil
}

func myFunc(x int) (result, error) {
   if x > 10 {
      return nil, errors.New("hello")
   }
   return result, nil
}

var maybeErr = errors.New("hello")
func myFunc(x int) (result, error) {
   if x > 10 {
      return nil, errors.WithStack(maybeErr)
   }
   return result, nil
}

Error messages, hints and codes

Error objects are structured. We use different parts of an error object for different purposes. Care should be taken to not stuff text/data intended for one field into another.

errors.Newf(“invalid input: %v”, userInput)

pgerror.New(pgcode.CheckViolation, "CHECK constraint failed")

pgerror.WithCandidateCode(someErr, pgcode.CheckViolation)

errors.WithHintf(
  errors.Newf("unknown value: %s", word),
  "Accepted values: %s", strings.Join(",", possibleValues)
)

errors.WithDetailf(
   errors.Newf(“invalid keyword: %s”, word),
   "Error encountered while processing input:\n%s", multiLineInput))

Errors as API

What does it mean that “errors are part of the documented API”?

• Whether an error can occur for given input situations is documented.

  • If an API is documented not to return an error, then users can consider CockroachDB defective if an error is returned.

• The set of possible errors is documented for these input situations.

  • If an API returns an error that was not documented as possible, then users can consider CockroachDB (or its documentation) defective.

• What to do when a given error occurs is documented.

  • If a API returns an error with no clear “next steps”, then users can consider CockroachDB (or its documentation) defective.

There is a careful balance to maintain: users want to have more guarantees, but each guarantee comes with an engineering burden.

Here is how we manage the amount of engineering work:

• We do not guarantee nor document the specific text of error messages, hints and details as part of our error API.

  • We emphasize “an error can occur” as the guarantee, not “this specific error will occur”.

  • Specific guarantees are expressed over the SQLSTATE values. These are unit tested.

  • Conversely, engineers are free to improve / extend / modify messages, hints and details without approval by the documentation and product team.

  • In some cases (this is a legacy case, which we strive to avoid nowadays), the guarantee includes a keyword at the first position in the message. For example “restart_transaction”.

• Mention when new SQLSTATE values are introduced, or when a single error case has been broken down into multiple alternatives, inside a release note in the commit message.

Checking errors, errors in unit tests

• Messages should not be considered stable:

  • inside Go code, use errors.Is , errors.As and errors.HasType / HasInterface, not .Error() = “…” or strings.Contains(…Error(), “…”)

  • in SQL logic tests, use regular expressions that only match the “important” part of a message

• In unit tests:

  • Check SQLSTATE values using SQL logic tests (error pgcode ….)

  • In Go unit tests, use testutils.IsError()

Sensitive data inside error objects

Many error objects are copied into logs, crash reports or other artifacts that are then communicated to CRL Tech Support automatically.

To preserve the confidentiality of our customer data, we are careful to isolate user-provided data from strings that are fixed inside CockroachDB. We call this “redactability”.

See the page Log and error redactability for more details.

General concepts:

• When something is potentially sensitive / confidential, we call it “unsafe” and it is automatically deleted / redacted out when sent to CRL Tech support.

  • This conservative approach maximally protects customer confidentiality.

  • We need to work extra to include bits of known-safe data into errors to make the error reports more useful during troubleshooting.

• The CockroachDB errors library already knows about redactability and helps engineers as follows:

  • The first literal string arguments to errors.New , errors.Newf, errors.Wrap etc is automatically considered non-confidential / non-redactable.

  • Most “simple” numeric values are automatically considered non-redactable.

  • All string values passed as positional arguments to error constructors and annotation functions are considered sensitive and thus redactable.

  • More non-redactability for values passed to error constructors are possible via the SafeFormatter interface (see implementations of SafeFormat throughout the source code)

  • Error objects used as input to a new error object are decomposed into redactable and non-redactable bits automatically.

• Errors constructed outside of cockroachdb/errors, e.g. via fmt.Errorf, are considered sensitive and thus fully redactable.