Page Comparison

...

--cross takes an optional argument which is the platform to cross-compile to: --cross=linux, --cross=windows, --cross=macos, --cross=linuxarm, --cross=macosarm. dev will copy the built binaries into the artifacts directory in this case. Note that cross-building requires you a docker-compatible system installed like Rancher Desktop.

...

# Build dev
./dev build dev
# Build crlfmt
./dev build crlfmt
# Build roachprod
./dev build roachprod
# Run acceptance tests
./dev acceptance
# Run compose tests
./dev compose
# Run benchmarks for pkg/sql/parser
./dev bench pkg/sql/parser
# Generate code and docs (run this before submitting your PR).
./dev generate
# Generate changes to BUILD.bazel files
./dev generate bazel --short
# Run lints
./dev lint
# logic tests!
./dev testlogic --files=$FILES --subtests=$SUBTESTS --config=$CONFIG
# Open a container running the "bazelbuilder" image. Requires Docker/Rancher Desktop/Podman/etc.
./dev builder
# Remove artifacts from building the UI
./dev ui clean --all
# Start the Bazel cache server after rebooting
./dev cache

dev vs. make

This is a (non-exhaustive) 1-to-1 mapping of dev commands to their make equivalents. Feel free to add to this

...

dev/bazel command

...

equivalent non-bazel command

...

./dev build

...

make build

...

./dev build short

...

make buildshort

...

./dev build pkg/sql/...

...

make build PKG=./pkg/sql/...

...

./dev test

...

make test

...

./dev test pkg/sql/parser -f TestParse

...

make test PKG=./pkg/sql/parser TESTS=TestParse

...

./dev test pkg/sql/parser -f TestParse --test-args '-test.count=5 -show-logs'

...

make test PKG=./pkg/sql/parser TESTS=TestParse TESTFLAGS='-count=5 -show-logs'

...

./dev bench pkg/sql/parser -f BenchmarkParse

...

make bench PKG=./pkg/sql/parser BENCHES=BenchmarkParse

...

./dev build --cross

...

build/builder.sh mkrelease

...

./dev builder

...

build/builder.sh

...

./dev testlogic base --files=fk --subtests=20042 --config=local

...

make testbaselogic FILES=fk SUBTESTS=20042 TESTCONFIG=local

...

./dev test ... pkg/kv/kvserver/ -- --define gotags=bazel,gss,X,Y

...

make test ... TAGS=X,Y

...

Add gc_goopts = ["S"], to the go_library target in the BUILD.bazel file for the package you’re interested in, then running dev

...

make ... GOFLAGS=-gcflags=-S

...

Update the go_repository() declaration in DEPS.bzl for your dependency to point to a new remote and commit (see top-level comment in DEPS.bzl for more information), then build/test

...

Update local sources in vendor including your changes, then build/test

General dev tips

The top-level dev script uses Bazel to build pkg/cmd/dev before running unless another dev binary with the same increasing integer ID has already been built. Generally dev will invoke the dev binary “as of” that commit, which should usually be the correct behavior. However, if the default behavior does not work for some reason, you can find all the built versions of dev under bin/dev-versions.

A (hopefully) fast and error proof dev workflow

1. Switch to a new branch

2. If your workflow involves an IDE, generate your protos ./dev gen protobuf

• Your IDE relies on generated files for many tasks (e.g. code navigation, IntelliSense, debugging), and will complain unless you have re-generated those files.

  • If you need to re-generate all generated go files, use the slower ./dev gen go

  • If the above fails, run the slowest ./dev gen to update all of your generated files.

  • You may recall that with make , this step was not necessary. If you’re curious why, see this slack thread.

3. If your workflow involves UI development, you’ll want additionally do the following:

./dev gen protobuf 
./dev generate js
# start a cockroach node, e.g.
./dev build && ./cockroach start-single-node
# in separate window, start UI watch for incremental UI builds
./dev ui watch
# now you're ready to write UI code!

4. Write some code!

• If you don’t have crlfmt already, you’ll need to ./dev build crlfmt to use it for formatting.

• If you add new files or imports, run ./dev gen bazel before compiling or running a test. compilepkg: missing strict dependencies: is usually the indicator that ./dev gen bazel needs to be re-run.

  • to skip this step, see tip below on ALWAYS_RUN_GAZELLE

• Build the binary: ./dev build short

5. Run a test

• On an IDE: your normal workflow should work if your generated files are up to date (See step 2).

• From the command line: ./dev test [path/to/pkg] --filter [test_name]

6. Before opening/updating a PR:

• Run ./dev lint --short (maybe additionally make lintshort as dev's linter doesn’t have 100% coverage yet)

• Assert your workspace is clean by running ./dev gen bazel . If you modified other generated files, run the appropriate ./dev gen [file_type] command.

Rapidly iterating with dependencies

The file DEPS.bzl tells Bazel how to download dependencies. For production, we point to .zip files that are mirrored on our internal infrastructure, protecting us against dependency yanking/”left-pad”-style failures. However, for local development, you have a few other options.

The top-level comment at the top of DEPS.bzl explains how to point to a custom remote for a dependency, for example:

go_repository(
    name = "com_github_cockroachdb_sentry_go",
    build_file_proto_mode = "disable_global",
    importpath = "github.com/cockroachdb/sentry-go",
    vcs = "git",
    remote = "https://github.com/rickystewart/sentry-go",  # Custom fork.
    commit = "6c8e10aca9672de108063d4953399bd331b54037",  # Custom commit.
)

In this example, github.com/cockroachdb/sentry-go will point to the given remote and commit instead of using the production version of the library. Note the remote can be either a normal git https remote or it can be a local clone.

...

google$ git clone https://github.com/google/btree
Cloning into 'btree'...
remote: Enumerating objects: 163, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 163 (delta 16), reused 24 (delta 10), pack-reused 123
Receiving objects: 100% (163/163), 77.18 KiB | 1.07 MiB/s, done.
Resolving deltas: 100% (84/84), done.
google$ cd btree
btree$ git checkout v1.0.1
Note: switching to 'v1.0.1'.
........
HEAD is now at 479b5e8 Minor documentation fix, DescendGreaterThan starts with the last item in the tree and decends to the least item greater than the pivot
btree$ pwd
/Users/ricky/go/src/github.com/google/btree

Back in cockroach I update .bazelrc.user to point to the clone I just made. The form of the flag is --override_repository=REPO_NAME=/path/to/local/repo. The flag tells Bazel to ignore where REPO_NAME "really is", and instead just use the local clone. Note that DEPS.bzl declares the "name of the repo" which in this context is Java-style, like com_github_google_btree. I am going to add it to .bazelrc.user so I don't have to remember to add the flag every time, although it's a normal Bazel flag so you can just include it on the command-line too.

cockroach$ echo 'build --override_repository=com_github_google_btree=/Users/ricky/go/src/github.com/google/btree' >> .bazelrc.user

The first thing I'll do is build just to demonstrate that what I've done so far is a no-op.

cockroach$ ./dev build short 
$ bazel build //pkg/cmd/cockroach-short:cockroach-short
INFO: Invocation ID: bc808544-70de-4fc9-968c-66a815f64437
ERROR: /Users/ricky/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/BUILD.bazel:4:11: //pkg/ccl/changefeedccl:changefeedccl depends on @com_github_google_btree//:btree in repository @com_github_google_btree which failed to fetch. no such package '@com_github_google_btree//': No WORKSPACE file found in /private/var/tmp/_bazel_ricky/be70b24e7357091e16c49d70921b7985/external/com_github_google_btree

Oh, whoops. The BUILD.bazel file and WORKSPACE files are missing because I didn't run Gazelle. Let me fix that. From back in the btree directory:

...

To pass -gcflags to the build of a library, add gc_goopts = ["S"], to the go_library target in the BUILD.bazel file for the package you’re interested in, then run devas usual.

To override a dependency for local builds when doing automation, update the go_repository() declaration in DEPS.bzl for your dependency to point to a new remote and commit (see top-level comment in DEPS.bzl for more information), then build/test as usual.

General dev tips

The top-level dev script uses Bazel to build pkg/cmd/dev before running unless another dev binary with the same increasing integer ID has already been built. Generally dev will invoke the dev binary “as of” that commit, which should usually be the correct behavior. However, if the default behavior does not work for some reason, you can find all the built versions of dev under bin/dev-versions.

A (hopefully) fast and error proof dev workflow

1. Switch to a new branch

2. If your workflow involves an IDE, generate your protos ./dev gen protobuf

• Your IDE relies on generated files for many tasks (e.g. code navigation, IntelliSense, debugging), and will complain unless you have re-generated those files.

  • If you need to re-generate all generated go files, use the slower ./dev gen go

  • If the above fails, run the slowest ./dev gen to update all of your generated files.

  • If this fails too, try git clean. If GOLAND complains about dependent packages, try git clean -dfx pkg instead. Then repeat the steps above.

  • You may recall that with make , this step was not necessary. If you’re curious why, see this slack thread.

3. If your workflow involves UI development, you’ll want additionally do the following:

./dev gen protobuf 
./dev generate js
# start a cockroach node, e.g.
./dev build && ./cockroach start-single-node
# in separate window, start UI watch for incremental UI builds
./dev ui watch
# now you're ready to write UI code!

4. Write some code!

• If you don’t have crlfmt already, you’ll need to ./dev build crlfmt to use it for formatting.

• If you add new files or imports, run ./dev gen bazel before compiling or running a test. compilepkg: missing strict dependencies: is usually the indicator that ./dev gen bazel needs to be re-run.

  • to skip this step, see tip below on ALWAYS_RUN_GAZELLE

• Build the binary: ./dev build short

5. Run a test

• On an IDE: your normal workflow should work if your generated files are up to date (See step 2).

• From the command line: ./dev test [path/to/pkg] --filter [test_name]

6. Before opening/updating a PR:

• Run ./dev lint --short (maybe additionally make lintshort as dev's linter doesn’t have 100% coverage yet)

• Assert your workspace is clean by running ./dev gen bazel . If you modified other generated files, run the appropriate ./dev gen [file_type] command.

Rapidly iterating with dependencies

The file DEPS.bzl tells Bazel how to download dependencies. For production, we point to .zip files that are mirrored on our internal infrastructure, protecting us against dependency yanking/”left-pad”-style failures. However, for local development, you have a few other options.

The top-level comment at the top of DEPS.bzl explains how to point to a custom remote for a dependency, for example:

go_repository(
    name = "com_github_cockroachdb_sentry_go",
    build_file_proto_mode = "disable_global",
    importpath = "github.com/cockroachdb/sentry-go",
    vcs = "git",
    remote = "https://github.com/rickystewart/sentry-go",  # Custom fork.
    commit = "6c8e10aca9672de108063d4953399bd331b54037",  # Custom commit.
)

In this example, github.com/cockroachdb/sentry-go will point to the given remote and commit instead of using the production version of the library. Note the remote can be either a normal git https remote or it can be a local clone.

In this case, iterating can be cumbersome as you have to update the commit whenever you want to pull a new version of the dependency. You can use the Bazel flag --override_repository to optimize for this case, so you can make changes locally on your machine and immediately re-build cockroach with your latest local changes instead of updating the dependency to point to a new commit whenever you want to test your changes. The following explanation is copy-pasted from internal Slack:

The process doesn't vary per dependency so I'll demonstrate with github.com/google/btree. First I'm going to clone that repo and check out the version of the code I want.

google$ git clone https://github.com/google/btree
Cloning into 'btree'...
remote: Enumerating objects: 163, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 163 (delta 16), reused 24 (delta 10), pack-reused 123
Receiving objects: 100% (163/163), 77.18 KiB | 1.07 MiB/s, done.
Resolving deltas: 100% (84/84), done.
google$ cd btree
btree$ git checkout v1.0.1
Note: switching to 'v1.0.1'.
........
HEAD is now at 479b5e8 Minor documentation fix, DescendGreaterThan starts with the last item in the tree and decends to the least item greater than the pivot
btree$ pwd
/Users/ricky/go/src/github.com/google/btree

Back in cockroach I update .bazelrc.user to point to the clone I just made. The form of the flag is --override_repository=REPO_NAME=/path/to/local/repo. The flag tells Bazel to ignore where REPO_NAME "really is", and instead just use the local clone. Note that DEPS.bzl declares the "name of the repo" which in this context is Java-style, like com_github_google_btree. I am going to add it to .bazelrc.user so I don't have to remember to add the flag every time, although it's a normal Bazel flag so you can just include it on the command-line too.

cockroach$ echo 'build --override_repository=com_github_google_btree=/Users/ricky/go/src/github.com/google/btree' >> .bazelrc.user

The first thing I'll do is build just to demonstrate that what I've done so far is a no-op.

cockroach$ ./dev build short 
$ bazel build //pkg/cmd/cockroach-short:cockroach-short
INFO: Invocation ID: bc808544-70de-4fc9-968c-66a815f64437
ERROR: /Users/ricky/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/BUILD.bazel:4:11: //pkg/ccl/changefeedccl:changefeedccl depends on @com_github_google_btree//:btree in repository @com_github_google_btree which failed to fetch. no such package '@com_github_google_btree//': No WORKSPACE file found in /private/var/tmp/_bazel_ricky/be70b24e7357091e16c49d70921b7985/external/com_github_google_btree

Oh, whoops. The BUILD.bazel file and WORKSPACE files are missing because I didn't run Gazelle. Let me fix that. From back in the btree directory:

# NB: The WORKSPACE file needs to exist, it can be empty though.
btree$ touch WORKSPACE
btree$ go install github.com/bazelbuild/bazel-gazelle/cmd/gazelle@latest
go: downloading github.com/bazelbuild/bazel-gazelle v0.29.0
go: downloading github.com/bazelbuild/buildtools v0.0.0-20230111132423-06e8e2436a75
go: downloading github.com/bmatcuk/doublestar/v4 v4.6.0
btree$ ~/go/bin/gazelle -go_prefix=github.com/google/btree -repo_root=.
# Validate the BUILD.bazel file was created
btree$ git status
HEAD detached at v1.0.1
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	BUILD.bazel
	WORKSPACE

nothing added to commit but untracked files present (use "git add" to track)

...

When you're done with testing your local changes, you can remove the --override_repository line from the --override_repository line from .bazelrc.user.

General Bazel tips

• Bazel has a configuration file called .bazelrc. You can put a global configuration file at ~/.bazelrc or a per-repository file at .bazelrc.user

...

General Bazel tips

• Bazel has a configuration file called .bazelrc. You can put a global configuration file at ~/.bazelrc or a per-repository file at .bazelrc.user in the root of your cockroach repo in the root of your cockroach repo.

• Stripping: stripping of symbols in built binaries is enabled by default, as disabling stripping slows down linking drastically. You can disable stripping with the Bazel flags -c dbg or -c opt (if you are making a binary you wish to debug, you will use -c dbg), or you can force-enable it with --strip=never.

  • Binaries built with dev build --cross, by the release process, or for nightly roachtests are built with -c opt and will therefore be unstripped.

• While Bazel is the “official” build system, you do not have to use it for normal development. For example, many people do development from their IDE’s, and this is expected to “just work”. Note that since not all generated code is checked into the repo, you’ll first have to generate code to get much of it to build from a non-Bazel build system. We refer to this as the “escape hatch”. This escape hatch is specifically supported so if you have difficulty running a test in another build system after generating code, that’s a bug you should report. You can run the following commands to make this happen:

  • dev gen go

    • Generates all .go code that goes into the build, including cgo code

  • dev gen cgo

    • Generates some stub files that tell cgo how to link in the c-deps; part of dev gen go

  • dev gen protobuf

    • Generates all .pb.go/.pb.gw.go files; part of dev gen go

• Tired of running ./dev gen bazel? Set the ALWAYS_RUN_GAZELLE env-var to automatically run gazelle before every dev test or dev build incantation. Note this does add a tiny delay – noticeable when iterating on small tests through dev test.

  • i.e. echo 'export ALWAYS_RUN_GAZELLE=1' >> ~/.zshrc

  • Note that gazelle is only a subset of the aactions that dev gen bazel performs. This by itself is able to handle most updates to the code, but is not able to handle things like vendoring new dependencies (dev gen bazel can do this for you).

• If you have ccache installed, bazel will fail with an error like ccache: error: Failed to create temporary file for /home/alyshanjahani/.ccache/tmp/message_li.stdout: Read-only file system. To avoid this you should get the ccache links out of your PATH manually (i.e. uninstall ccache), and then you might need to do bazel clean --expunge.

  • Alternatively, if you would like to use Bazel with ccache, you can enable support for writing outside the sandbox by adding the following to your $HOME/.bazelrc or <repo>/.bazelrc.user file:
    - For MacOS/Darwin:

    Code Block
    build --sandbox_writable_path=/Users/<USER>/Library/Caches/ccache/

    - For Linux:

    Code Block
    build --sandbox_writable_path=/home/<USER>/.ccache

...