...
This tutorial assumes you have a running cockroach process.
Debugging A Process
Step 1: Get delve (OSX, Linux):
Running go get should be the only thing you need to do:
| Code Block |
|---|
$ go get -u github.com/go-delve/delve/cmd/dlv |
Step 2: Attach to process:
Get the process ID of the cockroach node you want to attach to and run:
...
Note that you must have the source code at the same sha as the binary you are debugging checked out for any debugging to make sense.
Step 3: Do what you came to do:
Now you attached to the process, the process is paused. You can jump around and look at things. For example, you can list the currently running goroutines and their IDs:
...
This can be scripted to extract information from a running process with an interruption that is (hopefully) small enough to not disturb the situation.
...
Debugging Tests
A common place to drop into a debugger is right in the middle of a unit test. This allows one to quickly iterate in a TDD fashion. With Delve there is a unique sub-command for starting a debugging session with tests running: dlv test. After executing this command, you break at the start of a specific unit test by issuing a break <TestName> statement. Finally, if you don't know the name of the test (or don't want to stop dlv to find it), you can issue the funcs command with a regex expression to list the tests of similar names. For example, here is a snippet of dropping into a dlv test session and breaking on a test in the github.com/cockroachlabs/managed-service/pkg/auditlog directory:
| Code Block |
|---|
$ dlv test
Type 'help' for list of commands.
(dlv) funcs auditlog.Test*
github.com/cockroachlabs/managed-service/pkg/auditlog.TestAuditLogMiddleware_LogSuccessfulQueryWithSensitiveDataToScrub
github.com/cockroachlabs/managed-service/pkg/auditlog.TestAuditLogMiddleware_LogSuccessfulQueryWithSensitiveDataToScrub.func1
github.com/cockroachlabs/managed-service/pkg/auditlog.TestAuditLogMiddleware_LogSuccessfulQueryWithSensitiveDataToScrub.func2
github.com/cockroachlabs/managed-service/pkg/auditlog.TestAuditLogMiddleware_LogSuccessfulQueryWithSensitiveDataToScrub.func3
github.com/cockroachlabs/managed-service/pkg/auditlog.TestLogFailedQuery
github.com/cockroachlabs/managed-service/pkg/auditlog.TestLogFailedQuery.func1
github.com/cockroachlabs/managed-service/pkg/auditlog.TestLogFailedQuery.func2
github.com/cockroachlabs/managed-service/pkg/auditlog.TestLogFailedQuery.func2.1
github.com/cockroachlabs/managed-service/pkg/auditlog.TestRecordHttpRequest
github.com/cockroachlabs/managed-service/pkg/auditlog.TestRecordHttpRespAndErr
github.com/cockroachlabs/managed-service/pkg/auditlog.TestRecordRequestAndRecordRespAndErr
github.com/cockroachlabs/managed-service/pkg/auditlog.TestScrubNestedField
github.com/cockroachlabs/managed-service/pkg/auditlog.TestScrubSensitiveFields
(dlv) break TestLogFailedQuery
Breakpoint 1 set at 0x1ce06bb for github.com/cockroachlabs/managed-service/pkg/auditlog.TestLogFailedQuery() ./middleware_test.go:83
(dlv) c
> github.com/cockroachlabs/managed-service/pkg/auditlog.TestLogFailedQuery() ./middleware_test.go:83 (hits goroutine(78):1 total:1) (PC: 0x1ce06bb)
78: require.Equal(t, nulls.NewString("{}"), entry.Response) // default is null so this confirms the second write succeeded
79: require.Equal(t, nulls.NewString(""), entry.Error)
80: })
81: }
82:
=> 83: func TestLogFailedQuery(t *testing.T) {
84: conn, err := pop.Connect("test")
85: require.NoError(t, err)
86:
87: // Simulate authenticated request
88: om, cluster := models.CreateRandomOrgUserAndCluster(t, conn)
(dlv) |
Getting a Linux Delve binary onto a roachprod node
The instructions on the delve repo require go to be installed, which is something that the roachprod nodes do not have. The easiest way is to build the delve binary on your development machine. Skip the cloning if you already have the repo:
...