Debugging Go Code with GDB

This applies to the gc toolchain. Gccgo has native gdb support. Besides this overview you might want to consult the GDB manual.

GDB does not understand Go programs well. The stack management, threading, and runtime contain aspects that differ enough from the execution model GDB expects that they can confuse the debugger, even when the program is compiled with gccgo. As a consequence, although GDB can be useful in some situations, it is not a reliable debugger for Go programs, particularly heavily concurrent ones. Moreover, it is not a priority for the Go project to address these issues, which are difficult. In short, the instructions below should be taken only as a guide to how to use GDB when it works, not as a guarantee of success.

In time, a more Go-centric debugging architecture may be required.

Introduction

When you compile and link your Go programs with the gc toolchain on Linux, Mac OS X, FreeBSD or NetBSD, the resulting binaries contain DWARFv3 debugging information that recent versions (>7.1) of the GDB debugger can use to inspect a live process or a core dump.

Pass the '-w' flag to the linker to omit the debug information (for example, go build -ldflags "-w" prog.go).

The code generated by the gc compiler includes inlining of function invocations and registerization of variables. These optimizations can sometimes make debugging with gdb harder. To disable them when debugging, pass the flags -gcflags "-N -l" to the go command used to build the code being debugged.

Common Operations

Show file and line number for code, set breakpoints and disassemble:

(gdb) list
(gdb) list line
(gdb) list file.go:line
(gdb) break line
(gdb) break file.go:line
(gdb) disas

Show backtraces and unwind stack frames:
```
(gdb) bt
(gdb) frame n
```
Show the name, type and location on the stack frame of local variables, arguments and return values:
```
(gdb) info locals
(gdb) info args
(gdb) p variable
(gdb) whatis variable
```
Show the name, type and location of global variables:
```
(gdb) info variables regexp
```

Go Extensions

A recent extension mechanism to GDB allows it to load extension scripts for a given binary. The tool chain uses this to extend GDB with a handful of commands to inspect internals of the runtime code (such as goroutines) and to pretty print the built-in map, slice and channel types.

Pretty printing a string, slice, map, channel or interface:
```
(gdb) p var
```
A $len() and $cap() function for strings, slices and maps:
```
(gdb) p $len(var)
```
A function to cast interfaces to their dynamic types:
```
(gdb) p $dtype(var)
(gdb) iface var
```
Known issue: GDB can’t automatically find the dynamic type of an interface value if its long name differs from its short name (annoying when printing stacktraces, the pretty printer falls back to printing the short type name and a pointer).

Inspecting goroutines:

(gdb) info goroutines
(gdb) goroutine n cmd
(gdb) help goroutine

For example:

(gdb) goroutine 12 bt

If you'd like to see how this works, or want to extend it, take a look at src/runtime/runtime-gdb.py in the Go source distribution. It depends on some special magic types (hash<T,U>) and variables (runtime.m and runtime.g) that the linker (src/cmd/link/internal/ld/dwarf.go) ensures are described in the DWARF code.

If you're interested in what the debugging information looks like, run 'objdump -W 6.out' and browse through the .debug_* sections.

Known Issues

String pretty printing only triggers for type string, not for types derived from it.
Type information is missing for the C parts of the runtime library.
GDB does not understand Go’s name qualifications and treats "fmt.Print" as an unstructured literal with a "." that needs to be quoted. It objects even more strongly to method names of the form pkg.(*MyType).Meth.
All global variables are lumped into package "main".

Tutorial

In this tutorial we will inspect the binary of the regexp package's unit tests. To build the binary, change to $GOROOT/src/regexp and run go test -c. This should produce an executable file named regexp.test.

Getting Started

Launch GDB, debugging regexp.test:

$ gdb regexp.test
GNU gdb (GDB) 7.2-gg8
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv  3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
Type "show copying" and "show warranty" for licensing/warranty details.
This GDB was configured as "x86_64-linux".

Reading symbols from  /home/user/go/src/regexp/regexp.test...
done.
Loading Go Runtime support.
(gdb)

The message "Loading Go Runtime support" means that GDB loaded the extension from $GOROOT/src/runtime/runtime-gdb.py.

To help GDB find the Go runtime sources and the accompanying support script, pass your $GOROOT with the '-d' flag:

$ gdb regexp.test -d $GOROOT

If for some reason GDB still can't find that directory or that script, you can load it by hand by telling gdb (assuming you have the go sources in ~/go/):

(gdb) source ~/go/src/runtime/runtime-gdb.py
Loading Go Runtime support.

Inspecting the source

Use the "l" or "list" command to inspect source code.

(gdb) l

List a specific part of the source parametrizing "list" with a function name (it must be qualified with its package name).

(gdb) l main.main

List a specific file and line number:

(gdb) l regexp.go:1
(gdb) # Hit enter to repeat last command. Here, this lists next 10 lines.

Naming

Variable and function names must be qualified with the name of the packages they belong to. The Compile function from the regexp package is known to GDB as 'regexp.Compile'.

Methods must be qualified with the name of their receiver types. For example, the *Regexp type’s String method is known as 'regexp.(*Regexp).String'.

Variables that shadow other variables are magically suffixed with a number in the debug info. Variables referenced by closures will appear as pointers magically prefixed with '&'.

Setting breakpoints

Set a breakpoint at the TestFind function:

(gdb) b 'regexp.TestFind'
Breakpoint 1 at 0x424908: file /home/user/go/src/regexp/find_test.go, line 148.

Run the program:

(gdb) run
Starting program: /home/user/go/src/regexp/regexp.test

Breakpoint 1, regexp.TestFind (t=0xf8404a89c0) at /home/user/go/src/regexp/find_test.go:148
148	func TestFind(t *testing.T) {

Execution has paused at the breakpoint. See which goroutines are running, and what they're doing:

(gdb) info goroutines
  1  waiting runtime.gosched
* 13  running runtime.goexit

the one marked with the * is the current goroutine.

Inspecting the stack

Look at the stack trace for where we’ve paused the program:

(gdb) bt  # backtrace
#0  regexp.TestFind (t=0xf8404a89c0) at /home/user/go/src/regexp/find_test.go:148
#1  0x000000000042f60b in testing.tRunner (t=0xf8404a89c0, test=0x573720) at /home/user/go/src/testing/testing.go:156
#2  0x000000000040df64 in runtime.initdone () at /home/user/go/src/runtime/proc.c:242
#3  0x000000f8404a89c0 in ?? ()
#4  0x0000000000573720 in ?? ()
#5  0x0000000000000000 in ?? ()

The other goroutine, number 1, is stuck in runtime.gosched, blocked on a channel receive:

(gdb) goroutine 1 bt
#0  0x000000000040facb in runtime.gosched () at /home/user/go/src/runtime/proc.c:873
#1  0x00000000004031c9 in runtime.chanrecv (c=void, ep=void, selected=void, received=void)
 at  /home/user/go/src/runtime/chan.c:342
#2  0x0000000000403299 in runtime.chanrecv1 (t=void, c=void) at/home/user/go/src/runtime/chan.c:423
#3  0x000000000043075b in testing.RunTests (matchString={void (struct string, struct string, bool *, error *)}
 0x7ffff7f9ef60, tests=  []testing.InternalTest = {...}) at /home/user/go/src/testing/testing.go:201
#4  0x00000000004302b1 in testing.Main (matchString={void (struct string, struct string, bool *, error *)} 
 0x7ffff7f9ef80, tests= []testing.InternalTest = {...}, benchmarks= []testing.InternalBenchmark = {...})
at /home/user/go/src/testing/testing.go:168
#5  0x0000000000400dc1 in main.main () at /home/user/go/src/regexp/_testmain.go:98
#6  0x00000000004022e7 in runtime.mainstart () at /home/user/go/src/runtime/amd64/asm.s:78
#7  0x000000000040ea6f in runtime.initdone () at /home/user/go/src/runtime/proc.c:243
#8  0x0000000000000000 in ?? ()

The stack frame shows we’re currently executing the regexp.TestFind function, as expected.

(gdb) info frame
Stack level 0, frame at 0x7ffff7f9ff88:
 rip = 0x425530 in regexp.TestFind (/home/user/go/src/regexp/find_test.go:148); 
    saved rip 0x430233
 called by frame at 0x7ffff7f9ffa8
 source language minimal.
 Arglist at 0x7ffff7f9ff78, args: t=0xf840688b60
 Locals at 0x7ffff7f9ff78, Previous frame's sp is 0x7ffff7f9ff88
 Saved registers:
  rip at 0x7ffff7f9ff80

The command info locals lists all variables local to the function and their values, but is a bit dangerous to use, since it will also try to print uninitialized variables. Uninitialized slices may cause gdb to try to print arbitrary large arrays.

The function’s arguments:

(gdb) info args
t = 0xf840688b60

When printing the argument, notice that it’s a pointer to a Regexp value. Note that GDB has incorrectly put the * on the right-hand side of the type name and made up a 'struct' keyword, in traditional C style.

(gdb) p re
(gdb) p t
$1 = (struct testing.T *) 0xf840688b60
(gdb) p t
$1 = (struct testing.T *) 0xf840688b60
(gdb) p *t
$2 = {errors = "", failed = false, ch = 0xf8406f5690}
(gdb) p *t->ch
$3 = struct hchan<*testing.T>

That struct hchan<*testing.T> is the runtime-internal representation of a channel. It is currently empty, or gdb would have pretty-printed it's contents.

Stepping forward:

(gdb) n  # execute next line
149             for _, test := range findTests {
(gdb)    # enter is repeat
150                     re := MustCompile(test.pat)
(gdb) p test.pat
$4 = ""
(gdb) p re
$5 = (struct regexp.Regexp *) 0xf84068d070
(gdb) p *re
$6 = {expr = "", prog = 0xf840688b80, prefix = "", prefixBytes =  []uint8, prefixComplete = true, 
  prefixRune = 0, cond = 0 '\000', numSubexp = 0, longest = false, mu = {state = 0, sema = 0}, 
  machine =  []*regexp.machine}
(gdb) p *re->prog
$7 = {Inst =  []regexp/syntax.Inst = {{Op = 5 '\005', Out = 0, Arg = 0, Rune =  []int}, {Op = 
    6 '\006', Out = 2, Arg = 0, Rune =  []int}, {Op = 4 '\004', Out = 0, Arg = 0, Rune =  []int}}, 
  Start = 1, NumCap = 2}

We can step into the Stringfunction call with "s":

(gdb) s
regexp.(*Regexp).String (re=0xf84068d070, noname=void) at /home/user/go/src/regexp/regexp.go:97
97      func (re *Regexp) String() string {

Get a stack trace to see where we are:

(gdb) bt
#0  regexp.(*Regexp).String (re=0xf84068d070, noname=void)
    at /home/user/go/src/regexp/regexp.go:97
#1  0x0000000000425615 in regexp.TestFind (t=0xf840688b60)
    at /home/user/go/src/regexp/find_test.go:151
#2  0x0000000000430233 in testing.tRunner (t=0xf840688b60, test=0x5747b8)
    at /home/user/go/src/testing/testing.go:156
#3  0x000000000040ea6f in runtime.initdone () at /home/user/go/src/runtime/proc.c:243
....

Look at the source code:

(gdb) l
92              mu      sync.Mutex
93              machine []*machine
94      }
95
96      // String returns the source text used to compile the regular expression.
97      func (re *Regexp) String() string {
98              return re.expr
99      }
100
101     // Compile parses a regular expression and returns, if successful,

Pretty Printing

GDB's pretty printing mechanism is triggered by regexp matches on type names. An example for slices:

(gdb) p utf
$22 =  []uint8 = {0 '\000', 0 '\000', 0 '\000', 0 '\000'}

Since slices, arrays and strings are not C pointers, GDB can't interpret the subscripting operation for you, but you can look inside the runtime representation to do that (tab completion helps here):


(gdb) p slc
$11 =  []int = {0, 0}
(gdb) p slc-><TAB>
array  slc    len    
(gdb) p slc->array
$12 = (int *) 0xf84057af00
(gdb) p slc->array[1]
$13 = 0

The extension functions $len and $cap work on strings, arrays and slices:

(gdb) p $len(utf)
$23 = 4
(gdb) p $cap(utf)
$24 = 4

Channels and maps are 'reference' types, which gdb shows as pointers to C++-like types hash<int,string>*. Dereferencing will trigger prettyprinting

Interfaces are represented in the runtime as a pointer to a type descriptor and a pointer to a value. The Go GDB runtime extension decodes this and automatically triggers pretty printing for the runtime type. The extension function $dtype decodes the dynamic type for you (examples are taken from a breakpoint at regexp.go line 293.)

(gdb) p i
$4 = {str = "cbb"}
(gdb) whatis i
type = regexp.input
(gdb) p $dtype(i)
$26 = (struct regexp.inputBytes *) 0xf8400b4930
(gdb) iface i
regexp.input: struct regexp.inputBytes *