This page contains a summary of useful tools and related commands for debugging, tracing, and profiling native Android platform code. The pages within this section contain detailed information on other debugging tools for use during development of platform-level features.
For example, you may learn how to explore system services with Dumpsys and evaluate network and RAM use. See the subpages for tools and methods not described below.
When a dynamically-linked executable starts, several signal handlers are
registered that connect to debuggerd (or debuggerd64) in the event that signal
is sent to the process. The debuggerd process dumps registers and unwinds the
stack.
It's possible for debuggerd to attach only if nothing else is
already attached. This means that using tools like strace or
gdb will prevent debuggerd from working. Also, if
you call prctl(PR_SET_DUMPABLE, 0) you can prevent
debuggerd from attaching. This can be useful if you wish to
explicitly opt out of crash reporting.
Here is example output (with timestamps and extraneous information removed):
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'Android/aosp_flounder/flounder:5.1.51/AOSP/enh08201009:eng/test-keys'
Revision: '0'
ABI: 'arm'
pid: 1656, tid: 1656, name: crasher >>> crasher <<<
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
Abort message: 'some_file.c:123: some_function: assertion "false" failed'
r0 00000000 r1 00000678 r2 00000006 r3 f70b6dc8
r4 f70b6dd0 r5 f70b6d80 r6 00000002 r7 0000010c
r8 ffffffed r9 00000000 sl 00000000 fp ff96ae1c
ip 00000006 sp ff96ad18 lr f700ced5 pc f700dc98 cpsr 400b0010
backtrace:
#00 pc 00042c98 /system/lib/libc.so (tgkill+12)
#01 pc 00041ed1 /system/lib/libc.so (pthread_kill+32)
#02 pc 0001bb87 /system/lib/libc.so (raise+10)
#03 pc 00018cad /system/lib/libc.so (__libc_android_abort+34)
#04 pc 000168e8 /system/lib/libc.so (abort+4)
#05 pc 0001a78f /system/lib/libc.so (__libc_fatal+16)
#06 pc 00018d35 /system/lib/libc.so (__assert2+20)
#07 pc 00000f21 /system/xbin/crasher
#08 pc 00016795 /system/lib/libc.so (__libc_init+44)
#09 pc 00000abc /system/xbin/crasher
Tombstone written to: /data/tombstones/tombstone_06
This can be pasted into development/scripts/stack to get a more detailed unwind
with line number information (assuming the unstripped binaries can be found).
Some libraries on the system are built with LOCAL_STRIP_MODULE :=
keep_symbols to provide usable backtraces directly from debuggerd. This makes
your library or executable slightly larger, but not nearly as large as an
unstripped version.
Note also the last line of debuggerd output --- in addition to dumping a
summary to the log, debuggerd writes a full “tombstone� to disk. This contains
a lot of extra information that can be helpful in debugging a crash, in
particular the stack traces for all the threads in the crashing process (not
just the thread that caught the signal) and a full memory map.
If you don't have a specific crash that you're investigating right now,
the platform source includes a tool for testing debuggerd called crasher. If
you mm in system/core/debuggerd/ you'll get both a crasher
and a crasher64 on your path (the latter allowing you to test
64-bit crashes). Crasher can crash in a large number of interesting ways based
on the command line arguments you provide. Use crasher --help
to see the currently supported selection.
To introduce the difference pieces in a crash dump, let's work through the example above:
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
The line of asterisks with spaces is helpful if you're searching a log for native crashes. The string "*** ***" rarely shows up in logs other than at the beginning of a native crash.
Build fingerprint: 'Android/aosp_flounder/flounder:5.1.51/AOSP/enh08201009:eng/test-keys'
The fingerprint lets you identify exactly which build the crash occurred
on. This is exactly the same as the ro.build.fingerprint system property.
Revision: '0'
The revision refers to the hardware rather than the software. This is
usually unused but can be useful to help you automatically ignore bugs known
to be caused by bad hardware. This is exactly the same as the ro.revision
system property.
ABI: 'arm'
The ABI is one of arm, arm64, mips, mips64, x86, or x86-64. This is
mostly useful for the stack script mentioned above, so that it knows
what toolchain to use.
pid: 1656, tid: 1656, name: crasher >>> crasher <<<
This line identifies the specific thread in the process that crashed. In this case, it was the process' main thread, so the process ID and thread ID match. The first name is the thread name, and the name surrounded by >>> and <<< is the process name. For an app, the process name is typically the fully-qualified package name (such as com.facebook.katana), which is useful when filing bugs or trying to find the app in Google Play. The pid and tid can also be useful in finding the relevant log lines preceding the crash.
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
This line tells you which signal (SIGABRT) was received, and more about
how it was received (SI_TKILL). The signals reported by debuggerd are SIGABRT,
SIGBUS, SIGFPE, SIGILL, SIGSEGV, and SIGTRAP. The signal-specific codes vary
based on the specific signal.
Abort message: 'some_file.c:123: some_function: assertion "false" failed'
Not all crashes will have an abort message line, but aborts will. This is automatically gathered from the last line of fatal logcat output for this pid/tid, and in the case of a deliberate abort is likely to give an explanation of why the program killed itself.
r0 00000000 r1 00000678 r2 00000006 r3 f70b6dc8 r4 f70b6dd0 r5 f70b6d80 r6 00000002 r7 0000010c r8 ffffffed r9 00000000 sl 00000000 fp ff96ae1c ip 00000006 sp ff96ad18 lr f700ced5 pc f700dc98 cpsr 400b0010
The register dump shows the content of the CPU registers at the time the signal was received. (This section varies wildly between ABIs.) How useful these are will depend on the exact crash.
backtrace:
#00 pc 00042c98 /system/lib/libc.so (tgkill+12)
#01 pc 00041ed1 /system/lib/libc.so (pthread_kill+32)
#02 pc 0001bb87 /system/lib/libc.so (raise+10)
#03 pc 00018cad /system/lib/libc.so (__libc_android_abort+34)
#04 pc 000168e8 /system/lib/libc.so (abort+4)
#05 pc 0001a78f /system/lib/libc.so (__libc_fatal+16)
#06 pc 00018d35 /system/lib/libc.so (__assert2+20)
#07 pc 00000f21 /system/xbin/crasher
#08 pc 00016795 /system/lib/libc.so (__libc_init+44)
#09 pc 00000abc /system/xbin/crasher
The backtrace shows you where in the code we were at the time of
crash. The first column is the frame number (matching gdb's style where
the deepest frame is 0). The PC values are relative to the location of the
shared library rather than absolute addresses. The next column is the name
of the mapped region (which is usually a shared library or executable, but
might not be for, say, JIT-compiled code). Finally, if symbols are available,
the symbol that the PC value corresponds to is shown, along with the offset
into that symbol in bytes. You can use this in conjunction with objdump(1)
to find the corresponding assembler instruction.
Tombstone written to: /data/tombstones/tombstone_06
This tells you where debuggerd wrote extra information.
debuggerd will keep up to 10 tombstones, cycling through
the numbers 00 to 09 and overwriting existing tombstones as necessary.
The tombstone contains the same information as the crash dump, plus a
few extras. For example, it includes backtraces for all threads (not
just the crashing thread), the floating point registers, raw stack dumps,
and memory dumps around the addresses in registers. Most usefully it also
includes a full memory map (similar to /proc/pid/maps). Here's an
annotated example from a 32-bit ARM process crash:
memory map: (fault address prefixed with --->) --->ab15f000-ab162fff r-x 0 4000 /system/xbin/crasher (BuildId: b9527db01b5cf8f5402f899f64b9b121)
There are two things to note here. The first is that this line is prefixed with "--->". The maps are most useful when your crash isn't just a null pointer dereference. If the fault address is small, it's probably some variant of a null pointer dereference. Otherwise looking at the maps around the fault address can often give you a clue as to what happened. Some possible issues that can be recognized by looking at the maps include:
The second thing to note is that executables and shared libraries files
will show the BuildId (if present) in Android M and later, so you can see
exactly which version of your code crashed. (Platform binaries include a
BuildId by default since Android M. NDK r12 and later automatically pass
-Wl,--build-id to the linker too.)
ab163000-ab163fff r-- 3000 1000 /system/xbin/crasher ab164000-ab164fff rw- 0 1000 f6c80000-f6d7ffff rw- 0 100000 [anon:libc_malloc]
On Android the heap isn't necessarily a single region. Heap regions will
be labeled [anon:libc_malloc].
f6d82000-f6da1fff r-- 0 20000 /dev/__properties__/u:object_r:logd_prop:s0 f6da2000-f6dc1fff r-- 0 20000 /dev/__properties__/u:object_r:default_prop:s0 f6dc2000-f6de1fff r-- 0 20000 /dev/__properties__/u:object_r:logd_prop:s0 f6de2000-f6de5fff r-x 0 4000 /system/lib/libnetd_client.so (BuildId: 08020aa06ed48cf9f6971861abf06c9d) f6de6000-f6de6fff r-- 3000 1000 /system/lib/libnetd_client.so f6de7000-f6de7fff rw- 4000 1000 /system/lib/libnetd_client.so f6dec000-f6e74fff r-x 0 89000 /system/lib/libc++.so (BuildId: 8f1f2be4b37d7067d366543fafececa2) (load base 0x2000) f6e75000-f6e75fff --- 0 1000 f6e76000-f6e79fff r-- 89000 4000 /system/lib/libc++.so f6e7a000-f6e7afff rw- 8d000 1000 /system/lib/libc++.so f6e7b000-f6e7bfff rw- 0 1000 [anon:.bss] f6e7c000-f6efdfff r-x 0 82000 /system/lib/libc.so (BuildId: d189b369d1aafe11feb7014d411bb9c3) f6efe000-f6f01fff r-- 81000 4000 /system/lib/libc.so f6f02000-f6f03fff rw- 85000 2000 /system/lib/libc.so f6f04000-f6f04fff rw- 0 1000 [anon:.bss] f6f05000-f6f05fff r-- 0 1000 [anon:.bss] f6f06000-f6f0bfff rw- 0 6000 [anon:.bss] f6f0c000-f6f21fff r-x 0 16000 /system/lib/libcutils.so (BuildId: d6d68a419dadd645ca852cd339f89741) f6f22000-f6f22fff r-- 15000 1000 /system/lib/libcutils.so f6f23000-f6f23fff rw- 16000 1000 /system/lib/libcutils.so f6f24000-f6f31fff r-x 0 e000 /system/lib/liblog.so (BuildId: e4d30918d1b1028a1ba23d2ab72536fc) f6f32000-f6f32fff r-- d000 1000 /system/lib/liblog.so f6f33000-f6f33fff rw- e000 1000 /system/lib/liblog.so
Typically a shared library will have three adjacent entries. One will be
readable and executable (code), one will be read-only (read-only
data), and one will be read-write (mutable data). The first column
shows the address ranges for the mapping, the second column the permissions
(in the usual Unix ls(1) style), the third column the offset into the file
(in hex), the fourth column the size of the region (in hex), and the fifth
column the file (or other region name).
f6f34000-f6f53fff r-x 0 20000 /system/lib/libm.so (BuildId: 76ba45dcd9247e60227200976a02c69b) f6f54000-f6f54fff --- 0 1000 f6f55000-f6f55fff r-- 20000 1000 /system/lib/libm.so f6f56000-f6f56fff rw- 21000 1000 /system/lib/libm.so f6f58000-f6f58fff rw- 0 1000 f6f59000-f6f78fff r-- 0 20000 /dev/__properties__/u:object_r:default_prop:s0 f6f79000-f6f98fff r-- 0 20000 /dev/__properties__/properties_serial f6f99000-f6f99fff rw- 0 1000 [anon:linker_alloc_vector] f6f9a000-f6f9afff r-- 0 1000 [anon:atexit handlers] f6f9b000-f6fbafff r-- 0 20000 /dev/__properties__/properties_serial f6fbb000-f6fbbfff rw- 0 1000 [anon:linker_alloc_vector] f6fbc000-f6fbcfff rw- 0 1000 [anon:linker_alloc_small_objects] f6fbd000-f6fbdfff rw- 0 1000 [anon:linker_alloc_vector] f6fbe000-f6fbffff rw- 0 2000 [anon:linker_alloc] f6fc0000-f6fc0fff r-- 0 1000 [anon:linker_alloc] f6fc1000-f6fc1fff rw- 0 1000 [anon:linker_alloc_lob] f6fc2000-f6fc2fff r-- 0 1000 [anon:linker_alloc] f6fc3000-f6fc3fff rw- 0 1000 [anon:linker_alloc_vector] f6fc4000-f6fc4fff rw- 0 1000 [anon:linker_alloc_small_objects] f6fc5000-f6fc5fff rw- 0 1000 [anon:linker_alloc_vector] f6fc6000-f6fc6fff rw- 0 1000 [anon:linker_alloc_small_objects] f6fc7000-f6fc7fff rw- 0 1000 [anon:arc4random _rsx structure] f6fc8000-f6fc8fff rw- 0 1000 [anon:arc4random _rs structure] f6fc9000-f6fc9fff r-- 0 1000 [anon:atexit handlers] f6fca000-f6fcafff --- 0 1000 [anon:thread signal stack guard page]
Note that since Android 5.0 (Lollipop), the C library names most of its anonymous mapped regions so there are fewer mystery regions.
f6fcb000-f6fccfff rw- 0 2000 [stack:5081]
Regions named [stack:tid] are the stacks for the given threads.
f6fcd000-f702afff r-x 0 5e000 /system/bin/linker (BuildId: 84f1316198deee0591c8ac7f158f28b7) f702b000-f702cfff r-- 5d000 2000 /system/bin/linker f702d000-f702dfff rw- 5f000 1000 /system/bin/linker f702e000-f702ffff rw- 0 2000 f7030000-f7030fff r-- 0 1000 f7031000-f7032fff rw- 0 2000 ffcd7000-ffcf7fff rw- 0 21000 ffff0000-ffff0fff r-x 0 1000 [vectors]
Whether you see [vector] or [vdso] depends on the architecture. ARM uses [vector], while all other architectures use [vdso].
To connect to an already-running app or native daemon, use gdbclient.
Current versions of gdbclient just require the process ID (PID). So to debug a process with PID 1234, simply run:
$ gdbclient 1234
The script will set up port forwarding, start the appropriate
gdbserver on the device, start the appropriate gdb on
the host, configure gdb to find symbols, and connect
gdb to the remote gdbserver.
If you want to debug a process as it starts, you’ll need to use gdbserver
or gdbserver64 manually, but that’s easy too:
$ adb shell gdbserver :5039 /system/bin/my_test_app Process my_test_app created; pid = 3460 Listening on port 5039
Identify the app’s PID from the gdbserver output, and then in
another window:
$ gdbclient <app pid>
Then enter continue at the gdb prompt.
Note that to debug a 64-bit process, you'll need to use gdbserver64.
The error messages from gdb if you made the wrong choice are unhelpful
(along the lines of Reply contains invalid hex digit 59).
If you want debuggerd to suspend crashed processes so you can
attach gdb, set the appropriate property:
$ adb shell setprop debug.db.uid 999999 # <= M $ adb shell setprop debug.debuggerd.wait_for_gdb true # > M
At the end of the usual crash output, debuggerd will give you
instructions on how to connect gdb using the typical command:
$ gdbclient <pid>
If you don’t have symbols, sometimes gdb will get confused about the
instruction set it is disassembling (ARM or Thumb). The instruction set that is
chosen as the default when symbol information is missing can be switched
between ARM or Thumb like so:
$ set arm fallback-mode arm # or 'thumb'
The following steps show you how to use Valgrind on Android. This tool suite contains a number of tools including Memcheck for detecting memory-related errors in C and C++.
$ mmma -j6 external/valgrind
$ adb shell mkdir /data/local/tmp $ adb shell chmod 777 /data/local/tmp
$ adb shell setprop wrap.system_server "logwrapper valgrind" $ adb shell stop && adb shell start
/data/local/symbols:
$ adb shell mkdir /data/local/symbols $ adb push $OUT/symbols /data/local/symbols
out/target/product/XXXX/root/init.rc and
change:service example /system/bin/foo --arg1 --arg2service example /system/bin/logwrapper /system/bin/valgrind /system/bin/foo --arg1 --arg2boot.img and reflash the device.
See Systrace on developer.android.com for deriving execution times of applications and other Android system processes.