What is the format of Android's logs?

Question

I'm trying to gather data about my phone by analyzing the log files in /dev/log. I'm specifically looking at /dev/log/main. I always thought that any sane log format would be plain text, yet these files appear to be either binary or in some character set that neither I nor my Linux text editors can identify.

What is the format?

Here are a couple of screenshots:

• First, here's a snippet of the log as interpreted by vim (^@ refers to the null byte; I'm not sure about the other colored control sequences):

• Next, this is what the log looks like in a hex editor:

I'm using a Galaxy Nexus running Jelly Bean. The logs were collected using root and a terminal emulator, since aLogcat doesn't seem to use root and thus can't access all logging info.

Answer 1

If you want sane information, I recommend sane commands :) (no offense meant, just kidding). So the question should read:

How to obtain log information from an Android device?

And now we are on the better side. There are multiple approaches which can be used:

• utilize apps to display (color-coded) log information
• utilize ADB (part of the Android SDK) to remotely extract the very same information
• use ssh from remote (or a local terminal app) to aquire the information directly from the device

To fully handle this topic it takes more than this simple answer (if interested, you can e.g. find more detailed information on many web sites, or in Andrew Hoog's book Android Forensics: Investigation, Analysis and Mobile Security for Google Android, which I had the honour to translate into German. There are probably many other sources as well.

So I will just give a few examples here to get you started:

Utilizing apps

The probably best known app in this context is aLogcat, available for free in the playstore (and the dev will happily accept your donation for the other variant of the same app). You'll find a screenshot below¹. The app allows you to filter the logs, to start/stop recording log messages, and even to store the recorded snippets to your SD-Card -- of course in plain text, as you requested.

Another app in this section is Log Collector, which simply tries to grab the entire available log and send it via the share menu².

The Android Debug Bridge (ADB)

The Android Software Development Kit (SDK) includes the adb command for various tasks. Amongst many others, it offers the adb shell to execute commands on the device. Using this, you can gather your desired log information as well: Just prefix below commands with adb shell.

Command prompt on the device

Using a terminal app (e.g. Android Terminal Emulator or Terminal IDE) you can access the logs directly at the command prompt, locally on your device. A little more comfortable, this can be done running a ssh server (e.g. DroidSSHd or DropBear SSH Server) on your device, and access it from your computer. This way you can work on a big screen, while investigating your logs.

Commands to access your log information

There are a lot of powerful commands you can use to access your log information from the command line, and I will only give a few examples here.

dmesg

The dmesg command extracts the kernel log:

$ dmesg
<6>[82839.126586] PM: Syncing filesystems ... done.
<7>[82839.189056] PM: Preparing system for mem sleep
<4>[82839.189361] Freezing user space processes ... (elapsed 0.05 seconds) done.
<4>[82839.240661] Freezing remaining freezable tasks ... (elapsed 0.00 seconds) done.
<7>[82839.242279] PM: Entering mem sleep
<4>[82839.242889] Suspending console(s) (use no_console_suspend to debug)
<7>[82839.252410] vfp_pm_save_context: saving vfp state
<6>[82839.252716] PM: Resume timer in 26 secs (864747 ticks at 32768 ticks/sec.)
<6>[82842.091369] Successfully put all powerdomains to target state
<6>[82842.092468] wakeup wake lock: wifi_wake

logcat

With logcat, you can access many logging information -- but most times, this will require root. It has some parameters to filter the information, e.g. by selecting the log buffer to read with -b. Please read the information provided on the developers page on logcat for details. To give you two examples: logcat -b events would list up events, or logcat -b radio information on your device's radio modul.

dumpsys and dumpstate

The two commands dumpsys and dumpstate give you detailed system information:

$ dumpsys
Currently running services:
    LocationProxyService
    SurfaceFlinger
    accessibility
    account
    activity
<snip>
DUMP OF SERVICE account:
Accounts: 1
    Account {[email protected], type=com.google}
<snip>
DUMP OF SERVICE alarm:

$ dumpstate
========================================================
== dumpstate: 2012-08-18 23:39:53
========================================================
Build: Gingerbread GWK74 - CyanogenMilestone2
Bootloader: 0x0000
Radio: unknown
<snip>
------ MEMORY INFO (/proc/meminfo) ------
MemTotal: 487344 kB
MemFree:   10436 kB
Buffers:   14136 kB
Cached:    145460 kB
<snip>

bugreport

And if you are too lazy to remember them all, simply use the bugreport command -- which calls all above and bundles it for a nice, humm, bug report to the developer...

Of course, you can redirect the output from all those commands to a file to copy to your computer, and in most cases you should do so -- as your screen buffer would be far too small to handle it all: bugreport > /mnt/sdcard/bugreport.txt would be one example for that part.

Answer 2

For developers (or other interested parties) who need to parse this raw file, here are some resources:

• logcat (the program that can output binary logs), it calls android_logger_list_read() to gather log entries: https://android.googlesource.com/platform/system/core/+/master/logcat/logcat.cpp
• Information about Android Logging in general: http://elinux.org/Android_Logging_System
• The low-level logging library for reading/writing logs: https://android.googlesource.com/platform/system/core/+/master/liblog
• An utility to dump raw (binary) logs, optionally switching versions: https://lekensteyn.nl/files/logread.c
• Kernel source for logger: https://github.com/android/kernel_common/blob/android-3.10/drivers/staging/android/logger.c

The actual format of the log format is detailled at:

• https://android.googlesource.com/platform/system/core/+/master/include/log/logger.h

A copy of the relevant parts, slightly annotated and re-ordered for your convenience:

#define LOGGER_ENTRY_MAX_LEN (5*1024)

struct log_msg {
    union {
        /* Maximum size of entry: 5121 bytes */
        unsigned char buf[LOGGER_ENTRY_MAX_LEN + 1];

        struct logger_entry_v3 entry;
        struct logger_entry_v3 entry_v3;
        struct logger_entry_v2 entry_v2;
        struct logger_entry entry_v1;
    } __attribute__((aligned(4)));
}
/*
 * The userspace structure for version 1 of the logger_entry ABI.
 * This structure is returned to userspace by the kernel logger
 * driver unless an upgrade to a newer ABI version is requested.
 */
struct logger_entry {
    uint16_t    len;    /* length of the payload */
    uint16_t    __pad;  /* no matter what, we get 2 bytes of padding */
    int32_t     pid;    /* generating process's pid */
    int32_t     tid;    /* generating process's tid */
    int32_t     sec;    /* seconds since Epoch */
    int32_t     nsec;   /* nanoseconds */
    char        msg[0]; /* the entry's payload */
} __attribute__((__packed__));

/*
 * The userspace structure for version 2 of the logger_entry ABI.
 * This structure is returned to userspace if ioctl(LOGGER_SET_VERSION)
 * is called with version==2; or used with the user space log daemon.
 */
struct logger_entry_v2 {
    uint16_t    len;    /* length of the payload */
    uint16_t    hdr_size; /* sizeof(struct logger_entry_v2) */
    int32_t     pid;    /* generating process's pid */
    int32_t     tid;    /* generating process's tid */
    int32_t     sec;    /* seconds since Epoch */
    int32_t     nsec;   /* nanoseconds */
    uint32_t    euid;   /* effective UID of logger */
    char        msg[0]; /* the entry's payload */
} __attribute__((__packed__));

struct logger_entry_v3 {
    uint16_t    len;    /* length of the payload */
    uint16_t    hdr_size; /* sizeof(struct logger_entry_v3) */
    int32_t     pid;    /* generating process's pid */
    int32_t     tid;    /* generating process's tid */
    int32_t     sec;    /* seconds since Epoch */
    int32_t     nsec;   /* nanoseconds */
    uint32_t    lid;    /* log id of the payload */
    char        msg[0]; /* the entry's payload */
} __attribute__((__packed__));

You can tell the differences versions apart by looking at the third and four byte. The format is apparently also dependent on the endianness of your platform. For v1 messages, __pad equals to 0. v2 (and v3) messages use 24 (0x18).

For main, radio and system logs msg field is interpreted as follows (source):

• priority: 1 byte
• tag: 0 or more bytes
• literal \0 as separator
• message: 0 or more bytes
• literal \0 as terminator

If this message is truncated, the trailing \0 may be missing.

For events log however, msg field contain the following binary data:

• Tag: 4 bytes integer key from "/system/etc/event-log-tags" file.
• Message: serialized tree starting from the root node. Each tree node starts with a 1-byte value from AndroidEventLogType enumeration:
  • EVENT_TYPE_INT - node value is 4 bytes integer
  • EVENT_TYPE_LONG - node value is 8 bytes integer
  • EVENT_TYPE_STRING - node value is 4 bytes integer length, followed by the length bytes containing a UTF8-encoded string
  • EVENT_TYPE_LIST - node value is a single-byte length, followed by the length tree nodes each of its own AndroidEventLogType.