一文詳解Linux內(nèi)存檢測(cè)技術(shù)

Linux常見的內(nèi)存訪問錯(cuò)誤有：

越界訪問(out of bounds)

訪問已經(jīng)釋放的內(nèi)存(use after free)

重復(fù)釋放

內(nèi)存泄露(memory leak)

棧溢出(stack overflow)

不同的工具有不同的側(cè)重點(diǎn)，本章主要從slub_debug、kmemleak、kasan三個(gè)工具介紹。

kmemleak側(cè)重于內(nèi)存泄露問題發(fā)現(xiàn)。

slub_debug和kasan有一定的重復(fù)，部分slub_debug問題需要借助slabinfo去發(fā)現(xiàn)；kasan更快，所有問題獨(dú)立上報(bào)，缺點(diǎn)是需要高版本GCC支持(gcc 4.9.2 or gcc 5.0)。

測(cè)試環(huán)境準(zhǔn)備

更新內(nèi)核版本到Kernel v4.4，然后編譯：

git clone https://github.com/arnoldlu/linux.git -b running_kernel_4.4 export ARCH=arm64 export CROSS_COMPILE=aarch64-linux-gnu- make defconfig make bzImage -j4 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

slub_debug

關(guān)鍵詞：Red Zone、Padding、Object Layout。

Linux內(nèi)核中，小塊內(nèi)存大量使用slab/slub分配器，slub_debug提供了內(nèi)存檢測(cè)小功能。

內(nèi)存中比較容易出錯(cuò)的地方有：

訪問已經(jīng)釋放的內(nèi)存

越界訪問

重復(fù)釋放內(nèi)存

編譯支持slub_debug內(nèi)核

首先需要打開General setup -> Enable SLUB debugging support，然后再選擇Kernel hacking -> Memory Debugging -> SLUB debugging on by default。

CONFIG_SLUB=y CONFIG_SLUB_DEBUG=y CONFIG_SLUB_DEBUG_ON=y CONFIG_SLUB_STATS=y

測(cè)試環(huán)境：slabinfo、slub.ko

通過slub.ko模擬內(nèi)存異常訪問，有些可以直接顯示，有些需要通過slabinfo -v來(lái)查看。

在tools/vm目錄下，執(zhí)行如下命令，生成可執(zhí)行文件slabinfo。放入_install目錄，打包到zImage中。

make slabinfo CFLAGS=-static ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-

將編譯好的slabinfo放入sbin。

下面三個(gè)測(cè)試代碼：https://github.com/arnoldlu/linux/tree/running_kernel_4.4/test_code/slub_debug

在test_code/slub_debug目錄下執(zhí)行make.sh，將slub.ko/slub2.ko/slub3.ko放入data。

進(jìn)行測(cè)試

啟動(dòng)QEMU：

qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -smp 2 -m 2048 -kernel arch/arm64/boot/Image --append "rdinit=/linuxrc console=ttyAMA0 loglevel=8 slub_debug=UFPZ" -nographic

F：在free的時(shí)候會(huì)執(zhí)行檢查。

Z：表示Red Zone的意思。

P：是Poison的意思。

U：會(huì)記錄slab的使用者信息，如果打開，會(huì)會(huì)顯示分配釋放對(duì)象的棧回溯。

在slub_debug打開SLAB_STORE_USER選項(xiàng)后，可以清晰地看到問題點(diǎn)的backtrace。

測(cè)試結(jié)果

內(nèi)存越界訪問包括Redzone overwritten和Object padding overwritten。

重復(fù)釋放對(duì)應(yīng)Object already free。訪問已釋放內(nèi)存為Posion overwritten。

Redzone overwritten

執(zhí)行insmod data/slub.ko，使用slabinfo -v查看結(jié)果。

static void create_slub_error(void) { buf = kmalloc(32, GFP_KERNEL); if(buf) { memset(buf, 0x55, 80);-----------------------------------雖然分配32字節(jié)，但是對(duì)應(yīng)分配了64字節(jié)。所以設(shè)置為80字節(jié)訪問觸發(fā)異常。從buf開始的80個(gè)字節(jié)仍然被初始化成功。 } }

雖然kmalloc申請(qǐng)了32字節(jié)的slab緩沖區(qū)，但是內(nèi)核分配的是kmalloc-64。所以memset 36字節(jié)不會(huì)報(bào)錯(cuò)，將36改成大于64即可。

一個(gè)slub Debug輸出包括四大部分：

=============================================================================

BUG kmalloc-64 (Tainted: G O ): Redzone overwritten-------------------------------------------------------------1. 問題描述：slab名稱-kmalloc-64，什么錯(cuò)誤-Redzone overwritten。
-----------------------------------------------------------------------------

Disabling lock debugging due to kernel taint
INFO: 0xeddb3640-0xeddb3643. First byte 0x55 instead of 0xcc------------------------------------------------1.1 問題起始和結(jié)束地址，這里一共4字節(jié)。
INFO: Allocated in 0x55555555 age=1766 cpu=0 pid=771---------------------------------------------------------1.2 slab的分配棧回溯
0x55555555
0xbf002014
do_one_initcall+0x90/0x1d8
do_init_module+0x60/0x38c
load_module+0x1bac/0x1e94
SyS_init_module+0x14c/0x15c
ret_fast_syscall+0x0/0x3c
INFO: Freed in do_one_initcall+0x78/0x1d8 age=1766 cpu=0 pid=771-----------------------------------------1.3 slab的釋放?；厮?do_one_initcall+0x78/0x1d8
do_init_module+0x60/0x38c
load_module+0x1bac/0x1e94
SyS_init_module+0x14c/0x15c
ret_fast_syscall+0x0/0x3c
INFO: Slab 0xefdb5660 objects=16 used=14 fp=0xeddb3700 flags=0x0081-----------------------------------1.4 slab的地址，以及其它信息。
INFO: Object 0xeddb3600 @offset=1536 fp=0x55555555-----------------------------------------------------------1.5 當(dāng)前Object起始，及相關(guān)信息

Bytes b4 eddb35f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ------------2. 問題slab對(duì)象內(nèi)容。2.1 打印問題slab對(duì)象內(nèi)容之前一些字節(jié)。
Object eddb3600: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU---------2.2 slab對(duì)象內(nèi)容，全部為0x55。
Object eddb3610: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU
Object eddb3620: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU
Object eddb3630: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU
Redzone eddb3640: 55 55 55 55 UUUU----------------------------------------------------------------------------------2.3 Redzone內(nèi)容，問題出在這里。
Padding eddb36e8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ------------2.4 Padding內(nèi)容，為了對(duì)象對(duì)齊而補(bǔ)充。
Padding eddb36f8: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
CPU: 2 PID: 773 Comm: slabinfo Tainted: G B O 4.4.0+ #93--------------------------------------------------------3. 檢查問題點(diǎn)的棧打印，這里是由于slabinfo找出來(lái)的。
Hardware name: ARM-Versatile Express
[] (unwind_backtrace) from [] (show_stack+0x10/0x14)
[] (show_stack) from [] (dump_stack+0x78/0x88)
[] (dump_stack) from [] (check_bytes_and_report+0xd0/0x10c)
[] (check_bytes_and_report) from [] (check_object+0x164/0x234)
[] (check_object) from [] (validate_slab_slab+0x198/0x1bc)
[] (validate_slab_slab) from [] (validate_store+0xac/0x190)
[] (validate_store) from [] (kernfs_fop_write+0xb8/0x1b4)
[] (kernfs_fop_write) from [] (__vfs_write+0x1c/0xd8)
[] (__vfs_write) from [] (vfs_write+0x90/0x170)
[] (vfs_write) from [] (SyS_write+0x3c/0x90)
[] (SyS_write) from [] (ret_fast_syscall+0x0/0x3c)
FIX kmalloc-64: Restoring 0xeddb3640-0xeddb3643=0xcc----------------------------------------------------------4. 問題點(diǎn)是如何被解決的，此處恢復(fù)4個(gè)字節(jié)為0xcc。

Object padding overwritten

void create_slub_error(void)
{
  int i;

  buf = kmalloc(32, GFP_KERNEL);
  if(buf) {
    buf[-1] = 0x55;------------------------------------------------------------------------向左越界訪問
    kfree(buf);
  }
}
執(zhí)行insmod data/slub4.ko，結(jié)果如下。
這里的越界訪問和之前有點(diǎn)不一樣的是，這里向左越界。覆蓋到了Padding區(qū)域。
al: slub error test init
=============================================================================
BUG kmalloc-128 (Tainted: G O ): Object padding overwritten------------------------------------------------------覆蓋到Padding區(qū)域
-----------------------------------------------------------------------------

Disabling lock debugging due to kernel taint
INFO: 0xffff80007767e9ff-0xffff80007767e9ff. First byte 0x55 instead of 0x5a
INFO: Allocated in call_usermodehelper_setup+0x44/0xb8 age=1 cpu=1 pid=789
alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x1ec/0x260
call_usermodehelper_setup+0x44/0xb8
/ # kobject_uevent_env+0x494/0x500
kobject_uevent+0x10/0x18
load_module+0x18cc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2dd9f80 objects=16 used=9 fp=0xffff80007767ea00 flags=0x4081
INFO: Object 0xffff80007767e800 @offset=2048 fp=0xffff80007767ea00

Bytes b4 ffff80007767e7f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Object ffff80007767e800: 00 01 00 00 00 00 00 00 08 e8 67 77 00 80 ff ff ..........gw....
Object ffff80007767e810: 08 e8 67 77 00 80 ff ff f8 83 0c 00 00 80 ff ff ..gw............
Object ffff80007767e820: 00 00 00 00 00 00 00 00 00 6e aa 00 00 80 ff ff .........n......
Object ffff80007767e830: 00 23 67 78 00 80 ff ff 18 23 67 78 00 80 ff ff .#gx.....#gx....
Object ffff80007767e840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff80007767e850: b8 8e 32 00 00 80 ff ff 00 23 67 78 00 80 ff ff ..2......#gx....
Object ffff80007767e860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff80007767e870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Redzone ffff80007767e880: cc cc cc cc cc cc cc cc ........
Padding ffff80007767e9c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff80007767e9d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff80007767e9e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff80007767e9f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 55 ZZZZZZZZZZZZZZZU
CPU: 0 PID: 790 Comm: mdev Tainted: G B O 4.4.0+ #116
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x108
[] show_stack+0x14/0x20
[] dump_stack+0x94/0xd0
[] print_trailer+0x128/0x1b8
[] check_bytes_and_report+0xd8/0x118
[] check_object+0xa0/0x240
[] free_debug_processing+0x128/0x380
[] __slab_free+0x344/0x4a0
[] kfree+0x1ec/0x220
[] umh_complete+0x58/0x68
[] call_usermodehelper_exec_async+0x150/0x170
[] ret_from_fork+0x10/0x40
FIX kmalloc-128: Restoring 0xffff80007767e9ff-0xffff80007767e9ff=0x5a---------------------------------------------------------問題處理是將對(duì)應(yīng)字節(jié)恢復(fù)為0x5a。

Object already free

void create_slub_error(void)
{
  buf = kmalloc(32, GFP_KERNEL);
  if(buf) {
    memset(buf, 0x55, 32);
    kfree(buf);
    printk("al: Object already freed");
    kfree(buf);
  }
}
內(nèi)核中free執(zhí)行流程如下：
kfree
  ->slab_free
    ->__slab_free
      ->kmem_cache_debug
        ->free_debug_processing
         ->on_freelist   
執(zhí)行insmod data/slub2.ko，結(jié)果如下。
al: slub error test init
al: Object already freed
=============================================================================
BUG kmalloc-128 (Tainted: G B O ): Object already free------------------------------------------------------------------在64位系統(tǒng)，32字節(jié)的kmalloc變成了kmalloc-128，問題類型是：Object already free，也即重復(fù)釋放。
-----------------------------------------------------------------------------

INFO: Allocated in create_slub_error+0x20/0x80 [slub2] age=0 cpu=1 pid=791------------------------------------內(nèi)存分配點(diǎn)?；厮?alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x1ec/0x260
create_slub_error+0x20/0x80 [slub2]
my_test_init+0x14/0x28 [slub2]
do_one_initcall+0x90/0x1a0
do_init_module+0x60/0x1cc
load_module+0x18dc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Freed in create_slub_error+0x50/0x80 [slub2] age=0 cpu=1 pid=791------------------------------------------內(nèi)存釋放點(diǎn)?；厮?free_debug_processing+0x17c/0x380
__slab_free+0x344/0x4a0
kfree+0x1ec/0x220
create_slub_error+0x50/0x80 [slub2]
my_test_init+0x14/0x28 [slub2]
do_one_initcall+0x90/0x1a0
do_init_module+0x60/0x1cc
load_module+0x18dc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2dda800 objects=16 used=7 fp=0xffff8000776a0800 flags=0x4081
INFO: Object 0xffff8000776a0800 @offset=2048 fp=0xffff8000776a0a00

Bytes b4 ffff8000776a07f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Object ffff8000776a0800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk-----------------內(nèi)存內(nèi)容打印，供128字節(jié)。
Object ffff8000776a0810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
Redzone ffff8000776a0880: bb bb bb bb bb bb bb bb ........
Padding ffff8000776a09c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776a09d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776a09e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776a09f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
CPU: 1 PID: 791 Comm: insmod Tainted: G B O 4.4.0+ #116--------------------------------------------------------------此處問題在insmod就發(fā)現(xiàn)了，所以檢查出問題的進(jìn)程就是insmod。
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x108
[] show_stack+0x14/0x20
[] dump_stack+0x94/0xd0
[] print_trailer+0x128/0x1b8
[] free_debug_processing+0x29c/0x380
[] __slab_free+0x344/0x4a0
[] kfree+0x1ec/0x220
[] create_slub_error+0x60/0x80 [slub2]
[] my_test_init+0x14/0x28 [slub2]
[] do_one_initcall+0x90/0x1a0
[] do_init_module+0x60/0x1cc
[] load_module+0x18dc/0x1d78
[] SyS_init_module+0x150/0x178
[] el0_svc_naked+0x24/0x28
FIX kmalloc-128: Object at 0xffff8000776a0800 not freed------------------------------------------------------------------處理的結(jié)果是，此處slab 對(duì)象是沒有被釋放。

Poison overwritten

訪問已釋放內(nèi)存的測(cè)試代碼如下：

static void create_slub_error(void)
{
  buf = kmalloc(32, GFP_KERNEL);-----------------------此時(shí)的buf內(nèi)容都是0x6B
  if(buf) {
    kfree(buf);
    printk("al: Access after free");
    memset(buf, 0x55, 32);-----------------------------雖然被釋放，但是memset仍然生效了變成了0x55。
  }
}
執(zhí)行insmod data/slub3.ko ，使用slabinfo -v查看結(jié)果。
=============================================================================

BUG kmalloc-128 (Tainted: G B O ): Poison overwritten----------------------------------------------slab名稱為kmalloc-64，問題類型是：Poison overwritten，即訪問已釋放內(nèi)存。
-----------------------------------------------------------------------------

 

INFO: 0xffff800077692800-0xffff80007769281f. First byte 0x55 instead of 0x6b
INFO: Allocated in create_slub_error+0x28/0xf0 [slub3] age=1089 cpu=1 pid=793----------分配點(diǎn)的棧回溯
alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x1ec/0x260
create_slub_error+0x28/0xf0 [slub3]
0xffff7ffffc00e014
do_one_initcall+0x90/0x1a0
do_init_module+0x60/0x1cc
load_module+0x18dc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Freed in create_slub_error+0x80/0xf0 [slub3] age=1089 cpu=1 pid=793--------------釋放點(diǎn)的?；厮?free_debug_processing+0x17c/0x380
__slab_free+0x344/0x4a0
kfree+0x1ec/0x220
create_slub_error+0x80/0xf0 [slub3]
0xffff7ffffc00e014
do_one_initcall+0x90/0x1a0
do_init_module+0x60/0x1cc
load_module+0x18dc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2dda480 objects=16 used=16 fp=0x (null) flags=0x4080
INFO: Object 0xffff800077692800 @offset=2048 fp=0xffff800077692400

 

Bytes b4 ffff8000776927f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Object ffff800077692800: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU--------前32字節(jié)仍然被修改成功。
Object ffff800077692810: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU
Object ffff800077692820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
Redzone ffff800077692880: bb bb bb bb bb bb bb bb ........
Padding ffff8000776929c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776929d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776929e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776929f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
CPU: 0 PID: 795 Comm: slabinfo Tainted: G B O 4.4.0+ #116
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x108
[] show_stack+0x14/0x20
[] dump_stack+0x94/0xd0
[] print_trailer+0x128/0x1b8
[] check_bytes_and_report+0xd8/0x118
[] check_object+0x1cc/0x240
[] alloc_debug_processing+0x108/0x188
[] ___slab_alloc.constprop.30+0x3f8/0x440
[] __slab_alloc.isra.27.constprop.29+0x24/0x38
[] kmem_cache_alloc+0x1ec/0x260
[] seq_open+0x34/0x90
[] kernfs_fop_open+0x194/0x370
[] do_dentry_open+0x214/0x318
[] vfs_open+0x58/0x68
[] path_openat+0x460/0xdf0
[] do_filp_open+0x60/0xe0
[] do_sys_open+0x12c/0x218
[] compat_SyS_open+0x1c/0x28
[] el0_svc_naked+0x24/0x28
FIX kmalloc-128: Restoring 0xffff800077692800-0xffff80007769281f=0x6b

 

FIX kmalloc-128: Marking all objects used
SLUB: kmalloc-128 210 slabs counted but counter=211
slabinfo (795) used greatest stack depth: 12976 bytes left

kmemleak

kmemleak是內(nèi)核提供的一種檢測(cè)內(nèi)存泄露工具，啟動(dòng)一個(gè)內(nèi)核線程掃描內(nèi)存，并打印發(fā)現(xiàn)新的未引用對(duì)象數(shù)量。

支持kmemleak內(nèi)核選項(xiàng)

要使用kmemlieak，需要打開如下內(nèi)核選項(xiàng)。

Kernel hacking->Memory Debugging->Kernel memory leak detector：

CONFIG_HAVE_DEBUG_KMEMLEAK=y CONFIG_DEBUG_KMEMLEAK=y CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=400 # CONFIG_DEBUG_KMEMLEAK_TEST is not set CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y---------或者關(guān)閉此選項(xiàng)，則不需要在命令行添加kmemleak=on。

構(gòu)造測(cè)試環(huán)境

同時(shí)還需要在內(nèi)核啟動(dòng)命令行中添加kmemleak=on。

qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -smp 2 -m 2048 -kernel arch/arm64/boot/Image --append "rdinit=/linuxrc console=ttyAMA0 loglevel=8 kmemleak=on" -nographic

static char *buf; void create_kmemleak(void) { buf = kmalloc(120, GFP_KERNEL); buf = vmalloc(4096); }

進(jìn)行測(cè)試

進(jìn)行kmemleak測(cè)試之前，需要寫入scan觸發(fā)掃描操作。

然后通過讀kmemlean節(jié)點(diǎn)讀取相關(guān)信息。

打開kmemlean掃描功能：echo scan > sys/kernel/debug/kmemleak

加載問題module：insmod data/kmemleak.ko

等待問題發(fā)現(xiàn)：kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak)

查看kmemleak結(jié)果：cat /sys/kernel/debug/kmemleak

分析測(cè)試結(jié)果

每處泄露，都標(biāo)出泄露地址和大??；相關(guān)進(jìn)程信息；內(nèi)存內(nèi)容dump；棧回溯。
kmemleak會(huì)提示內(nèi)存泄露可疑對(duì)象的具體棧調(diào)用信息、可疑對(duì)象的大小、使用哪個(gè)函數(shù)分配、二進(jìn)制打印。
unreferenced object 0xede22dc0 (size 128):-------------------------------------第一處可疑泄露128字節(jié)
  comm "insmod", pid 765, jiffies 4294941257 (age 104.920s)--------------------相關(guān)進(jìn)程信息
  hex dump (first 32 bytes):---------------------------------------------------二進(jìn)制打印
    6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
    6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
  backtrace:-------------------------------------------------------------------?；厮?    [] 0xbf002014
    [] do_one_initcall+0x90/0x1d8
    [] do_init_module+0x60/0x38c
    [] load_module+0x1bac/0x1e94
    [] SyS_init_module+0x14c/0x15c
    [] ret_fast_syscall+0x0/0x3c
    [] 0xffffffff
unreferenced object 0xf12ba000 (size 4096):
  comm "insmod", pid 765, jiffies 4294941257 (age 104.920s)
  hex dump (first 32 bytes):
    d8 21 00 00 02 18 00 00 e4 21 00 00 02 18 00 00  .!.......!......
    46 22 00 00 02 18 00 00 52 22 00 00 02 18 00 00  F"......R"......
  backtrace:
    [] vmalloc+0x2c/0x34
    [] 0xbf002014
    [] do_one_initcall+0x90/0x1d8
    [] do_init_module+0x60/0x38c
    [] load_module+0x1bac/0x1e94
    [] SyS_init_module+0x14c/0x15c
    [] ret_fast_syscall+0x0/0x3c
    [] 0xffffffff

kasan

kasan暫不支持32位ARM，支持ARM64和X86。

kasan是一個(gè)動(dòng)態(tài)檢查內(nèi)存錯(cuò)誤的工具，可以檢查內(nèi)存越界訪問、使用已釋放內(nèi)存、重復(fù)釋放以及棧溢出。

使能kasan

使用kasan，必須打開CONFIG_KASAN。

Kernel hacking->Memory debugging->KASan: runtime memory debugger

CONFIG_HAVE_ARCH_KASAN=y CONFIG_KASAN=y # CONFIG_KASAN_OUTLINE is not set CONFIG_KASAN_INLINE=y CONFIG_TEST_KASAN=m

代碼分析

kasan_report ->kasan_report_error ->print_error_description ->print_address_description ->print_shadow_for_address

測(cè)試用及分析

kasan提供了一個(gè)測(cè)試程序test_kacan.c，將其編譯成模塊，加載到內(nèi)核?？梢阅M很多內(nèi)存錯(cuò)誤場(chǎng)景。

kasan可以檢測(cè)到越界訪問、訪問已釋放內(nèi)存、重復(fù)釋放等類型錯(cuò)誤，其中重復(fù)釋放借助于slub_debug。

insmod data/kasan.ko

越界訪問包括slab越界、棧越界、全局變量越界；訪問已釋放內(nèi)存use-after-free；重復(fù)釋放可以被slub_debug識(shí)別。

slab-out-of-bounds

static noinline void __init kmalloc_oob_right(void)
{
    char *ptr;
    size_t size = 123;

    pr_info("out-of-bounds to right\n");
    ptr = kmalloc(size, GFP_KERNEL);
    if (!ptr) {
        pr_err("Allocation failed\n");
        return;
    }

    ptr[size] = 'x';
    kfree(ptr);
}
此種錯(cuò)誤類型是對(duì)slab的越界訪問，包括左側(cè)、右側(cè)、擴(kuò)大、縮小后越界訪問。除了數(shù)組賦值，還包括memset、指針訪問等等。
al: kasan error test init
kasan test: kmalloc_oob_right out-of-bounds to right
==================================================================
BUG: KASAN: slab-out-of-bounds in kmalloc_oob_right+0xa4/0xe0 [kasan] at addr ffff800066539c7b----------------錯(cuò)誤類型是slab-out-of-bounds，在kmalloc_oob_right中產(chǎn)生。
Write of size 1 by task insmod/788
=============================================================================
BUG kmalloc-128 (Tainted: G O ): kasan: bad access detected-------------------------------------------------------------------slab非法非法訪問
-----------------------------------------------------------------------------

Disabling lock debugging due to kernel taint
INFO: Allocated in kmalloc_oob_right+0x54/0xe0 [kasan] age=0 cpu=1 pid=788--------------------------------------------問題點(diǎn)kmalloc_oob_right的棧回溯
alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x220/0x280
kmalloc_oob_right+0x54/0xe0 [kasan]
kmalloc_tests_init+0x18/0x70 [kasan]
do_one_initcall+0x11c/0x310
do_init_module+0x1cc/0x588
load_module+0x48cc/0x5dc0
SyS_init_module+0x1a8/0x1e0
el0_svc_naked+0x24/0x28
INFO: Freed in do_one_initcall+0x10c/0x310 age=0 cpu=1 pid=788
free_debug_processing+0x17c/0x368
__slab_free+0x344/0x4a0
kfree+0x21c/0x250
do_one_initcall+0x10c/0x310
do_init_module+0x1cc/0x588
load_module+0x48cc/0x5dc0
SyS_init_module+0x1a8/0x1e0
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2994e00 objects=16 used=2 fp=0xffff800066539e00 flags=0x4080
INFO: Object 0xffff800066539c00 @offset=7168 fp=0xffff800066538200

Bytes b4 ffff800066539bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................------------------------------內(nèi)存dump
Object ffff800066539c00: 00 82 53 66 00 80 ff ff 74 65 73 74 73 5f 69 6e ..Sf....tests_in
Object ffff800066539c10: 69 74 20 5b 6b 61 73 61 6e 5d 00 00 00 00 00 00 it [kasan]......
Object ffff800066539c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539db0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539dc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539dd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539de0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
CPU: 1 PID: 788 Comm: insmod Tainted: G B O 4.4.0+ #108------------------------------------------------------------------打印此log消息的?；厮?Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x270
[] show_stack+0x14/0x20
[] dump_stack+0x100/0x188
[] print_trailer+0xf8/0x160
[] object_err+0x3c/0x50
[] kasan_report_error+0x240/0x558
[] __asan_report_store1_noabort+0x48/0x50
[] kmalloc_oob_right+0xa4/0xe0 [kasan]
[] kmalloc_tests_init+0x18/0x70 [kasan]
[] do_one_initcall+0x11c/0x310
[] do_init_module+0x1cc/0x588
[] load_module+0x48cc/0x5dc0
[] SyS_init_module+0x1a8/0x1e0
[] el0_svc_naked+0x24/0x28
Memory state around the buggy address:
ffff800066539b00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff800066539b80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>ffff800066539c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03
^
ffff800066539c80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff800066539d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================

user-after-free

user-after-free是釋放后使用的意思。

static noinline void __init kmalloc_uaf(void)
{
    char *ptr;
    size_t size = 10;

    pr_info("use-after-free\n");
    ptr = kmalloc(size, GFP_KERNEL);
    if (!ptr) {
        pr_err("Allocation failed\n");
        return;
    }

    kfree(ptr);
    *(ptr + 8) = 'x';
}

測(cè)試結(jié)果如下：

kasan test: kmalloc_uaf use-after-free
==================================================================
BUG: KASAN: use-after-free in kmalloc_uaf+0xac/0xe0 [kasan] at addr ffff800066539e08
Write of size 1 by task insmod/788
=============================================================================
BUG kmalloc-128 (Tainted: G B O ): kasan: bad access detected
-----------------------------------------------------------------------------

INFO: Allocated in kmalloc_uaf+0x54/0xe0 [kasan] age=0 cpu=1 pid=788
alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x220/0x280
kmalloc_uaf+0x54/0xe0 [kasan]
kmalloc_tests_init+0x48/0x70 [kasan]
do_one_initcall+0x11c/0x310
do_init_module+0x1cc/0x588
load_module+0x48cc/0x5dc0
SyS_init_module+0x1a8/0x1e0
el0_svc_naked+0x24/0x28
INFO: Freed in kmalloc_uaf+0x84/0xe0 [kasan] age=0 cpu=1 pid=788
free_debug_processing+0x17c/0x368
__slab_free+0x344/0x4a0
kfree+0x21c/0x250
kmalloc_uaf+0x84/0xe0 [kasan]
kmalloc_tests_init+0x48/0x70 [kasan]
do_one_initcall+0x11c/0x310
do_init_module+0x1cc/0x588
load_module+0x48cc/0x5dc0
SyS_init_module+0x1a8/0x1e0
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2994e00 objects=16 used=1 fp=0xffff800066539e00 flags=0x4080
INFO: Object 0xffff800066539e00 @offset=7680 fp=0xffff800066539800

Bytes b4 ffff800066539df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539e00: 00 98 53 66 00 80 ff ff 00 00 00 00 00 00 00 00 ..Sf............
Object ffff800066539e10: 00 9e 53 66 00 80 ff ff d0 51 12 00 00 80 ff ff ..Sf.....Q......
Object ffff800066539e20: 00 00 00 00 00 00 00 00 e0 14 6d 01 00 80 ff ff ..........m.....
Object ffff800066539e30: 00 69 a3 66 00 80 ff ff 18 69 a3 66 00 80 ff ff .i.f.....i.f....
Object ffff800066539e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539e50: 30 da 73 00 00 80 ff ff 00 69 a3 66 00 80 ff ff 0.s......i.f....
Object ffff800066539e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
CPU: 1 PID: 788 Comm: insmod Tainted: G B O 4.4.0+ #108
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x270
[] show_stack+0x14/0x20
[] dump_stack+0x100/0x188
[] print_trailer+0xf8/0x160
[] object_err+0x3c/0x50
[] kasan_report_error+0x240/0x558
[] __asan_report_store1_noabort+0x48/0x50
[] kmalloc_uaf+0xac/0xe0 [kasan]
[] kmalloc_tests_init+0x48/0x70 [kasan]
[] do_one_initcall+0x11c/0x310
[] do_init_module+0x1cc/0x588
[] load_module+0x48cc/0x5dc0
[] SyS_init_module+0x1a8/0x1e0
[] el0_svc_naked+0x24/0x28
Memory state around the buggy address:
ffff800066539d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff800066539d80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>ffff800066539e00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff800066539e80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff800066539f00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================

stack-out-of-bounds

棧越界訪問是函數(shù)中數(shù)組越界，在實(shí)際工程中經(jīng)常出現(xiàn)，問題難以發(fā)現(xiàn)。

static noinline void __init kasan_stack_oob(void)
{
    char stack_array[10];
    volatile int i = 0;
    char *p = &stack_array[ARRAY_SIZE(stack_array) + i];

    pr_info("out-of-bounds on stack\n");
    *(volatile char *)p;
}
 




kasan test: kasan_stack_oob out-of-bounds on stack
==================================================================
BUG: KASAN: stack-out-of-bounds in kasan_stack_oob+0xa8/0xf0 [kasan] at addr ffff800066acb95a
Read of size 1 by task insmod/788
page:ffff7bffc29ab2c0 count:0 mapcount:0 mapping: (null) index:0x0
flags: 0x0()
page dumped because: kasan: bad access detected
CPU: 1 PID: 788 Comm: insmod Tainted: G B O 4.4.0+ #108
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x270
[] show_stack+0x14/0x20
[] dump_stack+0x100/0x188
[] kasan_report_error+0x530/0x558
[] __asan_report_load1_noabort+0x48/0x50
[] kasan_stack_oob+0xa8/0xf0 [kasan]
[] kmalloc_tests_init+0x58/0x70 [kasan]
[] do_one_initcall+0x11c/0x310
[] do_init_module+0x1cc/0x588
[] load_module+0x48cc/0x5dc0
[] SyS_init_module+0x1a8/0x1e0
[] el0_svc_naked+0x24/0x28
Memory state around the buggy address:
ffff800066acb800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff800066acb880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1
>ffff800066acb900: f1 f1 04 f4 f4 f4 f2 f2 f2 f2 00 02 f4 f4 f3 f3
^
ffff800066acb980: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1
ffff800066acba00: f1 f1 00 00 00 00 00 00 00 00 f3 f3 f3 f3 00 00
==================================================================

global-out-of-bounds

static char global_array[10];

static noinline void __init kasan_global_oob(void)
{
    volatile int i = 3;
    char *p = &global_array[ARRAY_SIZE(global_array) + i];

    pr_info("out-of-bounds global variable\n");
    *(volatile char *)p;
}

測(cè)試結(jié)果如下：

kasan test: kasan_global_oob out-of-bounds global variable
==================================================================
BUG: KASAN: global-out-of-bounds in kasan_global_oob+0x9c/0xe8 [kasan] at addr ffff7ffffc001c8d
Read of size 1 by task insmod/788
Address belongs to variable global_array+0xd/0xffffffffffffe3f8 [kasan]
CPU: 1 PID: 788 Comm: insmod Tainted: G B O 4.4.0+ #108
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x270
[] show_stack+0x14/0x20
[] dump_stack+0x100/0x188
[] kasan_report_error+0x530/0x558
[] __asan_report_load1_noabort+0x48/0x50
[] kasan_global_oob+0x9c/0xe8 [kasan]
[] kmalloc_tests_init+0x5c/0x70 [kasan]
[] do_one_initcall+0x11c/0x310
[] do_init_module+0x1cc/0x588
[] load_module+0x48cc/0x5dc0
[] SyS_init_module+0x1a8/0x1e0
[] el0_svc_naked+0x24/0x28
Memory state around the buggy address:
ffff7ffffc001b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff7ffffc001c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff7ffffc001c80: 00 02 fa fa fa fa fa fa 00 00 00 00 00 00 00 00
^
ffff7ffffc001d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff7ffffc001d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==================================================================

小結(jié)

kmemleak檢查內(nèi)存泄露的獨(dú)門絕技，讓其有一定市場(chǎng)空間。但功能比較單一，專注于內(nèi)存泄露問題。

對(duì)于非ARM64/x86平臺(tái)，只能使用slub_debug進(jìn)行內(nèi)存問題分析；kasan更高效，但也需要更高的內(nèi)核和GCC版本支持。

審核編輯：湯梓紅