Gem5 Arm Fullsystem仿真

1. 如果使用spec2017 X86編譯，那么會存在對intel比較新的指令不支持的問題；后來使用gcc march K6 m32來解決，即使用amd的k6 32bit編譯，但是這也只是權宜之計 ;

2. gem5的開發(fā)人員在郵件list中介紹intel對gem5的譯碼支持比較差，最開始gem5的x86也是基于amd的;

3. 很多論文中使用了ARM架構，gem5的開發(fā)人員也和ARM合作比較緊密，所以gem5對arm架構的指令支持比較好;

4. 即使使用上面的方法，se模式還是會遇到gem5沒有模擬的system call函數的問題

5. 同時還會遇到環(huán)境的問題，比如549.fotonik3d，需要手動將input輸入的壓縮文件OBJ.dat.xz手動解壓之后才能運行。

因為上述的問題，在郵件list中可以看出gem5的開發(fā)人員比較傾向于使用full system模式，用他的話就是"works magically"。

X86 full system

如果是基于X86 Ubuntu系統模擬gem5 arch，制作disk image比較簡單，可以git clone gem5 resource, 在spec2017文件夾下放入spec2017.iso，調用build.sh自動調用packer將spec2017裝入ubuntu的disk image生成spec-2017。

 ./build/X86/gem5.fast 
 --outdir=./m5out/ 
configs/example/gem5_library/x86-spec-cpu2017-benchmarks.py
--image=path/spec-2017
--benchmark=505.mcf_r
--size=ref
--partition=1

這里需要強調的是，gem5中介紹了多次qemu-kvm, 但是qemu-kvm并不是在X86架構中生成disk image和真正運行gem5時必須的工具。

在gem5中引入它的主要作用就是在boot階段使用qemu，在真實的cpu上運行boot 階段，進行加速，實際上如果我們在云服務器上跑，如果沒有qemu軟件或者權限，只是使用atomic cpu跑boot也是比較快的。

就是不要被qemu迷惑，fullsystem gem5可以理解成在gem5上跑app，不過這個app是os，單純的用atomic cpu跑也沒有任何問題。

ARM full system

X86 full system的問題是memory最大支持3GB，目前看gem5的設置是不支持5GB 6GB這樣的設置。

我們介紹一下如何生成gem5的disk image。

首先看一下最終成功運行full system 使用的指令:

./build/ARM/gem5.fast 
 -d./m5out/ARM/fullsystem64/spec2017 
./configs/example/fs.py
 --kernel2022/binaries/vmlinux.arm64
 --disk-imagepath/expanded-aarch64-ubuntu-trusty-headless.img
 --bootloader2022/binaries/boot.arm64--mem-type=DDR4_2400_4x16
 --param'system.highest_el_is_64=True'
 --script=./m5out/ARM/fullsystem64/spec2017_restore/spec2017.rcS

kernel 我的理解就是os內核程序，disk image則是裝載了benchmark的磁盤鏡像。

內核程序與我們無關，我們可以直接使用，disk image 則需要我們手動裝載。

https://www.gem5.org/documentation/general_docs/fullsystem/guest_binaries

script這里指定的是一個script

#!/bin/bash
source/root/.bashrc
/sbin/m5checkpoint1
echo"Arealmultinodeworkloadmightstarthere..."
cd/home/gem5/spec2017
sourceshrc
echo"Resetstats"
/sbin/m5resetstats
runcpu--sizetest--iterations1--configmyconfig.aarch64.cfg--nobuild605.mcf_s
/sbin/m5exit1

通過指定這個script，gem5在boot成功后，運行這個script，就調用了腳本內的runcpu，自動運行了spec2017對應的app。

如果我們不指定這個script，并且不對disk image進行任何修改，那么boot成功后，要求輸入用戶名和密碼，輸入root可以進入，不過這個操作比較麻煩，還是建議指定script。

首先遇到的問題就是官方提供的image 1GB或者2GB，然而spec2017有4GB我們需要對image進行擴容。按照博客的操作如下

```bash
$#Backuptheoriginaldiskimageifneeded
$cpaarch64-ubuntu-trusty-headless.imgexpanded-aarch64-ubuntu-trusty-headless.img
$#Increasediskimageby2G
$ddif=/dev/zerobs=1Gcount=2>>expanded-aarch64-ubuntu-trusty-headless.img
$sudopartedexpanded-aarch64-ubuntu-trusty-headless.imgresizepart1100%
$#Parsesomeinfofor'losetup'and'mount'later
$name=$(sudofdisk-lexpanded-aarch64-ubuntu-trusty-headless.img|tail-1|awk-F:'{print$1}'|awk-F""'{print$1}')
$start_sector=$(sudofdisk-lexpanded-aarch64-ubuntu-trusty-headless.img|grep$name|awk-F""'{print$2}')
$units=$(sudofdisk-lexpanded-aarch64-ubuntu-trusty-headless.img|grepUnits|awk-F""'{print$8}')
$#Attachtodeviceandrecordoutput,tomeitwas/dev/loop18
$ sudo losetup -f --show expanded-aarch64-ubuntu-trusty-headless.img -o $(($start_sector*$units))                         
$sudoe2fsck-f/dev/loop18#Fixpotentialerrors,pressYforallfixes
$sudoresize2fs/dev/loop18#Actualresizingstep
$sudoe2fsck-f/dev/loop18#Doublechecktherearenoerror
$sudolosetup-d/dev/loop18#Detachfromtheloopdevice
$#Mountimageandchecknewsize
$mkdirdisk_mnt
$sudomount-oloop,offset=$(($start_sector*$units))expanded-aarch64-ubuntu-trusty-headless.imgdisk_mnt
$df-h#ShouldshowthenewexpandedimagesizewiththeUsedandAvailfordisk_mnt
$sudoumountdisk_mnt

擴容之后mount image,就可以安裝spec2017到這個disk image了。

這里建議看一下 gem5-resources/src/spec-2017/disk-image/spec-2017/install-spec2017.sh 這個是裝載spec2017到x86 os的過程，我們裝載spec2017到arm，可以按照這個流程來。

1. sudo chroot . #將當前mount目錄切換為主目錄
2. 創(chuàng)建/home/gem5/文件夾 將cpu_spec2017.iso拷貝到這個文件夾
3. 按照cpu_spec2017.iso的install流程，mount cpu_spec2017.iso 然后install.sh
4. install 之后，我們可以build，生成spec2017的可執(zhí)行文件等。建議參考install-spec2017.sh

我們將spec2017安裝到了/home/gem5/，再結合一下剛才介紹的spec2017.rcS，就能看出來這個script的作用實際上就是進入文件夾，然后runcpu。

我們看一下gem5成功boot后運行spec2017的os系統界面，這個界面通過./util/term/m5term 得到，后面會介紹。

[  0.345190] sd 00 [sda] Attached SCSI disk
[  0.352995] EXT4-fs (sda1): mounted filesystem without journal. Opts: (null)
[  0.353004] VFS: Mounted root (ext4 filesystem) on device 8:1.
[  0.353626] devtmpfs: mounted
[  0.353684] Freeing unused kernel memory: 448K
[  0.359059] random: fast init done
Mount failed for selinuxfs on /sys/fs/selinux:  No such file or directory
[  0.372646] random: init: uninitialized urandom read (12 bytes read)
[  0.399519] random: mountall: uninitialized urandom read (12 bytes read)
Boot Success Reset stats
Run 602.gcc_s test
SPEC CPU(r) 2017 Benchmark Suites
Copyright1995-2017StandardPerformanceEvaluationCorporation(SPEC)
runcpu v5825
Using 'linux-aarch64' tools
Reading file manifests... read 32270 entries from 2 files in 0.44s (72680 files/s)
Loading runcpu modules.................
Locating benchmarks...found 47 benchmarks in 53 benchsets.
Reading config file '/home/gem5/spec2017/config/myconfig.aarch64.cfg'
1configurationselected:
 Action  Run Mode   Workload    Report Type    Benchmarks
--------   --------   --------   -----------------   --------------------------
validate   speed    test     SPECspeed2017_int   602.gcc_s               
-------------------------------------------------------------------------------
Settingupenvironmentforrunning

**** REAL SIMULATION ****
build/ARM/dev/arm/rv_ctrl.cc:176: warn: SCReg: Access to unknown device dcc0pos0dev0
build/ARM/arch/arm/insts/pseudo.cc:172: warn:     instruction 'csdb' unimplemented
build/ARM/dev/arm/gic_v2.cc:683: warn: GIC APRn write ignored because not implemented: 0xd0
build/ARM/dev/arm/gic_v2.cc:683: warn: GIC APRn write ignored because not implemented: 0xd4
build/ARM/dev/arm/gic_v2.cc:683: warn: GIC APRn write ignored because not implemented: 0xd8
build/ARM/dev/arm/gic_v2.cc:683: warn: GIC APRn write ignored because not implemented: 0xdc
AtomicCPU 0 At 103419026000 Tid[0] 100000000 instructions are executed.
build/ARM/dev/arm/rv_ctrl.cc:122: warn: Tried to read RealView I/O at offset 0x60 that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:122: warn: Tried to read RealView I/O at offset 0x48 that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:198: warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:122: warn: Tried to read RealView I/O at offset 0x8 that doesn't exist
build/ARM/dev/arm/rv_ctrl.cc:122: warn: Tried to read RealView I/O at offset 0x48 that doesn't exist
build/ARM/dev/arm/energy_ctrl.cc:77: warn: EnergyCtrl: Disabled handler, ignoring read from reg 0
AtomicCPU 0 At 437185231000 Tid[0] 200000000 instructions are executed.
AtomicCPU 0 At 499727384000 Tid[0] 300000000 instructions are executed.
AtomicCPU 0 At 582010000000 Tid[0] 400000000 instructions are executed.
AtomicCPU 0 At 641077500000 Tid[0] 500000000 instructions are executed.
AtomicCPU 0 At 700009321500 Tid[0] 600000000 instructions are executed.
AtomicCPU 0 At 759169539000 Tid[0] 700000000 instructions are executed.
AtomicCPU 0 At 818393124500 Tid[0] 800000000 instructions are executed.
AtomicCPU 0 At 877446054000 Tid[0] 900000000 instructions are executed.
showInstNum AtomicCPU 0 At 907078901500 Tid[0] 950264835 instructions are executed.
Writing checkpoint
build/ARM/sim/simulate.cc:194: info: Entering event queue @ 907078901500.  Starting simulation...
AtomicCPU 0 At 936518571000 Tid[0] 1000000000 instructions are executed.
AtomicCPU 0 At 1019589630500 Tid[0] 100000000 instructions are executed.
AtomicCPU 0 At 1075531936500 Tid[0] 200000000 instructions are executed.
AtomicCPU 0 At 1133110363000 Tid[0] 300000000 instructions are executed.
AtomicCPU 0 At 1190919530500 Tid[0] 400000000 instructions are executed.
AtomicCPU 0 At 1248468781000 Tid[0] 500000000 instructions are executed.
AtomicCPU 0 At 1305429062000 Tid[0] 600000000 instructions are executed.
AtomicCPU 0 At 1363401303000 Tid[0] 700000000 instructions are executed.
AtomicCPU 0 At 1421469390000 Tid[0] 800000000 instructions are executed.
AtomicCPU 0 At 1479484997500 Tid[0] 900000000 instructions are executed.
AtomicCPU 0 At 1537414678500 Tid[0] 1000000000 instructions are executed.
AtomicCPU 0 At 1595390184500 Tid[0] 1100000000 instructions are executed.
AtomicCPU 0 At 1646996227500 Tid[0] 1200000000 instructions are executed.
AtomicCPU 0 At 1698272492000 Tid[0] 1300000000 instructions are executed.
AtomicCPU 0 At 1754533327500 Tid[0] 1400000000 instructions are executed.
AtomicCPU 0 At 1814630599500 Tid[0] 1500000000 instructions are executed.
AtomicCPU 0 At 1866048774500 Tid[0] 1600000000 instructions are executed.
AtomicCPU 0 At 1922046022000 Tid[0] 1700000000 instructions are executed.
AtomicCPU 0 At 1978814167500 Tid[0] 1800000000 instructions are executed.
AtomicCPU 0 At 2036107971000 Tid[0] 1900000000 instructions are executed.
AtomicCPU 0 At 2093257147500 Tid[0] 2000000000 instructions are executed.
AtomicCPU 0 At 2150632827000 Tid[0] 2100000000 instructions are executed.
AtomicCPU 0 At 2206964371500 Tid[0] 2200000000 instructions are executed.
AtomicCPU 0 At 2264055743500 Tid[0] 2300000000 instructions are executed.
AtomicCPU 0 At 2324544549000 Tid[0] 2400000000 instructions are executed.
AtomicCPU 0 At 2381492086000 Tid[0] 2500000000 instructions are executed.
AtomicCPU 0 At 2439386832000 Tid[0] 2600000000 instructions are executed.
AtomicCPU 0 At 2497622146000 Tid[0] 2700000000 instructions are executed.
AtomicCPU 0 At 2556071830000 Tid[0] 2800000000 instructions are executed.
AtomicCPU 0 At 2613942974500 Tid[0] 2900000000 instructions are executed.
AtomicCPU 0 At 2672323657000 Tid[0] 3000000000 instructions are executed.
AtomicCPU 0 At 2730096081000 Tid[0] 3100000000 instructions are executed.
AtomicCPU 0 At 2788185618000 Tid[0] 3200000000 instructions are executed.
AtomicCPU 0 At 2846567200000 Tid[0] 3300000000 instructions are executed.
AtomicCPU 0 At 2906211195500 Tid[0] 3400000000 instructions are executed.
AtomicCPU0At2965004517500Tid[0]3500000000instructionsareexecuted.

在gem5開始運行后，會顯示system.terminal listening for connections on port NUMBER.

這時我們另開一個terminal，輸入./util/term/m5term NUMBER，即可觀察到當前os具體運行到哪一步，而上面說的，如果不指定script，需要手動輸入root也是在這里。

對我?guī)椭艽蟮膬善┛?

https://www.eecg.utoronto.ca/~elsayed9/website/blog/gem5_fs_arm_flow.php

https://lucian.run/2021/10/03/gem5%20FS/

有小伙伴后臺私信我申請讀博士的事，有意向的小伙伴歡迎私信。