# N100 玩转指南

- 安装PVE
- 搞定硬件直通
- 十二代CPU的显卡 SR-IOV

## 安装PVE

### 制作安装介质

- 下载 PVE 最新版 iso

- 制作安装介质

  - GNU Linux

  	```
  	# dd bs=1M conv=fdatasync if=./proxmox-ve_*.iso of=/dev/XYZ
  	```

  - MacOS 

      打开Terminal
      
      转换 `.iso` 文件到 `.dmg` 用这个 `hdiutil`
      
      ```
      # hdiutil convert proxmox-ve_*.iso -format UDRW -o proxmox-ve_*.dmg
      ```
      
      获取磁盘序列
      
      ```
      # diskutil list
      ```
      
      卸载需要写入的U盘
      
      ```
      # diskutil list
      # diskutil unmountDisk /dev/diskX
      ```
      
      dd 制作启动盘
      
      ```
      # sudo dd if=proxmox-ve_*.dmg bs=1M of=/dev/rdiskX
      ```
  
  	  - Windows

​		用 rufus 工具

- 开始安装
  - BIOS 设置U盘启动
  - 进入启动菜单
    - 图形安装方式
    - Console安装方式
## 开启硬件直通
 - 添加PVE源 apt list

   ```
   echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
   			>	/etc/apt/sources.list.d/pve.list
   apt update
   ```

 -  BIOS 相关设置

   - 开启VT-d
	- IOMMU
   - VT-x
	
 - 添加 kernel cmdline

   - Grub 设置

     编辑 `/etc/default/grub`
     ```
     GRUB_CMDLINE_LINUX_DEFAULT="quite"
     ```

     ```
     GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream"
     ```

   - kernel 内核模块设置

      ```
      #!/bin/sh
      cat >> /etc/modules-load.d/vfio.conf << EOF
      vfio
      vfio_iommu_type1
      vfio_pci
      vfio_virqfd 
      EOF 
      ```

      - 添加相应参数

        添加 参数到 `/etc/modprobe.d/vfio.conf` 

        ```
        echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" \
        > /etc/modprobe.d/vfio.conf
        ```

      - 更新 grub & initramfs
   
         ```
         # update-initramfs -u -k all
         # update-grub
         # proxmox-boot-tool refresh
         ```
   
      - 检查IOMMU 是否设置成功
   
        ```
        dmesg | grep -e DMAR -e IOMMU 
        ```
        
        输出类似如下
   
        ```
        [    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
        [    0.007697] ACPI: DMAR 0x0000000079195C18 0000A8 (v01 INTEL  EDK2     00000002      01000013)
        [    0.007741] ACPI: Reserving DMAR table memory at [mem 0x79195c18-0x79195cbf]
        [    0.130354] DMAR: IOMMU enabled
        [    0.388873] DMAR: Host address width 39
        [    0.389091] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
        [    0.389318] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
        ```
   
        `DMAR: IOMMU enabled` 注意输出有这个就是成功了！
        
      - 由于N100主机只支持 EFI启动，我遇到最大的坑就在这里
      
        > 我用的是zfs文件系统，PVE在EFI启动过程使用的是systemd-boot，没有使用grub，所以上面添加 kernel cmdline的方法是不管用的，需要修改loader.conf
        
      - Systems-boot kernel cmdline 修改
      
        修改 `/etc/kernel/cmdline`
        
         ```
         # cat /etc/kernel/cmdline
         root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
         ```
        
        ```
        # update-initramfs -u -k all
        ```
        
        

# 显卡SR-IOV

## 安装内核工具

`apt install build-* dkms`

## 下载内核补丁

注意：内核补丁还在测试，不建议用在生产环境

下载github源码 `i915-sriov-dkms`

进入下载的目录，修改`PACKAGE_NAME` to `i915-sriov-dkms`, 修改 `PACKAGE_VERSION` to `6.2`. 保存文件。

目前PVE的内核用的是 6.2

```
# uname -ar
Linux n100 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) x86_64 GNU/Linux
```

## 拷贝这个目录到 `/usr/src/i915-sriov-dkms-6.2`

## 安装dkms模块

`dkms -i -m i915-sriov-dkms -v 6.2`

## 安装 `sysfsutils`

`echo "devices/pci0000:00/0000:00:02.0/sriov_numvfs = 7" > /etc/sysfs.conf`

## 修改内核参数

这里vfs就是虚拟显卡的数量，最大多少我没查，这里我设置的是3。

```
intel_iommu=on i915.enable_guc=3 i915.max_vfs=3
```

- grub

  修改 `/etc/defautl/grub` 添加到 `GRUB_CMDLINE_LINUX_DEFAULT="quite"`这句里面

- Systems-boot

  我用的zfs 所以修改是这种方式

  `/etc/kernel/cmdline`修改成

  ```
  root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
  ```

## 更新内核和 initramfs

`update-initramfs -u -k all`

`proxmox-boot-tool refresh`

## 锁定内核

**由于最新的内核没有headers提供，为了避免更新把这个内核顶了，我们要锁定目前编译好的内核！**

​	`# proxmox-boot-tool kernel pin 6.2.16-3-pve`

## 重启

`reboot`

## 检查是否显卡 SR-IOV是否成功

`lcpci |grep -i vga`

**输出如下，就是成功了！**

```
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.1 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.2 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.3 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
```

**或者如下输出，看到我的已经开启了3个虚拟显卡**

```
# dmesg | grep i915
[    0.000000] Command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
[    0.077241] Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
[    4.811959] i915 0000:00:02.0: Running in SR-IOV PF mode
[    4.812864] i915 0000:00:02.0: [drm] VT-d active for gfx access
[    4.813302] i915 0000:00:02.0: vgaarb: deactivate vga console
[    4.813362] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[    4.813758] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    4.814738] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[    4.815136] i915 0000:00:02.0: Direct firmware load for i915/adlp_dmc.bin failed with error -2
[    4.817242] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
[    4.822141] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/tgl_guc_70.bin version 70.5.1
[    4.822154] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[    4.827027] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads!
[    4.827569] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[    4.827574] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[    4.827921] i915 0000:00:02.0: [drm] GuC RC: enabled
[    4.828778] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[    4.828956] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    4.854926] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[    4.856871] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[    4.857105] i915 0000:00:02.0: 3 VFs could be associated with this PF
[    4.887664] fbcon: i915drmfb (fb0) is primary device
[    4.972789] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[    5.638917] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[    5.639816] i915 0000:00:02.1: enabling device (0000 -> 0002)
[    5.640648] i915 0000:00:02.1: Running in SR-IOV VF mode
[    5.641809] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0
[    5.643254] i915 0000:00:02.1: [drm] VT-d active for gfx access
[    5.644017] i915 0000:00:02.1: [drm] Using Transparent Hugepages
[    5.645371] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0
[    5.646658] i915 0000:00:02.1: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[    5.647370] i915 0000:00:02.1: HuC firmware PRELOADED
[    5.650198] i915 0000:00:02.1: [drm] Protected Xe Path (PXP) protected content support initialized
[    5.650917] i915 0000:00:02.1: [drm] PMU not supported for this GPU.
[    5.651749] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.1 on minor 1
[    5.656497] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    5.657170] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.657938] i915 0000:00:02.2: enabling device (0000 -> 0002)
[    5.658678] i915 0000:00:02.2: Running in SR-IOV VF mode
[    5.659806] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0
[    5.661509] i915 0000:00:02.2: [drm] VT-d active for gfx access
[    5.662199] i915 0000:00:02.2: [drm] Using Transparent Hugepages
[    5.663417] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0
[    5.665322] i915 0000:00:02.2: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[    5.665956] i915 0000:00:02.2: HuC firmware PRELOADED
[    5.668308] i915 0000:00:02.2: [drm] Protected Xe Path (PXP) protected content support initialized
[    5.668947] i915 0000:00:02.2: [drm] PMU not supported for this GPU.
[    5.669666] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.2 on minor 2
[    5.673931] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    5.674572] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=none
[    5.675219] i915 0000:00:02.2: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.675964] i915 0000:00:02.3: enabling device (0000 -> 0002)
[    5.676640] i915 0000:00:02.3: Running in SR-IOV VF mode
[    5.677500] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0
[    5.678512] i915 0000:00:02.3: [drm] VT-d active for gfx access
[    5.679186] i915 0000:00:02.3: [drm] Using Transparent Hugepages
[    5.680354] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0
[    5.681176] i915 0000:00:02.3: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[    5.681765] i915 0000:00:02.3: HuC firmware PRELOADED
[    5.684137] i915 0000:00:02.3: [drm] Protected Xe Path (PXP) protected content support initialized
[    5.684777] i915 0000:00:02.3: [drm] PMU not supported for this GPU.
[    5.685487] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.3 on minor 3
[    5.686338] i915 0000:00:02.0: Enabled 3 VFs
```

# 感谢阅读

今天就到这里了，至于虚拟机如何使用这个显卡，后续再更新！