ob/pve
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

add some about n100 file

Browse Source
This commit is contained in:
littlesum
2023-11-19 20:36:18 +08:00
parent eaea772004
commit 0801b82999
6 changed files with 706 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
N100.md
+308
View File
@@ -0,0 +1,308 @@
# N100 玩转指南
- 安装PVE
- 搞定硬件直通
- 十二代CPU的显卡 SR-IOV
## 安装PVE
### 制作安装介质
- 下载 PVE 最新版 iso
- 制作安装介质
- GNU Linux
```
# dd bs=1M conv=fdatasync if=./proxmox-ve_*.iso of=/dev/XYZ
```
- MacOS
打开Terminal
转换 `.iso` 文件到 `.dmg` 用这个 `hdiutil`
```
# hdiutil convert proxmox-ve_*.iso -format UDRW -o proxmox-ve_*.dmg
```
获取磁盘序列
```
# diskutil list
```
卸载需要写入的U盘
```
# diskutil list
# diskutil unmountDisk /dev/diskX
```
dd 制作启动盘
```
# sudo dd if=proxmox-ve_*.dmg bs=1M of=/dev/rdiskX
```
- Windows
​ 用 rufus 工具
- 开始安装
- BIOS 设置U盘启动
- 进入启动菜单
- 图形安装方式
- Console安装方式
## 开启硬件直通
- 添加PVE源 apt list
```
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
> /etc/apt/sources.list.d/pve.list
apt update
```
- BIOS 相关设置
- 开启VT-d
- IOMMU
- VT-x
- 添加 kernel cmdline
- Grub 设置
编辑 `/etc/default/grub`
```
GRUB_CMDLINE_LINUX_DEFAULT="quite"
```
```
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream"
```
- kernel 内核模块设置
```
#!/bin/sh
cat >> /etc/modules-load.d/vfio.conf << EOF
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
EOF
```
- 添加相应参数
添加 参数到 `/etc/modprobe.d/vfio.conf`
```
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" \
> /etc/modprobe.d/vfio.conf
```
- 更新 grub & initramfs
```
# update-initramfs -u -k all
# update-grub
# proxmox-boot-tool refresh
```
- 检查IOMMU 是否设置成功
```
dmesg | grep -e DMAR -e IOMMU
```
输出类似如下
```
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.007697] ACPI: DMAR 0x0000000079195C18 0000A8 (v01 INTEL EDK2 00000002 01000013)
[ 0.007741] ACPI: Reserving DMAR table memory at [mem 0x79195c18-0x79195cbf]
[ 0.130354] DMAR: IOMMU enabled
[ 0.388873] DMAR: Host address width 39
[ 0.389091] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.389318] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
```
`DMAR: IOMMU enabled` 注意输出有这个就是成功了！
- 由于N100主机只支持 EFI启动，我遇到最大的坑就在这里
> 我用的是zfs文件系统，PVE在EFI启动过程使用的是systemd-boot，没有使用grub，所以上面添加 kernel cmdline的方法是不管用的，需要修改loader.conf
- Systems-boot kernel cmdline 修改
修改 `/etc/kernel/cmdline`
```
# cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
```
```
# update-initramfs -u -k all
```
# 显卡SR-IOV
## 安装内核工具
`apt install build-* dkms`
## 下载内核补丁
注意：内核补丁还在测试，不建议用在生产环境
下载github源码 `i915-sriov-dkms`
进入下载的目录，修改`PACKAGE_NAME` to `i915-sriov-dkms`, 修改 `PACKAGE_VERSION` to `6.2`. 保存文件。
目前PVE的内核用的是 6.2
```
# uname -ar
Linux n100 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) x86_64 GNU/Linux
```
## 拷贝这个目录到 `/usr/src/i915-sriov-dkms-6.2`
## 安装dkms模块
`dkms -i -m i915-sriov-dkms -v 6.2`
## 安装 `sysfsutils`
`echo "devices/pci0000:00/0000:00:02.0/sriov_numvfs = 7" > /etc/sysfs.conf`
## 修改内核参数
这里vfs就是虚拟显卡的数量，最大多少我没查，这里我设置的是3。
```
intel_iommu=on i915.enable_guc=3 i915.max_vfs=3
```
- grub
修改 `/etc/defautl/grub` 添加到 `GRUB_CMDLINE_LINUX_DEFAULT="quite"`这句里面
- Systems-boot
我用的zfs 所以修改是这种方式
`/etc/kernel/cmdline`修改成
```
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
```
## 更新内核和 initramfs
`update-initramfs -u -k all`
`proxmox-boot-tool refresh`
## 锁定内核
**由于最新的内核没有headers提供，为了避免更新把这个内核顶了，我们要锁定目前编译好的内核！**
​ `# proxmox-boot-tool kernel pin 6.2.16-3-pve`
## 重启
`reboot`
## 检查是否显卡 SR-IOV是否成功
`lcpci |grep -i vga`
**输出如下，就是成功了！**
```
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.1 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.2 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.3 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
```
**或者如下输出，看到我的已经开启了3个虚拟显卡**
```
# dmesg | grep i915
[ 0.000000] Command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
[ 0.077241] Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
[ 4.811959] i915 0000:00:02.0: Running in SR-IOV PF mode
[ 4.812864] i915 0000:00:02.0: [drm] VT-d active for gfx access
[ 4.813302] i915 0000:00:02.0: vgaarb: deactivate vga console
[ 4.813362] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[ 4.813758] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 4.814738] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[ 4.815136] i915 0000:00:02.0: Direct firmware load for i915/adlp_dmc.bin failed with error -2
[ 4.817242] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
[ 4.822141] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/tgl_guc_70.bin version 70.5.1
[ 4.822154] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[ 4.827027] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads!
[ 4.827569] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[ 4.827574] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[ 4.827921] i915 0000:00:02.0: [drm] GuC RC: enabled
[ 4.828778] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[ 4.828956] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[ 4.854926] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[ 4.856871] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[ 4.857105] i915 0000:00:02.0: 3 VFs could be associated with this PF
[ 4.887664] fbcon: i915drmfb (fb0) is primary device
[ 4.972789] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[ 5.638917] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[ 5.639816] i915 0000:00:02.1: enabling device (0000 -> 0002)
[ 5.640648] i915 0000:00:02.1: Running in SR-IOV VF mode
[ 5.641809] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.643254] i915 0000:00:02.1: [drm] VT-d active for gfx access
[ 5.644017] i915 0000:00:02.1: [drm] Using Transparent Hugepages
[ 5.645371] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.646658] i915 0000:00:02.1: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[ 5.647370] i915 0000:00:02.1: HuC firmware PRELOADED
[ 5.650198] i915 0000:00:02.1: [drm] Protected Xe Path (PXP) protected content support initialized
[ 5.650917] i915 0000:00:02.1: [drm] PMU not supported for this GPU.
[ 5.651749] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.1 on minor 1
[ 5.656497] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[ 5.657170] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 5.657938] i915 0000:00:02.2: enabling device (0000 -> 0002)
[ 5.658678] i915 0000:00:02.2: Running in SR-IOV VF mode
[ 5.659806] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.661509] i915 0000:00:02.2: [drm] VT-d active for gfx access
[ 5.662199] i915 0000:00:02.2: [drm] Using Transparent Hugepages
[ 5.663417] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.665322] i915 0000:00:02.2: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[ 5.665956] i915 0000:00:02.2: HuC firmware PRELOADED
[ 5.668308] i915 0000:00:02.2: [drm] Protected Xe Path (PXP) protected content support initialized
[ 5.668947] i915 0000:00:02.2: [drm] PMU not supported for this GPU.
[ 5.669666] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.2 on minor 2
[ 5.673931] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[ 5.674572] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=none
[ 5.675219] i915 0000:00:02.2: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 5.675964] i915 0000:00:02.3: enabling device (0000 -> 0002)
[ 5.676640] i915 0000:00:02.3: Running in SR-IOV VF mode
[ 5.677500] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.678512] i915 0000:00:02.3: [drm] VT-d active for gfx access
[ 5.679186] i915 0000:00:02.3: [drm] Using Transparent Hugepages
[ 5.680354] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.681176] i915 0000:00:02.3: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[ 5.681765] i915 0000:00:02.3: HuC firmware PRELOADED
[ 5.684137] i915 0000:00:02.3: [drm] Protected Xe Path (PXP) protected content support initialized
[ 5.684777] i915 0000:00:02.3: [drm] PMU not supported for this GPU.
[ 5.685487] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.3 on minor 3
[ 5.686338] i915 0000:00:02.0: Enabled 3 VFs
```
# 感谢阅读
今天就到这里了，至于虚拟机如何使用这个显卡，后续再更新！