ob/pve
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
ef5dfa43f2e759fc0d3835e140430cb5c600bfbf
pve/N100.md
T
ob 0801b82999 add some about n100 file
2023-11-19 20:36:18 +08:00

309 lines
11 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# N100 玩转指南
- 安装PVE
- 搞定硬件直通
- 十二代CPU的显卡 SR-IOV
## 安装PVE
### 制作安装介质
- 下载 PVE 最新版 iso
- 制作安装介质
- GNU Linux
```
# dd bs=1M conv=fdatasync if=./proxmox-ve_*.iso of=/dev/XYZ
```
- MacOS
打开Terminal
转换 `.iso` 文件到 `.dmg` 用这个 `hdiutil`
```
# hdiutil convert proxmox-ve_*.iso -format UDRW -o proxmox-ve_*.dmg
```
获取磁盘序列
```
# diskutil list
```
卸载需要写入的U盘
```
# diskutil list
# diskutil unmountDisk /dev/diskX
```
dd 制作启动盘
```
# sudo dd if=proxmox-ve_*.dmg bs=1M of=/dev/rdiskX
```
- Windows
​ 用 rufus 工具
- 开始安装
- BIOS 设置U盘启动
- 进入启动菜单
- 图形安装方式
- Console安装方式
## 开启硬件直通
- 添加PVE源 apt list
```
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
> /etc/apt/sources.list.d/pve.list
apt update
```
- BIOS 相关设置
- 开启VT-d
- IOMMU
- VT-x
- 添加 kernel cmdline
- Grub 设置
编辑 `/etc/default/grub`
```
GRUB_CMDLINE_LINUX_DEFAULT="quite"
```
```
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream"
```
- kernel 内核模块设置
```
#!/bin/sh
cat >> /etc/modules-load.d/vfio.conf << EOF
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
EOF
```
- 添加相应参数
添加 参数到 `/etc/modprobe.d/vfio.conf`
```
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" \
> /etc/modprobe.d/vfio.conf
```
- 更新 grub & initramfs
```
# update-initramfs -u -k all
# update-grub
# proxmox-boot-tool refresh
```
- 检查IOMMU 是否设置成功
```
dmesg | grep -e DMAR -e IOMMU
```
输出类似如下
```
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.007697] ACPI: DMAR 0x0000000079195C18 0000A8 (v01 INTEL EDK2 00000002 01000013)
[ 0.007741] ACPI: Reserving DMAR table memory at [mem 0x79195c18-0x79195cbf]
[ 0.130354] DMAR: IOMMU enabled
[ 0.388873] DMAR: Host address width 39
[ 0.389091] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.389318] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
```
`DMAR: IOMMU enabled` 注意输出有这个就是成功了！
- 由于N100主机只支持 EFI启动，我遇到最大的坑就在这里
> 我用的是zfs文件系统，PVE在EFI启动过程使用的是systemd-boot，没有使用grub，所以上面添加 kernel cmdline的方法是不管用的，需要修改loader.conf
- Systems-boot kernel cmdline 修改
修改 `/etc/kernel/cmdline`
```
# cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
```
```
# update-initramfs -u -k all
```
# 显卡SR-IOV
## 安装内核工具
`apt install build-* dkms`
## 下载内核补丁
注意：内核补丁还在测试，不建议用在生产环境
下载github源码 `i915-sriov-dkms`
进入下载的目录，修改`PACKAGE_NAME` to `i915-sriov-dkms`, 修改 `PACKAGE_VERSION` to `6.2`. 保存文件。
目前PVE的内核用的是 6.2
```
# uname -ar
Linux n100 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) x86_64 GNU/Linux
```
## 拷贝这个目录到 `/usr/src/i915-sriov-dkms-6.2`
## 安装dkms模块
`dkms -i -m i915-sriov-dkms -v 6.2`
## 安装 `sysfsutils`
`echo "devices/pci0000:00/0000:00:02.0/sriov_numvfs = 7" > /etc/sysfs.conf`
## 修改内核参数
这里vfs就是虚拟显卡的数量，最大多少我没查，这里我设置的是3。
```
intel_iommu=on i915.enable_guc=3 i915.max_vfs=3
```
- grub
修改 `/etc/defautl/grub` 添加到 `GRUB_CMDLINE_LINUX_DEFAULT="quite"`这句里面
- Systems-boot
我用的zfs 所以修改是这种方式
`/etc/kernel/cmdline`修改成
```
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
```
## 更新内核和 initramfs
`update-initramfs -u -k all`
`proxmox-boot-tool refresh`
## 锁定内核
**由于最新的内核没有headers提供，为了避免更新把这个内核顶了，我们要锁定目前编译好的内核！**
​ `# proxmox-boot-tool kernel pin 6.2.16-3-pve`
## 重启
`reboot`
## 检查是否显卡 SR-IOV是否成功
`lcpci |grep -i vga`
**输出如下，就是成功了！**
```
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.1 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.2 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.3 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
```
**或者如下输出，看到我的已经开启了3个虚拟显卡**
```
# dmesg | grep i915
[ 0.000000] Command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
[ 0.077241] Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
[ 4.811959] i915 0000:00:02.0: Running in SR-IOV PF mode
[ 4.812864] i915 0000:00:02.0: [drm] VT-d active for gfx access
[ 4.813302] i915 0000:00:02.0: vgaarb: deactivate vga console
[ 4.813362] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[ 4.813758] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 4.814738] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[ 4.815136] i915 0000:00:02.0: Direct firmware load for i915/adlp_dmc.bin failed with error -2
[ 4.817242] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
[ 4.822141] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/tgl_guc_70.bin version 70.5.1
[ 4.822154] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[ 4.827027] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads!
[ 4.827569] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[ 4.827574] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[ 4.827921] i915 0000:00:02.0: [drm] GuC RC: enabled
[ 4.828778] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[ 4.828956] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[ 4.854926] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[ 4.856871] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[ 4.857105] i915 0000:00:02.0: 3 VFs could be associated with this PF
[ 4.887664] fbcon: i915drmfb (fb0) is primary device
[ 4.972789] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[ 5.638917] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[ 5.639816] i915 0000:00:02.1: enabling device (0000 -> 0002)
[ 5.640648] i915 0000:00:02.1: Running in SR-IOV VF mode
[ 5.641809] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.643254] i915 0000:00:02.1: [drm] VT-d active for gfx access
[ 5.644017] i915 0000:00:02.1: [drm] Using Transparent Hugepages
[ 5.645371] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.646658] i915 0000:00:02.1: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[ 5.647370] i915 0000:00:02.1: HuC firmware PRELOADED
[ 5.650198] i915 0000:00:02.1: [drm] Protected Xe Path (PXP) protected content support initialized
[ 5.650917] i915 0000:00:02.1: [drm] PMU not supported for this GPU.
[ 5.651749] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.1 on minor 1
[ 5.656497] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[ 5.657170] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 5.657938] i915 0000:00:02.2: enabling device (0000 -> 0002)
[ 5.658678] i915 0000:00:02.2: Running in SR-IOV VF mode
[ 5.659806] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.661509] i915 0000:00:02.2: [drm] VT-d active for gfx access
[ 5.662199] i915 0000:00:02.2: [drm] Using Transparent Hugepages
[ 5.663417] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.665322] i915 0000:00:02.2: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[ 5.665956] i915 0000:00:02.2: HuC firmware PRELOADED
[ 5.668308] i915 0000:00:02.2: [drm] Protected Xe Path (PXP) protected content support initialized
[ 5.668947] i915 0000:00:02.2: [drm] PMU not supported for this GPU.
[ 5.669666] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.2 on minor 2
[ 5.673931] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[ 5.674572] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=none
[ 5.675219] i915 0000:00:02.2: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[ 5.675964] i915 0000:00:02.3: enabling device (0000 -> 0002)
[ 5.676640] i915 0000:00:02.3: Running in SR-IOV VF mode
[ 5.677500] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.678512] i915 0000:00:02.3: [drm] VT-d active for gfx access
[ 5.679186] i915 0000:00:02.3: [drm] Using Transparent Hugepages
[ 5.680354] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0
[ 5.681176] i915 0000:00:02.3: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[ 5.681765] i915 0000:00:02.3: HuC firmware PRELOADED
[ 5.684137] i915 0000:00:02.3: [drm] Protected Xe Path (PXP) protected content support initialized
[ 5.684777] i915 0000:00:02.3: [drm] PMU not supported for this GPU.
[ 5.685487] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.3 on minor 3
[ 5.686338] i915 0000:00:02.0: Enabled 3 VFs
```
# 感谢阅读
今天就到这里了，至于虚拟机如何使用这个显卡，后续再更新！
Reference in New Issue View Git Blame Copy Permalink