ob/pve
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
dev
pve/N100.md
T
ob 0801b82999 add some about n100 file
2023-11-19 20:36:18 +08:00

11 KiB
Raw Permalink Blame History

N100 玩转指南

  • 安装PVE
  • 搞定硬件直通
  • 十二代CPU的显卡 SR-IOV

安装PVE

制作安装介质

  • 下载 PVE 最新版 iso

  • 制作安装介质

    • GNU Linux

      # dd bs=1M conv=fdatasync if=./proxmox-ve_*.iso of=/dev/XYZ

    • MacOS

      打开Terminal

      转换 .iso 文件到 .dmg 用这个 hdiutil

      # hdiutil convert proxmox-ve_*.iso -format UDRW -o proxmox-ve_*.dmg

      获取磁盘序列

      # diskutil list

      卸载需要写入的U盘

      # diskutil list
# diskutil unmountDisk /dev/diskX

      dd 制作启动盘

      # sudo dd if=proxmox-ve_*.dmg bs=1M of=/dev/rdiskX
      • Windows

​ 用 rufus 工具

  • 开始安装
    • BIOS 设置U盘启动
    • 进入启动菜单
      • 图形安装方式
      • Console安装方式

开启硬件直通

  • 添加PVE源 apt list

    echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
			>	/etc/apt/sources.list.d/pve.list
apt update

  • BIOS 相关设置

  • 开启VT-d - IOMMU

  • VT-x

  • 添加 kernel cmdline

    • Grub 设置

      编辑 /etc/default/grub

      GRUB_CMDLINE_LINUX_DEFAULT="quite"

      GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream"

    • kernel 内核模块设置

      #!/bin/sh
cat >> /etc/modules-load.d/vfio.conf << EOF
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd 
EOF

      • 添加相应参数

        添加 参数到 /etc/modprobe.d/vfio.conf

        echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" \
> /etc/modprobe.d/vfio.conf

      • 更新 grub & initramfs

        # update-initramfs -u -k all
# update-grub
# proxmox-boot-tool refresh

      • 检查IOMMU 是否设置成功

        dmesg | grep -e DMAR -e IOMMU

        输出类似如下

        [    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.007697] ACPI: DMAR 0x0000000079195C18 0000A8 (v01 INTEL  EDK2     00000002      01000013)
[    0.007741] ACPI: Reserving DMAR table memory at [mem 0x79195c18-0x79195cbf]
[    0.130354] DMAR: IOMMU enabled
[    0.388873] DMAR: Host address width 39
[    0.389091] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.389318] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e

        DMAR: IOMMU enabled 注意输出有这个就是成功了！

      • 由于N100主机只支持 EFI启动，我遇到最大的坑就在这里

        我用的是zfs文件系统，PVE在EFI启动过程使用的是systemd-boot，没有使用grub，所以上面添加 kernel cmdline的方法是不管用的，需要修改loader.conf

      • Systems-boot kernel cmdline 修改

        修改 /etc/kernel/cmdline

        # cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction

        # update-initramfs -u -k all

显卡SR-IOV

安装内核工具

apt install build-* dkms

下载内核补丁

注意：内核补丁还在测试，不建议用在生产环境

下载github源码 i915-sriov-dkms

进入下载的目录，修改PACKAGE_NAME to i915-sriov-dkms, 修改 PACKAGE_VERSION to 6.2. 保存文件。

目前PVE的内核用的是 6.2

# uname -ar
Linux n100 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) x86_64 GNU/Linux

拷贝这个目录到 /usr/src/i915-sriov-dkms-6.2

安装dkms模块

dkms -i -m i915-sriov-dkms -v 6.2

安装 sysfsutils

echo "devices/pci0000:00/0000:00:02.0/sriov_numvfs = 7" > /etc/sysfs.conf

修改内核参数

这里vfs就是虚拟显卡的数量，最大多少我没查，这里我设置的是3。

intel_iommu=on i915.enable_guc=3 i915.max_vfs=3

  • grub

    修改 /etc/defautl/grub 添加到 GRUB_CMDLINE_LINUX_DEFAULT="quite"这句里面

  • Systems-boot

    我用的zfs 所以修改是这种方式

    /etc/kernel/cmdline修改成

    root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction

更新内核和 initramfs

update-initramfs -u -k all

proxmox-boot-tool refresh

锁定内核

由于最新的内核没有headers提供，为了避免更新把这个内核顶了，我们要锁定目前编译好的内核！

​ # proxmox-boot-tool kernel pin 6.2.16-3-pve

重启

reboot

检查是否显卡 SR-IOV是否成功

lcpci |grep -i vga

输出如下，就是成功了！

00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.1 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.2 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]
00:02.3 VGA compatible controller: Intel Corporation Alder Lake-N [UHD Graphics]

或者如下输出，看到我的已经开启了3个虚拟显卡

# dmesg | grep i915
[    0.000000] Command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
[    0.077241] Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3 pci_pt_e820_access=on pci=assign-busses pcie_acs_override=downstream,multifunction
[    4.811959] i915 0000:00:02.0: Running in SR-IOV PF mode
[    4.812864] i915 0000:00:02.0: [drm] VT-d active for gfx access
[    4.813302] i915 0000:00:02.0: vgaarb: deactivate vga console
[    4.813362] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[    4.813758] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    4.814738] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[    4.815136] i915 0000:00:02.0: Direct firmware load for i915/adlp_dmc.bin failed with error -2
[    4.817242] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
[    4.822141] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/tgl_guc_70.bin version 70.5.1
[    4.822154] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[    4.827027] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads!
[    4.827569] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[    4.827574] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[    4.827921] i915 0000:00:02.0: [drm] GuC RC: enabled
[    4.828778] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[    4.828956] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    4.854926] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[    4.856871] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[    4.857105] i915 0000:00:02.0: 3 VFs could be associated with this PF
[    4.887664] fbcon: i915drmfb (fb0) is primary device
[    4.972789] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[    5.638917] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[    5.639816] i915 0000:00:02.1: enabling device (0000 -> 0002)
[    5.640648] i915 0000:00:02.1: Running in SR-IOV VF mode
[    5.641809] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0
[    5.643254] i915 0000:00:02.1: [drm] VT-d active for gfx access
[    5.644017] i915 0000:00:02.1: [drm] Using Transparent Hugepages
[    5.645371] i915 0000:00:02.1: [drm] GT0: GUC: interface version 0.1.0.0
[    5.646658] i915 0000:00:02.1: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[    5.647370] i915 0000:00:02.1: HuC firmware PRELOADED
[    5.650198] i915 0000:00:02.1: [drm] Protected Xe Path (PXP) protected content support initialized
[    5.650917] i915 0000:00:02.1: [drm] PMU not supported for this GPU.
[    5.651749] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.1 on minor 1
[    5.656497] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    5.657170] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.657938] i915 0000:00:02.2: enabling device (0000 -> 0002)
[    5.658678] i915 0000:00:02.2: Running in SR-IOV VF mode
[    5.659806] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0
[    5.661509] i915 0000:00:02.2: [drm] VT-d active for gfx access
[    5.662199] i915 0000:00:02.2: [drm] Using Transparent Hugepages
[    5.663417] i915 0000:00:02.2: [drm] GT0: GUC: interface version 0.1.0.0
[    5.665322] i915 0000:00:02.2: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[    5.665956] i915 0000:00:02.2: HuC firmware PRELOADED
[    5.668308] i915 0000:00:02.2: [drm] Protected Xe Path (PXP) protected content support initialized
[    5.668947] i915 0000:00:02.2: [drm] PMU not supported for this GPU.
[    5.669666] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.2 on minor 2
[    5.673931] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=io+mem
[    5.674572] i915 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=none
[    5.675219] i915 0000:00:02.2: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.675964] i915 0000:00:02.3: enabling device (0000 -> 0002)
[    5.676640] i915 0000:00:02.3: Running in SR-IOV VF mode
[    5.677500] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0
[    5.678512] i915 0000:00:02.3: [drm] VT-d active for gfx access
[    5.679186] i915 0000:00:02.3: [drm] Using Transparent Hugepages
[    5.680354] i915 0000:00:02.3: [drm] GT0: GUC: interface version 0.1.0.0
[    5.681176] i915 0000:00:02.3: GuC firmware PRELOADED version 1.0 submission:SR-IOV VF
[    5.681765] i915 0000:00:02.3: HuC firmware PRELOADED
[    5.684137] i915 0000:00:02.3: [drm] Protected Xe Path (PXP) protected content support initialized
[    5.684777] i915 0000:00:02.3: [drm] PMU not supported for this GPU.
[    5.685487] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.3 on minor 3
[    5.686338] i915 0000:00:02.0: Enabled 3 VFs

感谢阅读

今天就到这里了，至于虚拟机如何使用这个显卡，后续再更新！

Reference in New Issue View Git Blame Copy Permalink