I've been using this older board with an 8th gen i7 and 64GB RAM as a Linux server for years with no issues. For the past couple of years or so I've used an Asus XG-C100C 10GBe NIC with the Aquantia chipset, which requires x4 lanes for 10GB, in slot PCI_E6 (which is rated at x4). It's been sustaining 10GBe throughput to my NAS and workstation PC with no issues.
I had no discrete GPU in this server as it is headless (well, other than a PiKVM connected to onboard video). However since I've started playing with local LLMs and had a spare RTX 2070 Super GPU sitting on the shelf collecting dust, I decided to install it for some lightweight LLM workloads. It's only 8GB of VRAM but that's better than zero dedicated VRAM.
When I plugged the GPU into PCI_E1, the Asus NIC on PCI_E6 stopped working. The OS recognized it (I could see it with lspci and it showed up as a NIC with ip a) but it would not even establish link much less negotiate an IP. LEDs on both the NIC and switch ports were dead. So I tried it in slot PCI_E4, the only other slot capable of x4, and it works fine. But this of course downgrades the GPU to run at x8 instead of x16.
From everything I'm seeing in the manual and online, the three x16 slots E1/E4/E6 should run either as x16/x0/x4 (which is what I want) or x8/x8/x4. Nothing I'm finding says that E6 won't work with E1 populated. In fact, E6 uses PCH lanes while E1 & E4 use CPU lanes, so E6 should be entirely isolated from E1/E4.
It's not a showstopper as x16 vs x8 isn't as significant for LLM as it is for graphics, but still I'd like to squeeze out what little extra performance I can. It just bugs me that this *should* work but doesn't.
Does anyone have an idea why a NIC in E6 won't negotiate link when E1 is populated, even though the OS still recognizes the NIC?
I had no discrete GPU in this server as it is headless (well, other than a PiKVM connected to onboard video). However since I've started playing with local LLMs and had a spare RTX 2070 Super GPU sitting on the shelf collecting dust, I decided to install it for some lightweight LLM workloads. It's only 8GB of VRAM but that's better than zero dedicated VRAM.
When I plugged the GPU into PCI_E1, the Asus NIC on PCI_E6 stopped working. The OS recognized it (I could see it with lspci and it showed up as a NIC with ip a) but it would not even establish link much less negotiate an IP. LEDs on both the NIC and switch ports were dead. So I tried it in slot PCI_E4, the only other slot capable of x4, and it works fine. But this of course downgrades the GPU to run at x8 instead of x16.
From everything I'm seeing in the manual and online, the three x16 slots E1/E4/E6 should run either as x16/x0/x4 (which is what I want) or x8/x8/x4. Nothing I'm finding says that E6 won't work with E1 populated. In fact, E6 uses PCH lanes while E1 & E4 use CPU lanes, so E6 should be entirely isolated from E1/E4.
It's not a showstopper as x16 vs x8 isn't as significant for LLM as it is for graphics, but still I'd like to squeeze out what little extra performance I can. It just bugs me that this *should* work but doesn't.
Does anyone have an idea why a NIC in E6 won't negotiate link when E1 is populated, even though the OS still recognizes the NIC?
