MPG Z690 Carbon System hang/freeze when installing graphics drivers or running DeviceDiagnostic

[spatula]

Really at a loss to figure this one out. After experiencing some mystery freezes of my i9-12900K system every few weeks, I decided to update to the latest drivers.

Every time I attempt to install the graphics drivers for my MPG Z690 Carbon Wifi on-board graphics on Windows 11 Pro (version 21H2, Build 22000.675) using the installer, a few seconds after the installation starts, the whole system freezes. If there was disk activity at the time, the HDD LED in the case will get stuck on / permanently lit. About half the time, the system will bluescreen/BSOD with an error code of CLOCK_WATCHDOG_TIMEOUT, but unfortunately because the system is in a hung state, it is not successful at producing a crash dump (it just stays at 0% forever). The rest of the time, the system just freezes with the contents of the screen in place. Using the ctrl-scroll-lock feature to try to force a crash dump is not successful.

This happens both with the drivers from the MSI site, and those directly from the Intel site. It happens in both regular and safe mode. I am not doing (and have not done) any overclocking of any kind.

sfc /scannow, dism /online /cleanup-image /scanhealth, sigverif, and chkdsk all fail to find any problems.

The only driver install experiencing this problem is the graphics driver - installing the management engine, chipset, RAID, network, etc., all worked beautifully.

Curiously, if I simply remove the graphics driver entirely, have Windows delete the driver files, and reboot, Windows successfully downloads and installs an Intel graphics driver after starting up (version 30.0.101.1340, dated 2/3/2022). This does NOT freeze the system, at least not immediately.

I've also done an over-the-top reinstall of Windows 11 just to rule out OS-related problems despite nothing showing up with the above tools, but this did not change anything. I've also updated to the latest BIOS, 7D30v14. I've run the paid version of Memtest86 for 16 hours and found no problems. Cinebench fails to produce any issues with a 20-minute multithreaded test. Similarly, prime95 doesn't demonstrate any problems with the CPU.

Additionally, I have found that running the Windows hardware troubleshooter ( msdt.exe -id DeviceDiagnostic ) will cause the system to hang in a similar way when it begins scanning for hardware changes.

Any ideas of what else I should be checking or trying?

 

[citay]

I have a very strong suspicion about the cause of this. It looks to be another case of that dreaded ASMedia SATA Controller chip (ASMedia ASM1061) causing problems.
MSI use this chip to provide two of the SATA ports (A and B), but something about the chip keeps causing these kinds of problems and BSODs.

Read here and follow the thread i linked there: https://forum-en.msi.com/index.php?...wifi-ddr5-bsod-clock-watchdog-timeout.376077/

 

[spatula]

I have a very strong suspicion about the cause of this. It looks to be another case of that dreaded ASMedia SATA Controller chip (ASMedia ASM1061) causing problems.
MSI use this chip to provide two of the SATA ports (A and B), but something about the chip keeps causing these kinds of problems and BSODs.

Read here and follow the thread i linked there: https://forum-en.msi.com/index.php?...wifi-ddr5-bsod-clock-watchdog-timeout.376077/

Thanks for the tip! The failure mode does seem similar. I have one device, and old BD-ROM, plugged into a SATA port, though I'm not sure which one offhand. The other drives are both SSDs on M.2 ports (configured as a RAID1 in VMD) so I could certainly move the BD-ROM do a different port if need be.

What's especially curious is that this is a relatively new problem; I'm sure I installed the graphics drivers successfully in the past. But I've also updated the BIOS at least once since then (from v10 to v13, then from v13 to v14 yesterday), so this could plausibly be a bug introduced between v10 and v13. I'll try your instructions to disable the external SATA controller if it's possible to do so on this board and see if that helps.

 

[spatula]

I have a very strong suspicion about the cause of this. It looks to be another case of that dreaded ASMedia SATA Controller chip (ASMedia ASM1061) causing problems.
MSI use this chip to provide two of the SATA ports (A and B), but something about the chip keeps causing these kinds of problems and BSODs.

Read here and follow the thread i linked there: https://forum-en.msi.com/index.php?...wifi-ddr5-bsod-clock-watchdog-timeout.376077/

Confirmed I had the option to disable the external chip, confirmed I did in fact have the BD-ROM plugged into either SATAA or SATAB (it disappeared after disabling the controller, so I'll just have to move it to some other port), and confirmed that after disabling the ASMedia chip, I can now successfully get through running msdt.exe -id DeviceDiagnostic (and its probe for new hardware) without freezing the system.

If I had to guess, I'd guess that the Intel Graphics driver installer also initiates a new hardware probe (indiscriminately I'd wager, maybe in case you had multiple graphics devices?), and it's that probe which causes the system to hang when it gets to probing SATA devices.

I guess the good news is there's a very easy workaround for those of us who don't have a lot of SATA devices. The bad news is if you don't disable that chip, your system may come to a screeching halt any time some installation, automatic or otherwise, triggers a hardware probe. Definitely something that MSI needs to address.

Can't tell you how much I appreciate the advice. I would never in a million years have guessed that an external SATA controller chip could be the culprit.

 

[citay]

You're welcome. Yeah, it's something that you will not immediately connect as being in some way related, but i can spot these issues from a mile away by now. The CLOCK_WATCHDOG_TIMEOUT is characteristic for it, and then when you have a board model that relies on that ASMedia chip for two additional SATA ports (instead of using all six from the chipset in the case of Z690), then it's pretty much an open-and-shut case for me.

I would advise you to report this problem to MSI, even though you have a workaround now. Only constant user feedback about this problem can make them address this, i reckon.

 

[spatula]

You're welcome. Yeah, it's something that you will not immediately connect as being in some way related, but i can spot these issues from a mile away by now. The CLOCK_WATCHDOG_TIMEOUT is characteristic for it, and then when you have a board model that relies on that ASMedia chip for two additional SATA ports (instead of using all six from the chipset in the case of Z690), then it's pretty much an open-and-shut case for me.

I would advise you to report this problem to MSI, even though you have a workaround now. Only constant user feedback about this problem can make them address this, i reckon.

Agreed, and I just opened a ticket for them detailing the symptoms and the workaround. It's curious why they'd even bother using an external controller chip when the PCH supports 8 SATA ports on its own, though I also read that using some of those ports takes away a PCIe lane. Or maybe this is a case of MSI not innovating unnecessarily, and adapting a strategy used with older chipsets for use with the Z690 PCH?

Either way, one more ticket is open for them now relating this problem. It's so easy to reproduce it that it seems like something that should be fixable if only they prioritize it.

 

[citay]

I don't really know the reasons for relying on the ASMedia chip, when they should've had enough PCIe lanes and native SATA ports to go around on the Z690 boards.
On the B660 boards i understand it, only four native SATA ports from the chipset. But Z690? Beats me. The boards should be completely new developments too.

I don't see any "sister boards" between B660 and Z690 either. The only two boards with an identical PCB are the MSI Pro Z690-A WIFI DDR4 and the MSI MAG H670 Tomahawk WIFI DDR4.
But the H670 chipset has up to eight SATA ports again, and both boards don't use the ASMedia chip.

Looking at other manufacturers:
Gigabyte: B660 boards all have four SATA ports from the chipset, Z690 all have six SATA ports from the chipset.
ASUS: B660 boards all have four SATA ports from the chipset, Z690 either have four or six SATA ports from the chipset, depending on the board model (Z690 DDR4 boards often have 4, not 6).
ASRock: B660 boards all have four SATA ports from the chipset, Z690 either have four, six, or eight (!) SATA ports from the chipset, depending on the board model.

None use an ASMedia chip for additional SATA ports, only MSI do.

 

[spatula]

I don't really know the reasons for relying on the ASMedia chip, when they should've had enough PCIe lanes and native SATA ports to go around on the Z690 boards.
On the B660 boards i understand it, only four native SATA ports from the chipset. But Z690? Beats me. The boards should be completely new developments too.

[snip]

None use an ASMedia chip for additional SATA ports, only MSI do.

Doing a little searching, one fascinating thing is that people have been reporting similar problems with ASMedia SATA controller chips going back as far as 2015. Apparently Asus used to use them way back when, and people back then were advising against plugging anything into the ASMedia-driven SATA ports for the exact same symptoms as we're discussing here. And it's not just Windows either- people report similar problems on Linux; evidently when they use a "16-byte ATA pass through command to identify ATAPI cdrom drives" it hangs the chip; disabling that command resolves it for Linux.

This strikes me as something that might have to be resolved with controller firmware, unless what's actually hanging up is the BIOS somehow.

 

[citay]

Interesting. It's puzzling how such a buggy behaviour survived unfixed all this time in the ASMedia controllers. But everyone who suffers from this should report it to the mainboard makers, only they have the leverage to push ASMedia for a bugfix, should it be a firmware issue.

 

[Svet]

Have a try the new beta for your board that can help, from google forum site here:

https://forum-en.msi.com/index.php?threads/msi-forum-moderator-tool-space-%E2%80%93-bios-software.374208/

then click on "Asmedia Port BSOD", and select the right BIOS for your board.

 

[spatula]

I'd like to do that, however, I need this machine to stay up and stable. I can't ensure that by shutting it down and playing with beta-quality BIOSes. The workaround of disabling the external controller and not using SATA-A or B ports is fine for me. Hopefully MSI can solve it for others who need to use more SATA ports or who want/need to use the external SATA controller for some reason.