Reporting back here... Been running a few hours now after I sent the card back to my retailer for them to check it out for possible refund (or exchange, but given stock is basically null, it was going to be a refund if they can't figure this out) and getting it back after told me this is a driver issue.
Funny enough, they were referencing this forum to me... so here I am. From my last post last Friday and this thread exploding to 9 pages from 1 page.
They did testing and they told me to install the 20.11.2 (November 13 2020) drivers and it should go away. Seems to be right so far and haven't had the issue, yet. The 20.11.2 drivers are also the WHQL certified drivers (certified by MSFT and validated by their Windows team) so they said it should be the most stable drivers until a newer WHQL certified driver comes out.