Most likely you have OC'ed too much. Try testing it out with Ati Tool's artifact scanner and see if you get any error (Do not trust ATI's Overdrive). You can get Ati Tool here http://www.techpowerup.com/wizzard/ATITool_0.27b3.exe
By the way what's your GPU's temperature?
You can monitor your GPU temp by using rivatuner from here http://downloads.guru3d.com/download.php?det=163
or through ATI's CCC.
I'm also currently using a Radeon HD3870 (although its a different brand) but I've managed to OC it up to 860 Core and 1210 memory without any issue. My unit is however equiped with a custom cooler. As I understand the stock cooler on the 3870 is notoriously hot!!!
I've checked on ATi's forums and found out that the HD3xxx cards are having fan control issues. Mostly they are reported to be stuck at 25% or 30% even under full load which helps explain the high temperature and could possibly lead to occasional blackouts. It is possible for you to use RivaTuner to change the fan's speed to a higher fixed percentage 70% to 100% to see if the card will run smoothly