AHA! It was ADC bailout causing the problems, not the actual refraction.
Turns out that two radiosity bounces get you just above the default adc
bailout. Add a little reduction by non-transparent media and it's dead.
turning it down ridiculously small fixed it.