I have the same situation with MKV files. I tried all kind of differnet encoding combinations in Handbrake 0.9.6 but I still couldn’t create an MKV that would play all the way through on the WD-SMP without freezing up.
I dropped back to Handbrake 0.9.5 and all is well again.
I don’t know whether it is Handbrake or WD that is causing the incompatability, but at least 0.9.5 works.