For sure you need to be set on “Stereo” – digital won’t work for you since your TV can’t decode it (not even the AC3 file – you need an audio receiver for that).
The Live *should* correct decode both DTS and AC3 as long as they are correctly encoded, but that isn’t always the case. You didn’t say how you encoded with Handbrake, or what version you used. I would recommend downloading one of the nightlies (NOT the old version .094) and encode with the High Profile, changing only output type to MKV (leave everything else alone and for sure don’t mess with the sound settings). The default will convert your audio to AAC which is what you want here (since you can’t decode 5.1 surround).
That should do it – I’ll bet you a dollar everything will be just fine then.
