The difference between AC3 and AC3(digital) is not a tag, and both are digital ofc, otherwise they wouldn’t be in a digital file to begin with. The only difference is the number of channels in the track, the codec is exactly the same. When you see AC3 it’s just a stereo track (2.0), while AC3(digital) is for a 5.1 track: tbh it would be nice if WD would change it to AC3 (2.0) for stereo and AC3 (5.1) or AC3 (5.1 EX) for surround mixes, it would clear things up a bit.
If the dialogs sound muddied it’s because in a surround track the only channel that contains the dialog in a inteligible form is the center channel ©. The front channels (L/R) only have the reverberations on the walls etc… of the dialogs to give more ambience, so if you are only hearing this it means that you are only listening to the (L/R) channels of the 5.1 mix.
The problem could be that the decoder in your speakers set is applying some postprocessing mode when it shouldn’t.
Check if there is any sound coming out of the center speaker (not just static), if there is some sound, even if it is a bit low and distorted, it could well be that the speakers are forcing an upmix mode or something. It should also clearly say something like 5.1 or 3/2.1 when you play those AC3(digital) compared to the track that say AC3, that should display as 2.0 (or stereo) on that little display. Again, make sure the speakers are not set as prologic, matrix, upmix or anything of the sort when you play a 5.1 track.
In order to troubleshoot both ends it would be nice if you could plug in the WDTV in an A/V receiver or plug a different audio source with optical output in the Z5500, if any of them are set wrong it would show with other devices as well.