For 5.1 AAC to work, your AVR must support it ie. be able to Decode it. (consult your AVR user manual) what brand & model ?
If your AVR supports it then select Setup > Audio/Video Ouput > Audio > HDMI/Optical Passthrough > AAC capable reciever
If your AVR doesent support it, un-check AAC capable reciever … output will only be 2 Channel Stereo
(My Audio gear doesent support AAC, so i convert it to AC3 … or sometimes, i chose the “Emulate” setting on my Audio reciever to simulate 5.1 from a stereo source)
Here’s some links regarding AAC
http://community.wd.com/t5/forums/searchpage/tab/message?filter=labels&q=5.1+AAC
If you want to convert AAC to AC3 here’s my guide: http://community.wd.com/t5/WD-TV-Live-Streaming/Full-Digital-Audio-Pass-through-via-HDMI-No-Decoding-on-WD-TV/m-p/785503#M32678