I have a similar setup. HDMI to TV and optical to my AVR. Normally I have it set to optical which runs the signal through my AVR only, but for quieter times I switch the audio output to “stereo” which will run it through both (if I choose). I then leave the AVR off and use the volume on the TV.
It’s a bit of a pain because you cannot do this on the fly and have to switch prior to entering Netfiix or any video, but it is a solution to keep the kiddies sleeping.