Hmmm. I don’t see that at all.
I just set up a low prioritization rule on Port 80 (Web), but it didn’t effect any results from www.speedtest.net at all until I started an FTP upload to my webserver.
I added a high priority rule for FTP.
Then the FTP took more bandwidth, and the Speedtest result went almost to 0.
… are you sure you have the rules configured correctly?
By the way, you need to keep in mind that the QoS rules affect UPLOAD more than DOWNLOAD.