If you're not on Fibre Extra then yes, it's likely you'll be moderately QoSed at peak times. I'm on Extra and have noticed occasional slowdown but that's not necessarily due to PN QoS, more likely peering contention or just a "bad" pool of YT CDN hosts.
YouTube's CDN is structured moderately randomly. The idea is that they all hang off *.c.youtube.com records, and you connect to one presumably through a set of qualifiers (IP geolocation, momentary load, preferred peering points etc).
The best (well, quickest) way to get round this is to eliminate the stream hosts that don't work well for you at your preferred quality levels. Follow the steps in this reddit post to effectively null route "bad" stream hosts, forcing the YouTube player to connected to another (hopefully reliable) stream host:
https://www.reddit.com/r/youtube/comments/3oog8e/suggestion_for_buffering_issues/
Someone also suggested (wisely) that you report "bad" peering arrangements direct to someone who works at YouTube so they can chase this up with BT/PN/intermediary hop network providers.
https://www.reddit.com/r/youtube/comments/3n6w72/my_youtube_is_slow_how_do_i_fix_it/
You might also be unlucky and be connecting to a congested PN gateway, so try the usual tricks to change gateway (quickest is to reboot router AIUI) and/or report it to PN,
