If you're not on Fibre Extra then yes, it's likely you'll be moderately QoSed at peak times. I'm on Extra and have noticed occasional slowdown but that's not necessarily due to PN QoS, more likely peering contention or just a "bad" pool of YT CDN hosts.
YouTube's CDN is structured moderately randomly. The idea is that they all hang off *.c.youtube.com records, and you connect to one presumably through a set of qualifiers (IP geolocation, momentary load, preferred peering points etc).
The best (well, quickest) way to get round this is to eliminate the stream hosts that don't work well for you at your preferred quality levels. Follow the steps in this reddit post to effectively null route "bad" stream hosts, forcing the YouTube player to connected to another (hopefully reliable) stream host: