Drops in noise margin - but only on downstream

[#1]

Hello all,
I'm having a problem where the downstream noise margin occasionally drops resulting in slow data throughput and high errors (routerstats graphs attached).
It's not dropping far enough to cause sync loss, but it causes problems for my Skype calls as packets are clearly being lost.
I'm going to try and resolve this myself, but I was wondering:
1) Is it unusual to see noise margin loss on just the downstream? Would it normally also drop on the upstream?
2) Does anyone have any suggestions on things I could look into to find the source of the noise?
Thanks all.

[#2]

First thing to check is whether you have something switching on and off at those times - perhaps something on a thermostat

[#3]

Thanks for your reply. I've already done some checks, but unfortunately I'm in a shared building so the interference could be coming from a lot of different places.
I'm going to try the AM radio trick to see if that helps me identify the source.
Does anyone else have any tips on how I might be able to track down the source of the interference?
Thanks.