Massive latency problems

[#1]

I normally notice this at night whilst playing games, at random times my latency goes through the roof (1500-3500+ms when the normal average is around 60) and the packet loss on VOIP becomes massive (often over 70% packet loss - normally at 0.01%).
These always occur together and make playing games and using VOIP nigh on impossible during that time.
Often the rest of the family does not notice as they are normally streaming programmes or using Facebook where the latency problem is not apparent.
The issue is not related to peak times within the household as this even happens when nobody else uses the network.
This has been happening since we got Plusnet broadbad and was not previously an issue when with BT.
Calls have been raised in the past about network drop outs and loss of connection but this has been somewhat resolved but the latency issue has not and remains a massive problem.
EDIT: I take back the bit about the network dropouts, it just dropped out only to come back again almost instantly...

[#2]

Hi,
This could possibly be caused by uploads.  You don't say what service you're on, or what equipment, but for just about all broadband services the upload speed is only a fraction of the download speed so is very easy to max it out.    When this happens some types of home routers buffer the traffic, causing the sort of latency you're seeing.  This impacts other services as various small packets like your VoIP, or DNS lookups or TCP acks for downloads get stuck at the back of the queue.  Reportedly the Plusnet 2704N is bad in this respect.
It can be hard to track down utilisation on a family network, with various devices, but you could have something like iCloud backup, or Google Drive sync or similar that runs in the background.  On a Mac  you can see current network activity via Activity Monitor, or Resource Monitor in Windows 7.  Or the brute force approach would be to shut off certain devices when the problem is occurring, to see whether it improves things.
Like you, I didn’t experience this issue prior to moving to Plusnet so I think there may be a weakness somewhere in Plusnet’s system.  I am 90% sure I’ve had the issue when our upload was only running at 195K (i.e. less than half), and a suspicion of the same happening at only 90K.  When I get a decent router and can get actual live figures I’ll do some more testing.  Meanwhile it might be worth your while making a note of which Plusnet gateway you are connecting via (look for the first Plusnet hop on a traceroute).
Hope this helps,
Tony