cancel
Showing results for 
Search instead for 
Did you mean: 

Odd TBB test ?

MrSilver
Pro
Posts: 550
Thanks: 82
Fixes: 9
Registered: ‎05-10-2016

Re: Odd TBB test ?

Thhe pcn bngs have been on the whole much better for me, if you want to check its fixed dare to connect to a psb gateway? 😀
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,869
Thanks: 4,950
Fixes: 315
Registered: ‎04-04-2007

Re: Odd TBB test ?

Right, findings from last night...

In summary, very similar to what @MrSilver found here.

I ran the script five times for > 45m each time (slightly modified script mind you to remove the attempts to delete .mp4 files ???).

After each test I dropped PPP session so as to reconnect on a different endpoint. I also ran two tests across PTA rather than L2TP.

Note that this is from a Raspberry Pi at the end of a 200Mbps 10/100 Powerline link which limits my throughput (I can get > 8MB/s from other parts of my network, this link runs up to ~3.5MB/s).

Full results attached, prevalent info below:

  1. First run: L2TP
    Time: 17:01-17:43
    Test volume: 30 tests
    File: results@201612291701-pcnbng04.txt
    Gateway: lo0.central10.pcn-bng04.plus.net [195.166.130.251]
    Notable results (these are all files from TBB):-
    output:2016-12-29 17:28:26 (111 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 17:33:46 (106 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 17:37:59 (105 KB/s) - '20MB.zip' saved [20971520/20971520]

  2. Second run: L2TP
    Time: 18.08-18:49
    Test volume: 24 tests
    File: results@201612291804-psbbng03.txt
    Gateway: lo0.central10.psb-bng03.plus.net [195.166.130.254]
    Notable results (again, they're all files downloaded from TBB's servers):-
    output:2016-12-29 18:07:31 (111 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 18:13:50 (105 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 18:21:55 (106 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 18:36:05 (107 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 18:44:35 (113 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 18:49:56 (106 KB/s) - '20MB.zip' saved [20971520/20971520]

  3. Third run: PTA
    Time: 19:09-19:57
    Test volume: 43
    File: results@201612291909-pcn-ir01.txt
    Gateway: out - be3-3102.pcn-ir02.plus.net [195.166.143.132] in - be10.pcn-ir01.plusnet.bt.net (195.99.125.133)
    Notable results:-
    None.

  4. Fourth run: PTA
    Time: 20:12-21:40
    Test volume: 74 tests
    File: results@201612291804-pcn-ir01(2).txt
    Gateway: out - be3-3102.pcn-ir02.plus.net [195.166.143.132] in - Huh (unresponsive to pings)
    Notable results (These are from exa.net.uk rather than TBB. I'm inclined to say they are red herrings, as they don't fit the pattern, and the other throughput results for this source are pretty low too):-
    output20mExaRandom:2016-12-29 20:44:07 (79.6 KB/s) - '20M-random.bin' saved [25042/25042]
    output20mExaRandom:2016-12-29 21:19:31 (112 KB/s) - '20M-random.bin' saved [25042/25042]
    output20mExaRandom:2016-12-29 21:20:42 (89.9 KB/s) - '20M-random.bin' saved [25042/25042]
    output20mExaRandom:2016-12-29 21:23:07 (90.8 KB/s) - '20M-random.bin' saved [25042/25042]
    output20mExaRandom:2016-12-29 21:26:32 (117 KB/s) - '20M-random.bin' saved [25042/25042]

  5. Fifth run: L2TP
    Time: 21:49-23:11
    Test volume: 58 tests
    File: results@201612292149-psb-bng02.txt
    Gateway: lo0.central10.psb-bng02.plus.net [195.166.130.253]
    Notable results (mostly TBB, couple from Exa again):-
    output:2016-12-29 21:53:42 (116 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 22:21:59 (99.7 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 22:30:27 (103 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 22:38:16 (94.1 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 22:42:38 (103 KB/s) - '20MB.zip' saved [20971520/20971520]
    output:2016-12-29 23:08:53 (96.1 KB/s) - '20MB.zip' saved [20971520/20971520]
    output20mExaRandom:2016-12-29 22:24:40 (253 KB/s) - '20M-random.bin' saved [25042/25042]
    output20mExaRandom:2016-12-29 22:45:23 (88.1 KB/s) - '20M-random.bin' saved [25042/25042]

TLDR: So, it seems that the slow single threaded issue is limited to L2TP (read: BNG's) and Think Broadband's servers. Granted though, this is only testing using a limited number of sources.

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

MrSilver
Pro
Posts: 550
Thanks: 82
Fixes: 9
Registered: ‎05-10-2016

Re: Odd TBB test ?

I'd agree the EXA ones are not a super fast source, I left in to see what a lower server would do if anything error wise!

 

Good that you are also able to recreate it, I still cant get it to error on PCN-BNG01/02 , and interesting your slowdown is to the same @ 100kb/sec I was getting, so that doesnt look too dependant on line speed.

 

Others have had slowness from them but not having run repetitive tests its hard to conclude if it is the general peak slowdown people are also having or or this issue.

 

It might highlight if there are any settings or connections on those that are good.

From my thinking about 20% of the time its really bad and gets lots of packetloss, so might be just one bad link or processor in the data stream which is why its being intermittent on gateway and session.

It may be TBB doesn't like packetloss so scales right back its throughput, whereas other services are having the problem just not being so aggressive in their scaling back of traffic.

 

Whats the next steps in getting it fixed? Are there any more tests / data you want us to run to validate theories or disprove things?

 

 

Browni
Aspiring Hero
Posts: 2,673
Thanks: 1,054
Fixes: 60
Registered: ‎02-03-2016

Re: Odd TBB test ?

I can't see where the original issue has been recreated? There was something like a 70x speed difference like in this post https://community.plus.net/t5/Fibre-Broadband/Odd-TBB-test/m-p/1395830#M52447
SpendLessTime
Hero
Posts: 3,000
Thanks: 928
Fixes: 86
Registered: ‎21-09-2009

Re: Odd TBB test ?

@MrSilver

TBB strongly refute that their infrastructure is at fault on the thread about this subject on their forums..

Ex - Plusnet Customer (2009 - 2023) now with BT
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,869
Thanks: 4,950
Fixes: 315
Registered: ‎04-04-2007

Re: Odd TBB test ?


@Browni wrote:
I can't see where the original issue has been recreated? There was something like a 70x speed difference like in this post https://community.plus.net/t5/Fibre-Broadband/Odd-TBB-test/m-p/1395830#M52447


My tests recreate this perfectly. I'm seeing downloads from ThinkBroadband intermittently drop from up to ~30mbps (3.8MB/s) to as low as ~800kbps (100KB/s). If I was on a faster part of my local network (the raspberry pi is limited by it's Powerline connection) then I imagine I'd see the same drop from ~70mbps down to ~800kbps.

@MrSilver wrote:

Whats the next steps in getting it fixed? Are there any more tests / data you want us to run to validate theories or disprove things?


I've forwarded the results of our testing to those looking into the problem. I think it would also help for us to reach out to our contacts at AAISP and Zen if it's reported they've fixed the problem (is it?).

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

rongtw
Seasoned Hero
Posts: 6,973
Thanks: 1,541
Fixes: 12
Registered: ‎01-12-2010

Re: Odd TBB test ?

@bobpullen  So Basically in laymans terms,  You belive the problem lies within the TBB test ?Huh

Is so how would this explain inconsistant page load speeds and Pings that go up and down ?Huh

Asus ROG Hero Vii Z97 , Intel i5 4690k ,ROG Asus Strix 1070,
samsung 850evo 250gig , WD black 2 TB . Asus Phoebus sound ,
16 gig Avexir ram 2400 , water cooling Corsair H100i gtx ,
Corsair 750HXI Psu , Phanteks Enthoo pro case .
Browni
Aspiring Hero
Posts: 2,673
Thanks: 1,054
Fixes: 60
Registered: ‎02-03-2016

Re: Odd TBB test ?

Apologies @bobpullen, my reply was based on reading the email notification on my phone, I can see the full picture now I'm at my PC.

bobpullen
Community Gaffer
Community Gaffer
Posts: 16,869
Thanks: 4,950
Fixes: 315
Registered: ‎04-04-2007

Re: Odd TBB test ?


@rongtw wrote:

@bobpullen  So Basically in laymans terms,  You belive the problem lies within the TBB test ?Huh


No, I haven't said that. What's clear though, is that whatever the cause of the problem, it's not evident to/from all destinations/sources. And at the moment, Think Broadband is the only conclusive source we have. Make of that as you will.

It could be that there are problems to a load of other destinations, we don't know without somebody being able to identify another source or until further investigation has been carried out.

FWIW, I haven't noticed any degradation at home, however I'm not a good use case because most of my browsing is from mobile devices/over wifi, and any large downloading I do tends to be multi-threaded and is either automated or happens in the background whilst I'm doing other stuff. I do have a load of servers running and nobody has complained about issues connecting to those and there's been some fairly solid VPN/streaming use today that I haven't noticed issue with. I'm not a online gamer but my latency from TBB looks OK (the red block is the result of my testing last night whilst connecting over PTA, the blocks of higher maximum latency are people streaming content from one of my servers and the yellow spikes seem to be something to do with my router - TP-Link VR-900 - or some routine activity on my network. When things are quiet it happens every four hours or so, but only with the TP-Link and not other routers I have):

 

My Broadband Ping - Halfway - Staff

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

MrSilver
Pro
Posts: 550
Thanks: 82
Fixes: 9
Registered: ‎05-10-2016

Re: Odd TBB test ?

So if we go on what we know I think this summarises it.

On some of the BNGs (L2TP) on the 20mb tbb speedtest you can get a situation that is repeatable where the speed drops to about 100k/sec for the whole duration right from the start.

On Plusnet PTA & BT it doesn't ever seem to occur - to the same test to the same TBB servers.

Taking the same file from the test and putting it on megaupload and dropbox doesn't ever suffer from the same issue, PTA or L2TP

AA/Zen use L2TP and have or have had the problem in the past to the same test.

 

 

 

Separately to this, there are also gremlins in the network causing

Latency (https://community.plus.net/t5/Fibre-Broadband/High-ping-congestion-to-multiplay-co-uk-game-servers/t...)

Peak Time slowdowns (https://community.plus.net/t5/Fibre-Broadband/Peak-Time-slowdown/td-p/1392498) 

 

Its really easy to script file downloads to test other places but so far all other reports have either been a one off or up and down in performance and don't fit the pattern of this thread, i.e. there is an other issue in the network - they may turn out to get solved by this getting fixed, e.g. multipath traffic and sometimes it uses a bad link whereas the single thread test can only ever use one etc.

 

Theories include:

L2TP problems with BTW (unlikely if only some BNGs are impacted but not all BTW may have identical configuration where they have multiple vendors).

L2TP - less tolerant to packet loss on certain links

Other packet loss - Somewhere on a link between the L2TP servers and TBB servers packets get lost and TCP gives up trying to scale up its window size again.

Traffic Shaping - Traffic is being inspected on a sampled basis somewhere that 1/5 of the time is grinding it to a halt - but why not to drop box in the same reliable way.

TBB is prioritised in traffic shaping but broken so actually on some servers killing the throughput

Congestion - Proven to be at all times of day minutes after a full speed test so highly unlikely

 

Either way, now we have this one accepted we can start to focus on peak time slowdown 🙂 I just need my line fixing first as it goes from 70 down to 10meg during the day - that's against the BTW tap3 test which bypasses plusnet network so there is a wider issue on my line at peak times so I cant do much testing there yet.

Andrue
Pro
Posts: 775
Thanks: 90
Fixes: 1
Registered: ‎12-01-2015

Re: Odd TBB test ?

For what it's worth I've been running two TBB tests most evenings since the week before Christmas when I swapped routers. I'm at work and can't remember what gateway I'm on (I think it was at Colindale) but I haven't seen any bad tests from TBB. I'm not trying to refute anything but suggesting my connection to @bpullen as an example of one that seems fault free.

 

And my TBBQM is a model of serenity - except when I'm running a test, lol.

52dab5682b66f907a0fe0357200d3920

SpendLessTime
Hero
Posts: 3,000
Thanks: 928
Fixes: 86
Registered: ‎21-09-2009

Re: Odd TBB test ?

Although the slowdown is clearly spotted on the TBB x1 test, the questionable BNG's show a picture of high (for me) latency on the TTB ping graph for normal everyday consumption

95916268b7e660a9616782f36468972f-30-12-2016

The peaks shown at for example, 10pm and from 1:30pm to 3pm are just from playing Youtube videos at 1080.

On the good old AG gateways the graph looked like

8bf069a4e72c4b3eaccc4923697b783b-10-11-2016

So a noticeable difference between the 2 sets of gateways

Ex - Plusnet Customer (2009 - 2023) now with BT
SteveA
Pro
Posts: 1,847
Thanks: 106
Fixes: 3
Registered: ‎17-06-2007

Re: Odd TBB test ?

Certainly my Latency just spikes when ever I put any sort of load on the line:

 

This is  pcn-bng04 today

 

b6d8ff116bbb18081af6c9fdeb55039b-30-12-2016

 

Whereas (not sure what gateway I was on on ) this is what it looked like on 1st August:

 

53050aef8df274cad5490eff09bb7241-01-08-2016

I was getting latency spikes but much lower and the average latency hardly increased at all.

 

Sync speeds have not shifted at all

 

SteveA
Pro
Posts: 1,847
Thanks: 106
Fixes: 3
Registered: ‎17-06-2007

Re: Odd TBB test ?

My wife plays quite a few Zinga games and apparently Farmville  ( which is flash based ) sometimes grinds to a halt when its trying to load stuff from its back end. Not sure if Zinga flash games are using single thread or multiple thread calls...

MrSilver
Pro
Posts: 550
Thanks: 82
Fixes: 9
Registered: ‎05-10-2016

Re: Odd TBB test ?

If you run WIndows 10 performance monitor (ctrl alt del-> task manager->performance) you can see total traffic, so if farmville or others are struggling you can quickly check on there is there is a background amount of traffic of it its just not coming through at all. Each time I've seen it there is a constant stream of data just always there slow with LOTs of retransmissions.