cancel
Showing results for 
Search instead for 
Did you mean: 

Level3 Peering is broken again

deathtrap
Grafter
Posts: 1,063
Thanks: 4
Registered: 23-04-2013

Level3 Peering is broken again

Target Name: lb-lsg-prod.ubisoft.com
        IP: 216.98.48.56
 Date/Time: 11/02/2014 02:54:07 to 11/02/2014 03:47:26
Hop Sent Err   PL% Min Max Avg  Host Name / [IP]
1  1000 176  17.6   0   1   0  home.gateway.home.gateway [192.168.1.254]
2  1000   0   0.0  13 346  20  lo0-central10.pcl-ag02.plus.net [195.166.128.183]
3  1000   0   0.0  13 108  13  link-b-central10.pcl-gw02.plus.net [212.159.2.166]
4  1000   0   0.0  12  48  13  xe-10-1-0.pcl-cr02.plus.net [212.159.0.198]
5  1000   0   0.0  12  66  14  ae2.pcl-cr01.plus.net [195.166.129.6]
6  1000   0   0.0  13 122  16  xe-11-2-0.edge3.London2.Level3.net [212.187.201.213]
7   608   2   0.3  25  40  26  ae-3-3.ebr1.Paris1.Level3.net [4.69.141.86]
8   608   0   0.0  25  38  25  ae-91-91.csw4.Paris1.Level3.net [4.69.161.90]
9   608   0   0.0  25  46  26  ae-92-92.ebr2.Paris1.Level3.net [4.69.161.105]
10   608   0   0.0  25  37  25  ae-46-46.ebr1.Frankfurt1.Level3.net [4.69.143.137]
11   608   3   0.5  25  41  25  ae-71-71.csw2.Frankfurt1.Level3.net [4.69.140.6]
12   608   1   0.2  25 116  26  ae-2-70.edge4.Frankfurt1.Level3.net [4.69.154.72]
13   608   0   0.0  25  85  26  ix-18-0.tcore1.FR0-Frankfurt.as6453.net [195.219.50.49]
14   608  25   4.1 119 176 122  if-9-2.tcore1.FNM-Frankfurt.as6453.net [195.219.50.42]
15   608  17   2.8 120 145 121  if-5-2.tcore1.AV2-Amsterdam.as6453.net [195.219.194.13]
16   608   0   0.0 119 131 119  if-13-2.tcore2.AV2-Amsterdam.as6453.net [80.231.152.22]
17   619   0   0.0 113 137 120  if-8-2.tcore2.L78-London.as6453.net [80.231.131.5]
18  1000  33   3.3 106 176 116  if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
19  1000  26   2.6 106 127 114  if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
20  1000  73   7.3 106 130 114  if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
21  1000  22   2.2 106 187 117  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
22   999 999 100.0   0   0   0   [-]

Destination not reached in 35 hops
Just look at the peering by Level 3  , London to Paris then to Frankfurt. then AS6453.net  Route to Amsterdam back to london before having a go at peering to the USA, and Montreal via a detour Chicago ,  increase in latency by another 13ms  on top off  an already ridiculously high ping of 106ms (should be and was 89-95ms ) for several years , and still is lower via other isp's , oh and what's happening re the ongoing latency issue to ubisoft.com,  have plusnet forgotten about it? i haven't and haven't heard anything since i was informed that level 3 had been contacted re the dodgy peering
Update:
Target Name: lb-lsg-prod.ubisoft.com
        IP: 216.98.48.56
  Date/Time: 11/02/2014 03:27:02 to 11/02/2014 04:19:33
Hop Sent Err  PL% Min Max Avg  Host Name / [IP]
1  1000 178  17.8  0  1  0  home.gateway.home.gateway [192.168.1.254]
2  1000  0  0.0  13 348  19  lo0-central10.pcl-ag02.plus.net [195.166.128.183]
3  1000  0  0.0  12  90  13  link-b-central10.pcl-gw02.plus.net [212.159.2.166]
4  1000  0  0.0  12  59  14  xe-10-1-0.pcl-cr02.plus.net [212.159.0.198]
5  1000  0  0.0  12  89  14  ae2.pcl-cr01.plus.net [195.166.129.6]
6  1000  1  0.1  13 126  16  xe-11-2-0.edge3.London2.Level3.net [212.187.201.213]
7  387  0  0.0  13  77  17  ae-0-11.edge4.London2.Level3.net [4.69.200.126]
8  387  1  0.3  13  25  13  ae-3-3.ebr1.London15.Level3.net [4.69.141.189]
9  387  1  0.3  13  35  13  ae-44-44.ebr1.London1.Level3.net [4.69.167.30]
10  387  1  0.3  13  25  13  ae-58-113.csw1.London1.Level3.net [4.69.153.122]
11  387  11  2.8  13  69  14  ae-116-3502.edge3.London1.Level3.net [4.69.166.134]
12  382  14  3.7  13  73  16  ix-20-0.tcore1.LDN-London.as6453.net [195.219.83.101]
13  382  20  5.2 106 114 106  if-17-2.tcore1.L78-London.as6453.net [80.231.130.129]
14  382  23  6.0 105 117 105  if-2-2.tcore2.L78-London.as6453.net [80.231.131.1]
15  1000  14  1.4 106 198 116  if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
16  1000  14  1.4 106 129 114  if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
17  1000  56  5.6 106 130 114  if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
18  1000  14  1.4 106 194 118  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]

Destination not reached in 35 hops
It changed to this minutes ago, why all the london hops from level 3? ubisoft should be 12-14 hops not 18+
27 REPLIES
Mathias
Grafter
Posts: 242
Registered: 21-08-2008

Re: Level3 Peering is broken again

Just so i can learn something new as this stuff is not familiar to me, is level 3 something that Plusnet choose to use or is it the game providers who choose this route?
Community Gaffer
Community Gaffer
Posts: 4,890
Thanks: 164
Fixes: 3
Registered: 04-04-2007

Re: Level3 Peering is broken again

Is anyone else seeing problems with Ubisoft? 
Deathtrap you appear to have packet loss at your first hop?  Do you have an explanation for that?    WHat ever is going on is causing latency between you and our gateway. (as you can see by the max time on lo0-central10.pcl-ag02.plus.net)
What are you actually doing which isn't working properly?  Don't fixate on the number of hops, there are more devices passing your packets along between your router and hop 1 on your trace route than in the entire rest of the trace.  It's the end to end performance we need to look at.
Plusnet Staff
Plusnet Staff
Posts: 6,346
Thanks: 31
Fixes: 5
Registered: 26-11-2011

Re: Level3 Peering is broken again

Also, what router are you using?
Chris Pettitt
Cloud Environments Engineer
Community Veteran
Posts: 6,818
Registered: 27-10-2012

Re: Level3 Peering is broken again

I think he uses a Billion looking at the home gateway address/IP.
The packet loss on the first hop is not new for deathtrap: http://community.plus.net/forum/index.php/topic,118180.msg1023453.html#msg1023453
Probably totally unrelated, but there was an issue recently with BT Retail/Ubisoft. Neil McRae at BT was supposed to explain what the problem was and how it was fixed, but he never did explain it.
Community Gaffer
Community Gaffer
Posts: 4,890
Thanks: 164
Fixes: 3
Registered: 04-04-2007

Re: Level3 Peering is broken again

Any other billion users showing a similar issue? 
Deathtrap's line is showing packet loss on first hop, variable latency in traceroutes, and slowdowns at peak.  I don't think these are seen by the majority of customers, so it just feels like there is something else wrong here.
Community Gaffer
Community Gaffer
Posts: 4,890
Thanks: 164
Fixes: 3
Registered: 04-04-2007

Re: Level3 Peering is broken again

Quote from: AndyH
Probably totally unrelated, but there was an issue recently with BT Retail/Ubisoft. Neil McRae at BT was supposed to explain what the problem was and how it was fixed, but he never did explain it.

I'm going to see if I can find out what happened with that, but I don't think it's related.  The problem was causing people to be completely unable to log into uplay, which is quite a distinct, disruptive issue!
Moderator
Moderator
Posts: 25,691
Thanks: 1,081
Fixes: 46
Registered: 14-04-2007

Re: Level3 Peering is broken again

Billion 7800n

Tracing route to ubisoft.com [216.98.48.40]
over a maximum of 30 hops:
  1    <1 ms    <1 ms    <1 ms  home.gateway.home.gateway [192.168.1.254]
  2    15 ms    14 ms    13 ms  lo0-central10.ptw-ag01.plus.net [195.166.128.195]
  3    14 ms    14 ms    13 ms  link-a-central10.ptw-gw01.plus.net [212.159.2.144]
  4    14 ms    14 ms    14 ms  xe-4-2-0.ptw-cr01.plus.net [212.159.0.240]
  5    14 ms    14 ms    14 ms  te-3-4.car5.London1.Level3.net [217.163.45.181]
  6    15 ms    15 ms    14 ms  ae-52-52.csw2.London1.Level3.net [4.69.139.120]
  7    14 ms    14 ms    14 ms  ae-225-3601.edge3.London1.Level3.net [4.69.166.146]
  8    19 ms    44 ms    30 ms  ix-20-0.tcore1.LDN-London.as6453.net [195.219.83.101]
  9  117 ms  117 ms  118 ms  if-17-2.tcore1.L78-London.as6453.net [80.231.130.129]
10  118 ms  117 ms  118 ms  if-2-2.tcore2.L78-London.as6453.net [80.231.131.1]
11  125 ms  129 ms  117 ms  if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
12  117 ms  116 ms  117 ms  if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
13  118 ms  117 ms  118 ms  if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
14  116 ms  130 ms  117 ms  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
15    *        *        *    Request timed out.
16    *        *        *    Request timed out.
17    *        *        *    Request timed out.
18    *        *        *    Request timed out.
19    *        *        *    Request timed out.
20    *        *        *    Request timed out.
21    *        *        *    Request timed out.
22    *        *        *    Request timed out.
23    *        *        *    Request timed out.
24    *        *        *    Request timed out.
25    *        *        *    Request timed out.
26    *        *        *    Request timed out.
27    *        *        *    Request timed out.
28    *        *        *    Request timed out.
29    *        *        *    Request timed out.
30    *        *        *    Request timed out.
Trace complete.
Customer and Forum Moderator.
Product of the Tyrell Corporation
I see dead Parsnips.
deathtrap
Grafter
Posts: 1,063
Thanks: 4
Registered: 23-04-2013

Re: Level3 Peering is broken again

What appears to be packet loss isn't it's because the router is busy, and like some switches on the internet  doesn't respond to every icmp request, as to how is this affecting me,  well mm, lagggg,  for me if i join someone's game or they experience it  when i'm hosting the game,  due to the degree of fluctuation  of the base latency at times , all those hops can only make that worse , regardless of if they are adding latency or not
Target Name: lb-lsg-prod.ubisoft.com
        IP: 216.98.48.56
 Date/Time: 11/02/2014 16:18:37 to 11/02/2014 16:30:31
Hop Sent Err   PL% Min Max Avg  Host Name / [IP]
1       540   0   0.0      0     0     0  home.gateway.home.gateway [192.168.1.254]
2       540   0   0.0    13 359  20  lo0-central10.pcl-ag02.plus.net [195.166.128.183]
3       540   0   0.0    13 187  13  link-b-central10.pcl-gw02.plus.net [212.159.2.166]
4       540   0   0.0    12  85  14  xe-10-1-0.pcl-cr02.plus.net [212.159.0.198]
5       540   0   0.0    12  65  15  ae2.pcl-cr01.plus.net [195.166.129.6]
6       540   0   0.0    13  76  14  xe-11-2-0.edge3.London2.Level3.net [212.187.201.213]
7       540   0   0.0    13  72  15  ae-0-11.edge4.London2.Level3.net [4.69.200.126]
8       540   1   0.2    13  37  13  ae-3-3.ebr1.London15.Level3.net [4.69.141.189]
9       540   0   0.0    13  49  13  ae-43-43.ebr1.London1.Level3.net [4.69.167.26]
10      540   0   0.0    13  25  13  ae-59-114.csw1.London1.Level3.net [4.69.153.126]
11      540   0   0.0    13  79  15  ae-117-3503.edge3.London1.Level3.net [4.69.166.138]
12      540   0   0.0    13  57  14  ix-20-0.tcore1.LDN-London.as6453.net [195.219.83.101]
13      540   5   0.9    106 125 106  if-17-2.tcore1.L78-London.as6453.net [80.231.130.129]
14      540  13  2.4    105 118 105  if-2-2.tcore2.L78-London.as6453.net [80.231.131.1]
15      540   2   0.4   106 146 108  if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
16      540   0   0.0   106 118 106  if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
17      540  21  3.9   106 115 106  if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
18      540   0   0.0   106 174 109  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]

Destination not reached in 35 hops
Router wasn't busy this time
Heres a tcp ping
Target Name: lb-lsg-prod.ubisoft.com
        IP: 216.98.48.56
 Date/Time: 11/02/2014 16:18:37 to 11/02/2014 16:38:05
Hop Sent Err  PL% Min Max Avg  Host Name / [IP]
1   892   1  0.1   0   4   0  home.gateway.home.gateway [192.168.1.254]
2   892   0  0.0  13 359  22  lo0-central10.pcl-ag02.plus.net [195.166.128.183]
3   892   0  0.0  13 187  13  link-b-central10.pcl-gw02.plus.net [212.159.2.166]
4   892   0  0.0  12  85  14  xe-10-1-0.pcl-cr02.plus.net [212.159.0.198]
5   879   0  0.0  12  65  14  ae2.pcl-cr01.plus.net [195.166.129.6]
6    17   0  0.0  13  14  13  xe-11-1-0.edge3.London2.Level3.net [212.187.201.209]
7   872   4  0.5  13  72  15  ae-0-11.edge4.London2.Level3.net [4.69.200.126]
8   879   2  0.2  13  37  13  ae-3-3.ebr1.London15.Level3.net [4.69.141.189]
9     4   0  0.0  13  13  13  ae-226-3602.edge3.London1.Level3.net [4.69.166.150]
10    16   0  0.0  13  22  14  ae-56-111.csw1.London1.Level3.net [4.69.153.114]
11    15   0  0.0  14  14  14  ae-116-3502.edge3.London1.Level3.net [4.69.166.134]
12   884   0  0.0  13  57  14  ix-20-0.tcore1.LDN-London.as6453.net [195.219.83.101]
13   880   2  0.2 106 149 108  if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
14   884   1  0.1 106 160 106  if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
15     8   0  0.0 107 129 112  if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
16   884   0  0.0 106 174 109  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
17    16   0  0.0 107 112 108  if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
18     8   0  0.0 107 122 111  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
19    28   0  0.0 107 144 111  [66.198.96.18]
20   887 855 96.4 107 121 111  lb-lsg-prod.ubisoft.com [216.98.48.56]
one of many routes as it switches back and forth a lot, causing the fluctuation in latency
Target Name: lb-lsg-prod.ubisoft.com
        IP: 216.98.48.56
 Date/Time: 11/02/2014 16:19:52 to 11/02/2014 16:40:50
Hop Sent Err  PL% Min Max Avg  Host Name / [IP]
1  1000   1  0.1   0   4   0  home.gateway.home.gateway [192.168.1.254]
2  1000   0  0.0  13 359  22  lo0-central10.pcl-ag02.plus.net [195.166.128.183]
3  1000   0  0.0  13 187  13  link-b-central10.pcl-gw02.plus.net [212.159.2.166]
4  1000   0  0.0  12  85  14  xe-10-1-0.pcl-cr02.plus.net [212.159.0.198]
5   963   0  0.0  12  65  14  ae2.pcl-cr01.plus.net [195.166.129.6]
6   846   0  0.0  13  76  14  xe-11-2-0.edge3.London2.Level3.net [212.187.201.213]
7   948  14  1.5  13  72  15  ae-0-11.edge4.London2.Level3.net [4.69.200.126]
8   962   2  0.2  13  37  13  ae-3-3.ebr1.London15.Level3.net [4.69.141.189]
9    58   0  0.0  13  14  13  ae-42-42.ebr1.London1.Level3.net [4.69.167.22]
10   829   0  0.0  13  25  13  ae-59-114.csw1.London1.Level3.net [4.69.153.126]
11    41   0  0.0  13  56  16  ae-115-3501.edge3.London1.Level3.net [4.69.166.130]
12   928   1  0.1  13  57  15  ix-20-0.tcore1.LDN-London.as6453.net [195.219.83.101]
13   939  13  1.4 106 137 106  if-17-2.tcore1.L78-London.as6453.net [80.231.130.129]
14    85   1  1.2 105 116 105  if-2-2.tcore2.L78-London.as6453.net [80.231.131.1]
15   928   8  0.9 106 149 108  if-20-2.tcore2.NYY-NewYork.as6453.net [216.6.99.13]
16   940   1  0.1 106 174 109  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]
17   170   1  0.6 107 216 111  [66.198.96.18]
18   162   2  1.2 106 121 107  msr-onl-fw01.ubisoft.com [216.98.51.10]
19   992 813 82.0 106 174 112  lb-lsg-prod.ubisoft.com [216.98.48.56]
doesn't route via Chicago, so correctly routed  i believe this could be the issue  it shouldn't use the Chicago route at all
But lets not loose sight on what the original problem was the increase in latency from 89-95ms to what i have today,  it shouldn't be this high  have plusnet complained to level 3  or asked them to contact AS6453.net  and ask them why traffic is being routed away from Montreal  from NewYork or asked if they can provide a shorter route with lower latency ? as far as i can nothing much has or is being done
As for what is causing the spikes in latency  between  the router and your gateway , better ask BT wholesale about that or check the gateway, as it's nothing this side causing the spiking , but it's nothing new
Community Veteran
Posts: 6,818
Registered: 27-10-2012

Re: Level3 Peering is broken again

When you say your router wasn't busy, what do you mean (i.e. what were you doing)?
Some of those hops will be set to low priority ICMP - I can ping bbc.co.uk and get 4-5 ms over 1000 pings, but if I ping my PN gateway I get anything ranging from 5-300ms.

Community Gaffer
Community Gaffer
Posts: 4,890
Thanks: 164
Fixes: 3
Registered: 04-04-2007

Re: Level3 Peering is broken again

Yeah, we've had a conversation with Level3 and they can't see anything wrong.  I need to digest the rest of your post a bit.
deathtrap
Grafter
Posts: 1,063
Thanks: 4
Registered: 23-04-2013

Re: Level3 Peering is broken again

AndyH I was pinging not with icmp but TCP which is supposed show a more accurate picture, and the router/switches should treat it as normal UDP or TCP traffic , hence being able to see the routing being swapped around, , the thing it doesn't show is the return path ,
Community Gaffer
Community Gaffer
Posts: 4,890
Thanks: 164
Fixes: 3
Registered: 04-04-2007

Re: Level3 Peering is broken again

Quote from: deathtrap
What appears to be packet loss isn't it's because the router is busy, and like some switches on the internet  doesn't respond to every icmp request, as to how is this affecting me,  well mm, lagggg,  for me if i join someone's game or they experience it  when i'm hosting the game,  due to the degree of fluctuation  of the base latency at times , all those hops can only make that worse , regardless of if they are adding latency or not

Which games?  Are other PN players seeing the same problems, or is it just your line.
What are you doing to keep your router that busy?  Does the lag disappear if you are only gaming?
deathtrap
Grafter
Posts: 1,063
Thanks: 4
Registered: 23-04-2013

Re: Level3 Peering is broken again

I have one game that connects via ubisoft the problem i have is with the way traffic from plusnet is getting routed and the associated increase in latency since it started being routed this way, as well as the amount of fluctuation across those links, even some of level 3's uk kit shows a lot of fluctuation , as well as traffic getting passed across some very busy switches, this is gaming traffic for a game that is sensitive to changes in latency level3 ect should take this into account, also those Level 3 and AS6453.net links to ubisoft show increases in base latency sometimes the "hump"
so are probably being run close to or at capacity ,not good for on-line gaming or even voice coms The lag is only a problem when gaming, what was i doing to keep my router that busy, nothing much apart from pinging some other destinations using pingplotter pro
Community Gaffer
Community Gaffer
Posts: 4,890
Thanks: 164
Fixes: 3
Registered: 04-04-2007

Re: Level3 Peering is broken again

A traceroute on it's own doesn't prove a problem though.  If you have a game which is showing jitter, that's a different matter.  I'm trying to work out how to prove an issue, and we just don't have enough yet.