cancel
Showing results for 
Search instead for 
Did you mean: 

LEVEL3 peering to USA heavy Packet loss tonight

deathtrap
Grafter
Posts: 1,064
Thanks: 4
Registered: ‎23-04-2013

LEVEL3 peering to USA heavy Packet loss tonight

As above
Target Name: steampowered.com
        IP: 208.64.202.68
 Date/Time: 21/04/2014 18:49:36 to 21/04/2014 19:20:10
Hop Sent Err  PL% Min Max Avg  Host Name / [IP]
1  1000   0  2 0   0   2   0  home.gateway.home.gateway [192.168.1.254]
2  1000   0  0.0  13 412  27  lo0-central10.ptw-ag04.plus.net [195.166.128.194]
3  1000   0  0.0  13 104  13  link-a-central10.ptw-gw01.plus.net [212.159.2.156]
4  1000   0  0.0  12  55  14  xe-7-2-0.ptw-cr01.plus.net [212.159.0.252]
5  1000  85  8.5  13 437  60  te-4-2.car5.London1.Level3.net [217.163.45.249]
6   992 109 11.0  13 172  81  ae-52-52.csw2.London1.Level3.net [4.69.139.120]
7   964 272 28.2 153 245 156  ae-58-223.ebr2.London1.Level3.net [4.69.153.137]
8   502  78 15.5 151 157 154  ae-42-42.ebr1.NewYork1.Level3.net [4.69.137.70]
9   946 218 23.0 151 219 155  ae-46-46.ebr1.NewYork2.Level3.net [4.69.201.42]
10   991 198 20.0 151 211 154  ae-1-100.ebr2.NewYork2.Level3.net [4.69.135.254]
11   998 238 23.8 151 169 156  ae-2-2.ebr1.Chicago1.Level3.net [4.69.132.65]
12  1000 251 25.1 152 167 156  ae-6-6.ebr1.Chicago2.Level3.net [4.69.140.190]
13   999 164 16.4 151 179 154  ae-3-3.ebr2.Denver1.Level3.net [4.69.132.61]
14   998 221 22.1 153 171 156  ae-2-2.ebr2.Seattle1.Level3.net [4.69.132.53]
15   996 279 28.0 152 255 159  ae-2-52.edge1.Seattle3.Level3.net [4.69.147.170]
16   994 273 27.5 153 172 156  CENTURYTEL.edge1.Seattle3.Level3.net [4.59.232.74]
17   997 225 22.6 153 235 156  [205.196.6.37]
18   837 212 25.3 153 225 157  208-64-202-24.valve.net [208.64.202.24]
19   990 223 22.5 153 169 157  208-64-202-68.valve.net [208.64.202.68]

Target Name: lb-lsg-prod.ubisoft.com
        IP: 216.98.48.56
  Date/Time: 21/04/2014 18:42:25 to 21/04/2014 19:21:46
Hop Sent Err  PL% Min Max Avg  Host Name / [IP]
1  1000  0  0.0  0  1  0  home.gateway.home.gateway [192.168.1.254]
2  1000  0  0.0  13 370  27  lo0-central10.ptw-ag04.plus.net [195.166.128.194]
3  1000  0  0.0  13  32  13  link-a-central10.ptw-gw01.plus.net [212.159.2.156]
4  1000  0  0.0  12  51  14  xe-7-2-0.ptw-cr01.plus.net [212.159.0.252]
5  1000 105  10.5  13 785  62  te-3-4.car5.London1.Level3.net [217.163.45.181]
6  443 128  28.9  13  27  17  ae-51-51.csw1.London1.Level3.net [4.69.139.88]
7  358  3  0.8  13  82  14  ae-118-3504.edge3.London1.Level3.net [4.69.166.142]
8  1000  39  3.9  13  43  13  ix-20-0.tcore1.LDN-London.as6453.net [195.219.83.101]
9  1000 283  28.3 105 124 108  if-17-2.tcore1.L78-London.as6453.net [80.231.130.129]
10  1000 260  26.0 108 127 111  if-2-2.tcore2.L78-London.as6453.net [80.231.131.1]
11  1000 306  30.6 106 190 112  if-20-2.tcore2.NYY-New-York.as6453.net [216.6.99.13]
12  1000 298  29.8 106 121 109  if-12-6.tcore1.CT8-Chicago.as6453.net [216.6.99.46]
13  998 242  24.2 106 121 109  if-22-2.tcore2.CT8-Chicago.as6453.net [64.86.79.1]
14  1000 233  23.3 106 121 109  if-3-2.tcore1.W6C-Montreal.as6453.net [66.198.96.45]

Destination not reached in 35 hops
26 REPLIES 26
npr
Pro
Posts: 1,898
Thanks: 119
Fixes: 9
Registered: ‎21-01-2013

Re: LEVEL3 peering to USA heavy Packet loss tonight

No packet loss here, but ping times to streampowered.com are not looking good.

Quote
Ping statistics for 208.64.202.68:
    Packets: Sent = 195, Received = 195, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 176ms, Maximum = 563ms, Average = 336ms


Tracert does suggests a issue with level3 in London.
Quote
Tracing route to steampowered.com [208.64.202.68]
over a maximum of 30 hops:
  1  100 ms    54 ms    97 ms  192.168.1.254
  2    28 ms    31 ms    27 ms  lo0-central10.ptw-ag02.plus.net [195.166.128.196]
  3    28 ms    26 ms    26 ms  link-b-central10.ptw-gw02.plus.net [212.159.2.150]
  4    28 ms    31 ms    26 ms  xe-0-2-0.ptw-cr02.plus.net [212.159.0.250]
  5    27 ms    26 ms    26 ms  ae2.ptw-cr01.plus.net [195.166.129.4]
  6    27 ms    26 ms    26 ms  ae1.pcl-cr01.plus.net [195.166.129.1]
  7    28 ms    26 ms    26 ms  xe-11-2-0.edge3.London2.Level3.net [212.187.201.213]
  8  299 ms  304 ms  305 ms  vl-3201-ve-128.ebr2.London2.Level3.net [4.69.202.177]
  9  485 ms  304 ms  305 ms  ae-43-43.ebr1.Paris1.Level3.net [4.69.159.90]
10  406 ms  304 ms  305 ms  ae-61-61.csw1.Paris1.Level3.net [4.69.161.78]
11  407 ms  303 ms  304 ms  ae-62-62.ebr2.Paris1.Level3.net [4.69.161.93]
12  405 ms  305 ms  304 ms  ae-43-43.ebr2.Washington1.Level3.net [4.69.137.58]
13  411 ms  304 ms  304 ms  ae-47-47.ebr2.Washington12.Level3.net [4.69.202.58]
14  403 ms  304 ms  304 ms  ae-6-6.ebr2.Chicago2.Level3.net [4.69.148.146]
15  303 ms  442 ms  327 ms  ae-1-100.ebr1.Chicago2.Level3.net [4.69.132.113]
16  495 ms  328 ms  214 ms  ae-3-3.ebr2.Denver1.Level3.net [4.69.132.61]
17  363 ms  347 ms  281 ms  ae-2-2.ebr2.Seattle1.Level3.net [4.69.132.53]
18  185 ms  221 ms  176 ms  ae-2-52.edge1.Seattle3.Level3.net [4.69.147.170]
19  258 ms  305 ms  305 ms  CENTURYTEL.edge1.Seattle3.Level3.net [4.59.232.74]
20  410 ms  305 ms  305 ms  205.196.6.37
21  285 ms  304 ms  203 ms  208-64-202-24.valve.net [208.64.202.24]
22  424 ms  305 ms  304 ms  208-64-202-68.valve.net [208.64.202.68]
Terranova667
Pro
Posts: 1,511
Thanks: 125
Fixes: 5
Registered: ‎19-02-2014

Re: LEVEL3 peering to USA heavy Packet loss tonight

level3 routing at it again, I use to be with BE broadband and they used them  for their routing and always had issues with them making things a damn pain especially when it would take hours if not days for BE to do anything about it ,  no packet loss but I'm getting pretty much 155ms pings via them to steam which does not help.  
Pinging steampowered.com [208.64.202.68] with 32 bytes of data:
Reply from 208.64.202.68: bytes=32 time=155ms TTL=46
Reply from 208.64.202.68: bytes=32 time=155ms TTL=46
Reply from 208.64.202.68: bytes=32 time=155ms TTL=46
Reply from 208.64.202.68: bytes=32 time=155ms TTL=46
Ping statistics for 208.64.202.68:
   Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
   Minimum = 155ms, Maximum = 155ms, Average = 155ms
trace route
Tracing route to steampowered.com [208.64.202.68]
over a maximum of 30 hops:
 1    <1 ms     1 ms     1 ms  dsldevice.lan [192.168.1.254]
 2    28 ms    18 ms    14 ms  lo0-central10.ptw-ag04.plus.net [195.166.128.194
]
 3    14 ms    13 ms    13 ms  link-a-central10.ptw-gw01.plus.net [212.159.2.15
6]
 4    14 ms    14 ms    14 ms  xe-7-2-0.ptw-cr01.plus.net [212.159.0.252]
 5     *        *       14 ms  te-3-4.car5.London1.Level3.net [217.163.45.181]
 6   154 ms   154 ms   155 ms  ae-52-52.csw2.London1.Level3.net [4.69.139.120]
 7   153 ms   153 ms   153 ms  ae-57-222.ebr2.London1.Level3.net [4.69.153.133]
 8   154 ms   154 ms   154 ms  ae-44-44.ebr1.NewYork1.Level3.net [4.69.137.78]
 9   153 ms   152 ms   153 ms  ae-48-48.ebr1.NewYork2.Level3.net [4.69.201.50]
10   154 ms   153 ms   153 ms  ae-1-100.ebr2.NewYork2.Level3.net [4.69.135.254]
11   154 ms   154 ms   154 ms  ae-2-2.ebr1.Chicago1.Level3.net [4.69.132.65]
12   153 ms   153 ms   153 ms  ae-6-6.ebr1.Chicago2.Level3.net [4.69.140.190]
13   153 ms   153 ms   156 ms  ae-3-3.ebr2.Denver1.Level3.net [4.69.132.61]
14   154 ms   154 ms   153 ms  ae-2-2.ebr2.Seattle1.Level3.net [4.69.132.53]
15   217 ms   153 ms   154 ms  ae-2-52.edge1.Seattle3.Level3.net [4.69.147.170]
16   154 ms   154 ms   154 ms  CENTURYTEL.edge1.Seattle3.Level3.net [4.59.232.7
4]
17   154 ms   154 ms   155 ms  205.196.6.36
18   154 ms   154 ms   158 ms  208-64-202-24.valve.net [208.64.202.24]
19   156 ms   155 ms   155 ms  208-64-202-68.valve.net [208.64.202.68]
Trace complete.
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: LEVEL3 peering to USA heavy Packet loss tonight

@ deathtrap - I think your issues might be related to this http://community.plus.net/forum/index.php/topic,126332.0.html
Gus
Aspiring Pro
Posts: 3,236
Thanks: 26
Fixes: 3
Registered: ‎31-07-2007

Re: LEVEL3 peering to USA heavy Packet loss tonight

@andy nope unrelated, level3 is the bain for gamers playing on US servers.  But PN pay them so they are accountable.
Now if only this forum would work this time
I was also getting loss on all level3 nodes at the time of the post and currently seeing issues gaming on USA servers, logins timing out, switching characters drops the connection to the login server.
I am being routed via france to 208.95.186.167 and still using the existing route to steampowered.com which is jumping between 4 US cities
FTTP 500 regrade from Tues 28th November
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: LEVEL3 peering to USA heavy Packet loss tonight

Quote from: Gus
still using the existing route to steampowered.com which is jumping between 4 US cities

Probably because you're posting a traceroute to Steam's server in Bellevue, WA.
deathtrap
Grafter
Posts: 1,064
Thanks: 4
Registered: ‎23-04-2013

Re: LEVEL3 peering to USA heavy Packet loss tonight

The packet loss was only on usa servers that used Level3's peering transit,  that Steampowered IP address is only one that the steam client uses each time launch it, here's another IPit uses with an even higher  ping,
Target Name: 208-64-200-137.steampowered.com
        IP: 208.64.200.137
 Date/Time: 22/04/2014 02:30:54 to 22/04/2014 02:33:17
Hop Sent Err PL% Min Max Avg  Host Name / [IP]
1   144   0 0.0   0   0   0  home.gateway.home.gateway [192.168.1.254]
2   144   0 0.0  13 648  37  lo0-central10.ptw-ag04.plus.net [195.166.128.194]
3   144   0 0.0  12  25  13  link-b-central10.ptw-gw02.plus.net [212.159.2.158]
4   144   0 0.0  12  46  13  xe-7-2-0.ptw-cr02.plus.net [212.159.0.254]
5   144   0 0.0  13  40  14  ae2.ptw-cr01.plus.net [195.166.129.4]
6   144   0 0.0  12  36  13  ae1.pcl-cr01.plus.net [195.166.129.1]
7   144   0 0.0  13  69  14  xe-11-1-0.edge3.London2.Level3.net [212.187.201.209]
8   144   0 0.0 158 165 158  vl-3201-ve-128.ebr2.London2.Level3.net [4.69.202.177]
9   144   0 0.0 157 159 157  ae-44-44.ebr1.Paris1.Level3.net [4.69.159.94]
10   144   0 0.0 158 159 158  ae-61-61.csw1.Paris1.Level3.net [4.69.161.78]
11   144   0 0.0 158 160 158  ae-62-62.ebr2.Paris1.Level3.net [4.69.161.93]
12   144   0 0.0 159 164 159  ae-44-44.ebr2.Washington1.Level3.net [4.69.137.62]
13   139   0 0.0 159 166 159  ae-46-46.ebr2.Washington12.Level3.net [4.69.202.54]
14   144   0 0.0 159 166 159  ae-6-6.ebr2.Chicago2.Level3.net [4.69.148.146]
15   144   0 0.0 159 168 159  ae-1-100.ebr1.Chicago2.Level3.net [4.69.132.113]
16   144   0 0.0 157 159 157  ae-3-3.ebr2.Denver1.Level3.net [4.69.132.61]
17   144   0 0.0 160 168 160  ae-2-2.ebr2.Seattle1.Level3.net [4.69.132.53]
18   144   0 0.0 160 221 162  ae-2-52.edge1.Seattle3.Level3.net [4.69.147.170]
19   144   0 0.0 159 189 160  CENTURYTEL.edge1.Seattle3.Level3.net [4.59.232.74]
20   144   0 0.0 160 164 160  [205.196.6.36]
21   144   0 0.0 160 161 160  208-64-200-137.steampowered.com [208.64.200.137]
Oh look it goes via paris and is 7ms higher than my other tracert that is routed in a more direct fashion , exactly the same effect it had on latency to Ubisoft.com Level 3 cack handed routing peering , must be the cheapest peering available  is all i can say , oh, and they won't fix it neither will Plusnet  just have a read of my thread in the gaming forum 5 mths  before they turned around and say they can't /wont do anything for me
And another Steampowered.com IP with no Paris in the mix
Target Name: 208-64-200-204.steampowered.com
        IP: 208.64.200.204
  Date/Time: 22/04/2014 02:44:27 to 22/04/2014 02:45:12
Hop Sent Err  PL% Min Max Avg  Host Name / [IP]
1    81  33 40.7  0  0  0  home.gateway.home.gateway [192.168.1.254]
2    84  0  0.0  13 107  15  lo0-central10.ptw-ag04.plus.net [195.166.128.194]
3    84  0  0.0  13  13  13  link-a-central10.ptw-gw01.plus.net [212.159.2.156]
4    84  0  0.0  12  38  13  xe-7-2-0.ptw-cr01.plus.net [212.159.0.252]
5    84  0  0.0  12 350  59  te-3-4.car5.London1.Level3.net [217.163.45.181]
6    84  0  0.0 151 161 152  ae-52-52.csw2.London1.Level3.net [4.69.139.120]
7    84  0  0.0 152 154 152  ae-56-221.ebr2.London1.Level3.net [4.69.153.129]
8    84  0  0.0 152 153 152  ae-43-43.ebr1.NewYork1.Level3.net [4.69.137.74]
9    83  0  0.0 152 153 152  ae-4-4.ebr1.NewYork2.Level3.net [4.69.141.18]
10    84  1  1.2 151 159 151  ae-1-100.ebr2.NewYork2.Level3.net [4.69.135.254]
11    84  1  1.2 153 162 153  ae-2-2.ebr1.Chicago1.Level3.net [4.69.132.65]
12    84  1  1.2 152 153 152  ae-6-6.ebr1.Chicago2.Level3.net [4.69.140.190]
13    84  0  0.0 150 151 150  ae-3-3.ebr2.Denver1.Level3.net [4.69.132.61]
14    84  0  0.0 153 155 153  ae-2-2.ebr2.Seattle1.Level3.net [4.69.132.53]
15    84  1  1.2 153 211 156  ae-2-52.edge1.Seattle3.Level3.net [4.69.147.170]
16    84  1  1.2 152 153 152  CENTURYTEL.edge1.Seattle3.Level3.net [4.59.232.74]
17    84  2  2.4 152 180 153  [205.196.6.37]
18    84  0  0.0 151 152 151  208-64-200-204.steampowered.com [208.64.200.204]

Fortunately the packet loss did stop a short time after i reported it here, but it shouldn't be happening in the first place, if level3 didn't overload their links  perhaps it wouldn't happen , but  some will say and believe that this wont be the case
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: LEVEL3 peering to USA heavy Packet loss tonight

A 150-160ms ping to Bellvue, WA is perfectly acceptable.
What I don't get is why you would use a West Coast server - can you not chose something on the East Coast (which would halve your latency)?
deathtrap
Grafter
Posts: 1,064
Thanks: 4
Registered: ‎23-04-2013

Re: LEVEL3 peering to USA heavy Packet loss tonight

As already  said These steampowered.com IP addresses are what the steam client connects to each time it is launched The end user doesn't get a choice in that , as it's probably the Authentication server where they store users account details , So maybe you should be asking Valve that very question , not that you would get much sense out of them, if you ever got an answer , But i do know if the steam client  can't connect  you can't play many games online , because the steam client has to be online also ,
As for those latency times, i have seen lower latency times to that first steampowered.com IP so lower latency is possible , But that wasn't may point although  i did point out that by routing traffic via Paris  does add 7-8ms  to the overall figure, and that this as far as I'm concerned is a dumb idea by Level 3,But they appeared to of had clear issues yesterday  to probably almost every  destination in the usa   nothing more and nothing less
npr
Pro
Posts: 1,898
Thanks: 119
Fixes: 9
Registered: ‎21-01-2013

Re: LEVEL3 peering to USA heavy Packet loss tonight

Quote from: deathtrap
Target Name: 208-64-200-137.steampowered.com
        IP: 208.64.200.137
 Date/Time: 22/04/2014 02:30:54 to 22/04/2014 02:33:17
Hop Sent Err PL% Min Max Avg  Host Name / [IP]
1   144   0 0.0   0   0   0  home.gateway.home.gateway [192.168.1.254]
2   144   0 0.0  13 648  37  lo0-central10.ptw-ag04.plus.net [195.166.128.194]
3   144   0 0.0  12  25  13  link-b-central10.ptw-gw02.plus.net [212.159.2.158]
4   144   0 0.0  12  46  13  xe-7-2-0.ptw-cr02.plus.net [212.159.0.254]
5   144   0 0.0  13  40  14  ae2.ptw-cr01.plus.net [195.166.129.4]
6   144   0 0.0  12  36  13  ae1.pcl-cr01.plus.net [195.166.129.1]
7   144   0 0.0  13  69  14  xe-11-1-0.edge3.London2.Level3.net [212.187.201.209]
8   144   0 0.0 158 165 158  vl-3201-ve-128.ebr2.London2.Level3.net [4.69.202.177]
9   144   0 0.0 157 159 157  ae-44-44.ebr1.Paris1.Level3.net [4.69.159.94]
10   144   0 0.0 158 159 158  ae-61-61.csw1.Paris1.Level3.net [4.69.161.78]
11   144   0 0.0 158 160 158  ae-62-62.ebr2.Paris1.Level3.net [4.69.161.93]
12   144   0 0.0 159 164 159  ae-44-44.ebr2.Washington1.Level3.net [4.69.137.62]
snip

Target Name: 208-64-200-204.steampowered.com
         IP: 208.64.200.204
  Date/Time: 22/04/2014 02:44:27 to 22/04/2014 02:45:12
Hop Sent Err  PL% Min Max Avg  Host Name / [IP]
1    81  33 40.7   0   0   0  home.gateway.home.gateway [192.168.1.254]
2    84   0  0.0  13 107  15  lo0-central10.ptw-ag04.plus.net [195.166.128.194]
3    84   0  0.0  13  13  13  link-a-central10.ptw-gw01.plus.net [212.159.2.156]
4    84   0  0.0  12  38  13  xe-7-2-0.ptw-cr01.plus.net [212.159.0.252]
5    84   0  0.0  12 350  59  te-3-4.car5.London1.Level3.net [217.163.45.181]
6    84   0  0.0 151 161 152  ae-52-52.csw2.London1.Level3.net [4.69.139.120]
7    84   0  0.0 152 154 152  ae-56-221.ebr2.London1.Level3.net [4.69.153.129]
8    84   0  0.0 152 153 152  ae-43-43.ebr1.NewYork1.Level3.net [4.69.137.74]
snip


Whether the route goes through Paris or not appears to be depend on the route taken through plusnets network.
So, to some extent, this must be under Plusnets control.
Both tracert are on the same gateway.
Both taken within 15minutes of each other.

AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: LEVEL3 peering to USA heavy Packet loss tonight

Once the traffic leaves Plusnet's network, they have little or no control over where it goes (contrary to what deathtrap thinks).
Plusnet do not control whether your traffic goes to Level3's London or Paris cores - that's decided by Level3.
deathtrap
Grafter
Posts: 1,064
Thanks: 4
Registered: ‎23-04-2013

Re: LEVEL3 peering to USA heavy Packet loss tonight

But ISP's do have a choice of which peering providers they peer directly with, they are called peering agreements, Plusnet appear to favoured Level 3, over other available choices for some usa destinations, And i have no doubt that even level3 will offer different service levels of peering, So this boils down to expenditure and how important the ISP regards it's peering , If they don't care and opt for the economy service level ,and pass the buck when something goes wrong, that's my opinion , regardless of what Andy h's  opinion is, the way that he always seems to be ready to defend plusnet  one does wonder if he's being rewarded for his loyalty , what is it with ISP's & forums and those few that will always defend their ISP regardless i believe they get called  fanbois  
And that was for the dig he had at me
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: LEVEL3 peering to USA heavy Packet loss tonight

So who do you want Plusnet to peer with?
npr
Pro
Posts: 1,898
Thanks: 119
Fixes: 9
Registered: ‎21-01-2013

Re: LEVEL3 peering to USA heavy Packet loss tonight

Quote from: AndyH
Once the traffic leaves Plusnet's network, they have little or no control over where it goes (contrary to what deathtrap thinks).

That goes without saying, but the point where plusnet connect in to the level3 network does appear influence the route.
Hopefully PN has some control over the point where they connect to level3.
dave
Plusnet Help Team
Plusnet Help Team
Posts: 12,257
Thanks: 306
Fixes: 4
Registered: ‎04-04-2007

Re: LEVEL3 peering to USA heavy Packet loss tonight

We've got a case raised with Level3 about this, will let you know what we hear back.
Dave Tomlinson
Enterprise Architect - Network & OSS
Plusnet Technology