Akamai v6 - peering issue?

[#1]

Quote from: AndyH

@ Bob - Did you ever find out what the cause of this was here http://community.plus.net/forum/index.php/topic,126633.msg1116201.html#msg1116201 ?

Sadly not, sorry
I only recently switched back to using IPv6 too. Had a bit of a sabbatical whilst testing something else over recent months.

[#2]

Ah - it would be interesting to see if the problem re-occurs for you.
It would seem that the issue is Akamai's end. I found a few forums where users have had the same issues with Facebook images hosted on Akamai's CDN over IPv6.
Out of interest, do you have any contacts at Akamai to reach out to them to see if there's a known issue at the moment?

[#3]

Quote from: AndyH

Out of interest, do you have any contacts at Akamai to reach out to them to see if there's a known issue at the moment?

I'm assuming Paul might, but I don't personally. I don't think he's in the office at present else I'd give him a nudge.

[#4]

Just come across this which is of interest:

That's traffic over our Akamai caching servers. Took a dip the same day you posted about problems.
The traffic is being picked up by the peering links on the core routers, and based on a problem our engineers are working on (ref: 84186), those links are creeping close to maximum utilisation. I believe we're in the process of trying to get things sorted with Akamai so it will be interesting to know whether or not your problem goes away once the caches are running at normal capacity again.

[#5]

Thanks.
That's interesting to know. I take it's not specific to IPv6 traffic?
I saw this thread http://community.plus.net/forum/index.php/topic,135270.0.html and was wondering if it was perhaps related. It will be useful to see if the OP can identify whether it's Akamai hosted content that is not loading for him.

[#6]

The links aren't specific to IPv6, no.
I doubt the other poster's problem is related though as this issue with the caches only arose earlier this week.

[#7]

Hi
I'm on leave so haven't been following things that closely but like Bob said there was an issue with the cache last week.
In answer to your question (AndyH) about the problem we had earlier in the year, I disabled the v6 peering directly to Akamai (we didn't have the cache then) which fixed the intermittent problem you saw.  Akamai seemed to suggest they didn't change anything to fix it but after a bit of back and forth the issue went away.

[#8]

Hi Paul
Thanks for the info.
It's interesting how no one else is seeing the issue as me, which is on-going at the moment. There doesn't appear to be any correlation between the gateways and the issue for me. The only consistent thing is some *.akamaihd.net hosted content does not load and timeouts out.
I'm using OpenDNS for v6, although I will see if the PN assigned servers make any difference.

[#9]

Its interesting you mention the problem only affects akamaihd.net domains. You could say that I am rather well versed with the Akamai network, so may be able to help in some small way.
If you see a similar issue, it would be most helpful if you could provide a tracert to the ipv6 akamaihd.net server your browser is connecting to (i tend to just check netstat myself, but there are plenty of methods as I am sure you know). At the same time, if you could do the same for this domain: a1007.dspr.akamai.net that will give me an idea of the routing being assigned to you at the time.
OpenDNS should be a non issue - Akamai and OpenDNS work together to keep things moving correctly there. Im not saying there cant be issues, its just highly unlikely to be because of using OpenDNS

[#10]

Surely the DNS server can make a difference. I have my router set to use Google's IPv6 DNS server (2001:4860:4860::8888) and get:

C:\Users\John>tracert a1007.dspr.akamai.net
Tracing route to a1007.dspr.akamai.net [2a01:3e0:1700::50e7:e460]
over a maximum of 30 hops:
  1     1 ms     1 ms     1 ms  dsldevice.lan [2a02:16c8:2000:******]
  2    37 ms    17 ms    17 ms  2a02:16c8:0:1::1f
  3    18 ms    19 ms    17 ms  2a02:16c8:1:2::d
  4    19 ms    17 ms    17 ms  2a02:16c8::b
  5    24 ms    23 ms    23 ms  lonap.he.net [2001:7f8:17::1b1b:1]
  6    25 ms    31 ms    16 ms  10ge5-2.core1.lon2.he.net [2001:470:0:2cd::1]
  7    25 ms    25 ms    25 ms  100ge5-1.core1.par2.he.net [2001:470:0:2ce::2]
  8    38 ms    39 ms    39 ms  globeinternet-as6453.10gigabitethernet11-2.core1.par2.he.net [2001:470:0:2c4::2]
  9    33 ms    33 ms    34 ms  if-ae2.2.tcore1.PYE-Paris.ipv6.as6453.net [2a01:3e0:fff0:400::e]
 10    33 ms    33 ms    34 ms  if-ae3.6.tcore1.L78-London.ipv6.as6453.net [2001:5a0:2000:400::29]
 11    32 ms    32 ms    32 ms  2a01:3e0:1700::50e7:e460
Trace complete.

Testing other servers and tracing to the result:

C:\Users\John>nslookup a1007.dspr.akamai.net 212.159.6.9
Server:  cdns01.plus.net
Address:  212.159.6.9
Non-authoritative answer:
Name:    a1007.dspr.akamai.net
Addresses:  2a02:16c8:5fff:6::d438:4911
          2a02:16c8:5fff:6::d438:491a
          212.56.73.91
          212.56.73.88

C:\Users\John>tracert 2a02:16c8:5fff:6::d438:4911
Tracing route to 2a02:16c8:5fff:6::d438:4911 over a maximum of 30 hops
  1     1 ms     1 ms     1 ms  dsldevice.lan [2a02:16c8:2000:*****]
  2    18 ms    18 ms    17 ms  2a02:16c8:0:1::1f
  3    17 ms    17 ms    17 ms  2a02:16c8:1:2::b
  4    17 ms    19 ms    17 ms  2a02:16c8::a
  5    17 ms    17 ms    18 ms  2a02:16c8:1:2::7
  6    18 ms    17 ms    21 ms  2a02:16c8:5fff:6::d438:4911
Trace complete.
C:\Users\John>nslookup a1007.dspr.akamai.net 208.67.222.222
Server:  resolver1.opendns.com
Address:  208.67.222.222
Non-authoritative answer:
Name:    a1007.dspr.akamai.net
Addresses:  2a02:16c8:5fff:6::d438:491a
          2a02:16c8:5fff:6::d438:4911
          212.56.73.91
          212.56.73.88

C:\Users\John>tracert 2a02:16c8:5fff:6::d438:491a
Tracing route to 2a02:16c8:5fff:6::d438:491a over a maximum of 30 hops
  1     1 ms    <1 ms     1 ms  dsldevice.lan [2a02:16c8:2000:*****]
  2    19 ms    22 ms    17 ms  2a02:16c8:0:1::1f
  3    17 ms    17 ms    17 ms  2a02:16c8:1:2::b
  4    17 ms    17 ms    23 ms  2a02:16c8::a
  5    17 ms    19 ms    17 ms  2a02:16c8:1:2::7
  6    17 ms    17 ms    16 ms  2a02:16c8:5fff:6::d438:491a
Trace complete.

[#11]

Hi Plexy
Here's are the traceroutes:

Quote

$ traceroute6 -I instagramstatic-a.akamaihd.net traceroute6: Warning: a1168.dsw4.akamai.net has multiple addresses; using 2001:668:108:d::d5c8:6c1b traceroute6 to a1168.dsw4.akamai.net (2001:668:108:d::d5c8:6c1b) from 2a02:16c8:2000:1100:fc0f:fb43:69b9:d0c, 64 hops max, 16 byte packets 1  2a02:16c8:xxxx:xxxx::x  0.435 ms  0.391 ms  0.334 ms 2  2a02:16c8:0:1::1e  6.273 ms  47.661 ms  114.953 ms 3  2a02:16c8:1:2::d  4.637 ms  4.422 ms  4.511 ms 4  2a02:16c8::b  4.516 ms  4.658 ms  4.188 ms 5  lonap.he.net  10.105 ms  9.135 ms  4.565 ms 6  10ge5-2.core1.lon2.he.net  8.319 ms  4.469 ms  4.213 ms 7  2001:7f8:4::cb9:1  4.754 ms  4.954 ms  4.865 ms 8  xe-9-3-1.lon11.ip6.gtt.net  5.116 ms  5.167 ms  4.816 ms 9  2001:668:108:d::d5c8:6c1b  4.922 ms  5.162 ms  5.363

Quote

$ traceroute6 -I a1007.dspr.akamai.net traceroute6: Warning: a1007.dspr.akamai.net has multiple addresses; using 2a01:3e0:1700::50e7:e460 traceroute6 to a1007.dspr.akamai.net (2a01:3e0:1700::50e7:e460) from 2a02:16c8:2000:1100:fc0f:fb43:69b9:d0c, 64 hops max, 16 byte packets 1  2a02:16c8:xxxx:xxxx::x  0.384 ms  0.315 ms  0.319 ms 2  2a02:16c8:0:1::1e  11.146 ms  9.152 ms  7.490 ms 3  2a02:16c8:1:2::d  4.751 ms  4.137 ms  4.474 ms 4  2a02:16c8::b  4.651 ms  4.422 ms  4.154 ms 5  lonap.he.net  4.114 ms  4.333 ms  4.184 ms 6  10ge5-2.core1.lon2.he.net  4.149 ms  4.175 ms  4.321 ms 7  100ge5-1.core1.par2.he.net  12.721 ms  20.374 ms  12.476 ms 8  globeinternet-as6453.10gigabitethernet11-2.core1.par2.he.net  26.272 ms  26.551 ms  26.295 ms 9  if-ae2.2.tcore1.pye-paris.ipv6.as6453.net  26.650 ms  26.465 ms  26.301 ms 10  if-ae3.6.tcore1.l78-london.ipv6.as6453.net  25.352 ms  25.393 ms  25.396 ms 11  2a01:3e0:1700::50e7:e460  19.958 ms  20.568 ms  20.392 ms

Quote

$ traceroute6 -I fbstatic-a.akamaihd.net traceroute6: Warning: a1168.dsw4.akamai.net has multiple addresses; using 2001:668:108:d::d5c8:6c1b traceroute6 to a1168.dsw4.akamai.net (2001:668:108:d::d5c8:6c1b) from 2a02:16c8:2000:1100:fc0f:fb43:69b9:d0c, 64 hops max, 16 byte packets 1  2a02:xxxx:xxxx::x  0.375 ms  0.377 ms  0.347 ms 2  2a02:16c8:0:1::1e  5.234 ms  5.332 ms  5.117 ms 3  2a02:16c8:1:2::d  4.433 ms  4.718 ms  4.244 ms 4  2a02:16c8::b  23.828 ms  4.294 ms  4.586 ms 5  lonap.he.net  4.737 ms  11.261 ms  4.558 ms 6  10ge5-2.core1.lon2.he.net  8.103 ms  4.211 ms  4.317 ms 7  2001:7f8:4::cb9:1  4.963 ms  5.231 ms  4.870 ms 8  xe-9-3-1.lon11.ip6.gtt.net  5.677 ms  5.042 ms  4.989 ms 9  2001:668:108:d::d5c8:6c1b  5.274 ms  5.443 ms  5.027 ms

Quote

$ traceroute6 -I fbcdn-profile-a.akamaihd.net traceroute6: Warning: a2047.dspl.akamai.net has multiple addresses; using 2001:728:1401::5119:c6b1 traceroute6 to a2047.dspl.akamai.net (2001:728:1401::5119:c6b1) from 2a02:16c8:2000:1100:fc0f:fb43:69b9:d0c, 64 hops max, 16 byte packets 1  2a02:xxxx:xxxx::x  0.387 ms  0.281 ms  0.293 ms 2  2a02:16c8:0:1::1e  7.647 ms  11.991 ms  8.004 ms 3  2a02:16c8:1:2::b  4.511 ms  4.750 ms  4.546 ms 4  2a02:16c8::a  14.220 ms  4.575 ms  4.309 ms 5  40ge1-3.core1.lon2.he.net  12.667 ms  7.982 ms  4.532 ms 6  ge-0.linx.londen03.uk.bb.gin.ntt.net  5.661 ms  5.749 ms  5.806 ms 7  ae-0.r23.londen03.uk.bb.gin.ntt.net  5.734 ms  5.757 ms  5.745 ms 8  ae-8.r04.parsfr01.fr.bb.gin.ntt.net  20.644 ms  20.028 ms  22.562 ms 9  2001:728:1401::5119:c6b1  12.750 ms  13.400 ms  12.935 ms

I've been playing around with it this morning and it's an MTU issue (again!). Nothing has changed my end for quite some time, but I've had issues with Facebook and Instagram since the end of Dec with every device on my network. This morning I rebooted my router, ended up on ptw-ag02 and I couldn't access a lot of dual stacked sites like thinkbroadband.com.
I played around with the MTU on my Mac and set it to 1492. Now everything loads again, including Facebook and Instagram. If I change it to 1493, quite a few dual stack sites will not load.
A bit more digging though...
I've now changed to a BNG gateway, ptn-bng02, and with an MTU of 1500 on my Mac, I can access sites like thinkbroadband.com again without any issues over v6. However, parts of the Akamai CDN for Instagram/Facebook do not load and are stuck at pending. If I drop my MTU back to 1492 again, everything loads fine.

[#12]

Thanks both for the traces and feedback - I will will do some digging my end to see what I can find out
From the replies, I can see two things that stand out. Lets look at the first, around MTU;
MTU issues are known to cause problems with dual stack for many services, not just Akamai hosted ones. I run a IPv6 tunnel from HE.net as I missed out on the PN trial, but that I have to force my MTU to 20 bytes below that of the PPPoE line MTU set in my router (1492 is the MTU over FTTC to make sure our packets move through the BTOR network correctly - so 1472 is what I set my HE.net IPv6 tunnel MTU). Without that, I get the same symptoms you are reporting when trying to load FB pages (and a host of other non-akamai sites that run dual stack too).
In general this problem is caused by a shift in the way MTU is handled in IPv6. In IPv4 packet fragmentation could be handled anywhere en route, so any given router could go 'hey this packet is too large, lets fragment.". In IPv6, the packet can only be fragmented at source. So what can end up happening is you get a 1500 packet, which is then encapsulated over an IPv6 in IPv4 tunnel and the addition of the tunneling headers puts the packet at 1520 bytes. In IPv4 the upstream kit worries about that so its non-issue. In IPv6, the source only can do fragmentation so the packet is dropped as its larger than the MTU (or possibly truncated, im not sure - either way its the same result, no remote host reply). A good explanation is on this page http://www.potaroo.net/ispcol/2009-01/mtu6.html (see the 5 paragraphs above the sub heading "why is this happening")
That being said, using a IPv6 MTU 20 bytes less than MTU really should only apply when using IPv6 tunneling. Native IPv6 with no 6 over 4 tunnel between client and server should (at least as far as I can ascertain) be happy with using the 1492 MTU.

So ; if you are using IPv6 over PPPoE then your max MTU should be 1492 and no larger. If using a IPv6 in IPv4 tunnel over a PPPoE connections, MTU should be 1472 and no larger.

So now lets take a look at the DNS resolutions. Akamai relies on DNS to route users to the nearest edge server for fast delivery;

Quote

Surely the DNS server can make a difference. I have my router set to use Google's IPv6 DNS server (2001:4860:4860::8888)

It can if you are using a IPv6 tunnel provider. It actually should not if you are using a native IPv6 connection from your ISP as Akamai has been using the client-subnet directive of EDNS0 for quite some time - both Google and OpenDNS work with Akamai and have implemented EDNS0. This means that even when using a open DNS resolver, the Akamai network can still make a decision to route to the nearest Akamai IP based on your client subnet.  
But when using an IPv6 tunneling provider, the Akamai network (and any CDN for that matter) will see your subnet in the EDNS0 DNS packets as arriving from that tunnel provider. The Akamai network would then have no valid reason to route your request to Akamai server caches inside of the PN network (much the same as an end use on "Big ISP A" wouldn't be routed to the Akamai caches within the internal network of "Big ISP B").
While that is pertinent knowledge, from your tracert is you do not appear to be using a tunneling provider you are on on native IPv6 from PN. Yet somehow, your packet gets routed off to HE.net anyway (and I wont even ask what HE.net internal network routing is doing sending your packets for a jolly around Europe and back first, lets stick to the matters at hand)

Quote

C:\Users\John>tracert a1007.dspr.akamai.net Tracing route to a1007.dspr.akamai.net [2a01:3e0:1700::50e7:e460] over a maximum of 30 hops:  1     1 ms     1 ms     1 ms  dsldevice.lan [2a02:16c8:2000:******]  2    37 ms    17 ms    17 ms  2a02:16c8:0:1::1f  3    18 ms    19 ms    17 ms  2a02:16c8:1:2::d  4    19 ms    17 ms    17 ms  2a02:16c8::b  5    24 ms    23 ms    23 ms  lonap.he.net [2001:7f8:17::1b1b:1]  6    25 ms    31 ms    16 ms  10ge5-2.core1.lon2.he.net [2001:470:0:2cd::1]  7    25 ms    25 ms    25 ms  100ge5-1.core1.par2.he.net [2001:470:0:2ce::2]  8    38 ms    39 ms    39 ms  globeinternet-as6453.10gigabitethernet11-2.core1.par2.he.net [2001:470:0:2c4::2]  9    33 ms    33 ms    34 ms  if-ae2.2.tcore1.PYE-Paris.ipv6.as6453.net [2a01:3e0:fff0:400::e] 10    33 ms    33 ms    34 ms  if-ae3.6.tcore1.L78-London.ipv6.as6453.net [2001:5a0:2000:400::29] 11    32 ms    32 ms    32 ms  2a01:3e0:1700::50e7:e460

Do you by any chance have multiple IPv6 Interfaces? What IP's are your DNS resolvers set to (ipconfig /all output, if you can?)
I will PM you a link I would like you to click for me. You can validate the domain ownership if you are concerned about clicking links given to you by other people. Its a simple diagnostic tool which will allow me to ascertain further information about what the Akamai network saw and how it routed you.
Cheers!

[#13]

scratch that PM of diagnostic link, its IPv4 only. Looks like this is going to be done the old fashioned way!
Please could both of you PM me your IPv6 address and the output of ipconfig /all ?

[#14]

What I don't quite get is how everything works fine with v6 and then all of a sudden Akamai hosted content stops working - even though nothing has changed my end. Even today, a reboot of my router and dual stacked sites stop working on ptw-ag02 for no obvious reason. Moving to a BNG seems to fix that for dual stacked sites, with the exception of Akamai v6 content.
The interesting thing is that somewhere between my network and the end hosts, packets are being dropped because of the MTU size however there are no "packet too big" messages coming back for whatever reason (or at least they appear to not be coming back).

[#15]

funny enough plexy I found the same, even on native ipv6 I had to set - 20bytes for android to be fully happy.  so 1472 mtu on ipv6.