cancel
Showing results for 
Search instead for 
Did you mean: 

Akamai v6 - peering issue?

plexy
Grafter
Posts: 38
Registered: ‎28-08-2013

Re: Akamai v6 - peering issue?

Quote from: chrcoluk
funny enough plexy I found the same, even on native ipv6 I had to set - 20bytes for android to be fully happy.  so 1472 mtu on ipv6.

Quote from: AndyH
What I don't quite get is how everything works fine with v6 and then all of a sudden Akamai hosted content stops working - even though nothing has changed my end.

Interesting. I dont know how the carriage between PN and Akamai is set up, or how PNs internal IPv6 is handed. But having to set MTU of 1472 on native IPv6 would, to me, indicate there is a IPv4 tunnel being used somewhere upstream. If it only happens 'sometimes' then perhaps there is some native connectivity and some tunneled connectivity upstream, when native is used the 1492 is fine, but when tunneled is used you need 1472. I would be inclined to think that, based on your traceroutes, that if there is a tunnel involved its not something thats plusnet<>akamai (as your traces show you going off to akamai nodes outside of plusnets network, as opposed to the internal nodes). Perhaps Paul or Bob could confirm if there is any PN IPv6 egress which is tunneled over IPv4? Not necessarily to Akamai - just to 'anywhere'.
Dont forget, the Akamai node you are routed to can change every 30 seconds - so lets say you are using things just fine one minute, then the route changes and something breaks. It would be great to know the 'before' and 'after' routes used when that happens, though I see it being a pain to try to figure that out (perhaps some kind of IP logging/showing browser extension?)
Personally I would recommend setting your IPv6 MTU to 1472 anyway, as who knows what routing out there is tunneled - even out in transit providers. There could be a link somewhere which is affected, then another which is not and it just depends on which you get routed onto that the problem occurs or not. Again, speculation but seems likely given the symptoms.

So for me there are a few open questions;
- Is the max PPPoE MTU of 1492 valid for IPv6 native, or would 1472 be a better value just incase something somewhere out in transit land gets tunneled?
- Does PN have any IPv6 egress to any transit provider, peer or CDN which is tunneled over IPv4?
- Is there a problem with carrying the IPv6 family in EDNS0 when using google DNS (Jelv's tracert shows the PN Akamai nodes in use when using PN DNS or OpenDNS, but not when using Googles DNS. AndyH seems to not be routed to the PN Akamai nodes at all)
- Has Akamai got some internal IPv6 mapping issue thats contributing to this (I can try to validate this if you PM me the IPs you are making requests from).
We have to remember that the PN team do seem to be aware of an issue and its being looked at. That is probably compounding things. Lets imagine this scenario (hypothetical!):

- You do a DNS request for an akamaized site
- The PN Akamai nodes nearest to you has capacity, is in service and healthy
- Your request is kept inside the PN network, things display OK
Then this one;
- You do a DNS request for an akamaized site
- The PN Akamai nodes nearest to you are out of service/at capacity/under maintenance/whatever
- You are routed to the nearest available node, which takes you outside of PN's network

during technical issues, you could 'flip flop' between the two (up to every 20 seconds!). What if the problem is as I suspect, a MTU issue, but is only happening internal to some transit provider outside of PNs control. We know there is a potential PN<>Akamai issue, based on PN's graphs earlier in this thread. So if Akamai is routing a percentage of end user requests out of plusnet, through peering in order to keep the load down on PN Akamai nodes - AND - an upstream provider/Plusnet egress is using an ipv6 in ipv4 tunnel somewhere - what you see with a MTU of 1492 is 'this works.. yay'.... then later 'this doens t work.. booo'... rinse and repeat.
Try with MTU 1472
chrcoluk
Grafter
Posts: 1,990
Thanks: 5
Registered: ‎11-12-2013

Re: Akamai v6 - peering issue?

Does google play and youtube go over akamai?  Those were what I was having issues with on my phone.
plexy
Grafter
Posts: 38
Registered: ‎28-08-2013

Re: Akamai v6 - peering issue?

Quote from: chrcoluk
Does google play and youtube go over akamai?  Those were what I was having issues with on my phone.

Thats a grey area for me to answer unfortunately. Best way to find out if you are talking to an akamai server is nslookup/dig on the domain you are trying to reach. Anything hostname that is CNAMED to one of the Akamai 'primary domains' listed on Wikipedia http://en.wikipedia.org/wiki/Akamai_Technologies#Primary_domains is definitely akamaized. Similar for direct hostnames to akamaihd.net et al.
I do also note that elsewhere in the forums, the PN staff have stated that they have on-site caches for Netflix, Google and Akamai. It would make little sense to me for google to then route plusnet src Google/Youtube traffic over an Akamai node. I would hazard a guess that if you are experiencing issues over youtube and google services, as well as Akamai, then the problem is unlikely to be specific to Akamai. 
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: Akamai v6 - peering issue?

Quote from: plexy
So for me there are a few open questions;
- Is the max PPPoE MTU of 1492 valid for IPv6 native, or would 1472 be a better value just incase something somewhere out in transit land gets tunneled?

I've been running an MTU of 1500 for a little while now without any problems, although I have had issues in the past. At the moment, I can send a do-not-fragment 1500 bytes packet from an outside server to my home Mac without any issues.
There does seem to be a difference between the setup on the BNGs vs the older AGs. As far as I can see, the BNGs seem happier to handle an MTU > 1492 for IPv6 than the AGs. However, there seems to be something amiss with Akamai when I have a 1500 MTU on a BNG.
Quote from: plexy
I can try to validate this if you PM me the IPs you are making requests from

PM sent!
plexy
Grafter
Posts: 38
Registered: ‎28-08-2013

Re: Akamai v6 - peering issue?

Quote from: AndyH

I've been running an MTU of 1500 for a little while now without any problems, although I have had issues in the past. At the moment, I can send a do-not-fragment 1500 bytes packet from an outside server to my home Mac without any issues.
There does seem to be a difference between the setup on the BNGs vs the older AGs. As far as I can see, the BNGs seem happier to handle an MTU > 1492 for IPv6 than the AGs. However, there seems to be something amiss with Akamai when I have a 1500 MTU on a BNG.

Interesting - whats the MTU on your pppoe link? Generally it should be no more than 1492 as PPP adds 8 bytes of header and it is then itself carried over 1500byte ethernet presented at the CPE modem - so if you have a 1500 MTU at the TCP layer then problems start cropping up, especially with ipv6. Ipv4 will likely mask the issues due to fragmentation happening at hops. Of course, if the upstream provider supports baby jumbo frames over PPPoE (where your MTU would be set to 1508 - to give 1500 bytes of packet and 8 bytes of PPPoE header) this could easily work. Andrews&Arnold ISP did some work on this on the BTOR network a few years ago http://aa.net.uk/kb-broadband-mtu.html. Not sure if PN support that approach though as im not on the trial.
Also bear in mind that MTU is not MRU (though they should still be governed by the same ppp link packet requirements). Perhaps the receive path is allowing baby jumbo but not the xmit path? im clutching at straws here, mind you - but logically that would fit. PN would really need to specify what customers need to set as MTU on IPv6 based on the underlying last mile network and any upstream IPv6 carriage paths.
Quote from: AndyH
Quote from: plexy
I can try to validate this if you PM me the IPs you are making requests from

PM sent!

Grand, I can confirm the Akamai systems know that IP you sent me is Plusnet. I dont see anything untoward that would cause a mapping problem in DNS with your specific IP.
plexy
Grafter
Posts: 38
Registered: ‎28-08-2013

Re: Akamai v6 - peering issue?

Just had another thought - make sure your firewall (on router and local machine) is not blocking ICMP PMTU for ipv6. That can cause a whole host of headaches which can manifest as stalled connections/downloads
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: Akamai v6 - peering issue?

Quote from: plexy
Interesting - whats the MTU on your pppoe link? Generally it should be no more than 1492 as PPP adds 8 bytes of header and it is then itself carried over 1500byte ethernet presented at the CPE modem - so if you have a 1500 MTU at the TCP layer then problems start cropping up, especially with ipv6. Ipv4 will likely mask the issues due to fragmentation happening at hops. Of course, if the upstream provider supports baby jumbo frames over PPPoE (where your MTU would be set to 1508 - to give 1500 bytes of packet and 8 bytes of PPPoE header) this could easily work. Andrews&Arnold ISP did some work on this on the BTOR network a few years ago http://aa.net.uk/kb-broadband-mtu.html. Not sure if PN support that approach though as im not on the trial.

The PPPoE MTU is set to 1492. Even though this is lower than my Mac MTU, 1500 byte packets can be sent without fragmentation because the eth0 and vlan1 interfaces are set to 1500.
I think the Openreach/BTW networks are fine to cope with the 1500 byte packets for PPPoE. There's been enough upgrades etc. to make sure they are compatible.
This whole situation does raise quite a big issue in terms of a general IPv6 rollout. A lot of routers out the box will default to a router advertised MTU of 1500. With IPv4, this doesn't create an issue because of the way large packets are fragmented. However, with IPv6 it creates an issue because many networks have ICMP disabled for security reasons and therefore they will not provide 'packet too big' messages (if a packet is sent larger than the specified network MTU). This is where you get the situation of sites not loading properly etc.
Quote from: plexy
Grand, I can confirm the Akamai systems know that IP you sent me is Plusnet. I dont see anything untoward that would cause a mapping problem in DNS with your specific IP.

Thanks - I'll keep the response on the nameserver on here in case any PN guys want to review.
I am using OpenDNS (2620:0:ccc::2 and 2620:0:ccd::2). Previously, their v6 nameservers were based solely in Amsterdam (Telecity). However they have since rolled out globally v6 nameservers in all their data centre locations. In London, they are at Linx/LONAP - (2a04:e4c0:10::/48)
Quote
Non-authoritative answer:
which.opendns.com text = "5.lon"

chrcoluk
Grafter
Posts: 1,990
Thanks: 5
Registered: ‎11-12-2013

Re: Akamai v6 - peering issue?

Something I didnt mention before.
With radvd windows was fine, only android had issues.
With dnsmasq, windows also bombed out, ipv6 pages simply were waiting forever no throughput occuring beyond the dns lookup.
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: Akamai v6 - peering issue?

Quote from: plexy
Just had another thought - make sure your firewall (on router and local machine) is not blocking ICMP PMTU for ipv6. That can cause a whole host of headaches which can manifest as stalled connections/downloads

Yep, checked this one and it doesn't seem to be an issue my end.
Quote from: chrcoluk
With dnsmasq, windows also bombed out, ipv6 pages simply were waiting forever no throughput occuring beyond the dns lookup.

Is it possible you were on a AG gateway?
For me, I have major issues accessing pretty much everything when on a AG and a MTU of 1500. Again this morning, I've gateway hopped to ptw-ag04 and I've had to disable IPv6 to get pages to work.
Edit: Back on ptn-bng02 and everything is good again (other than Akamai!).
jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Akamai v6 - peering issue?

Strange that I'm not seeing issues. My MTU is still on the default of 1500 and because I'm on 20CN I only connect to ag gateways.
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: Akamai v6 - peering issue?

Does http://ipv6-test.com/pmtud/ come up with 1500 for you?
It is interesting you're not affected - I wonder if someone else on fibre can try and replicate setting a 1500 MTU on a AG gateway and then try sites like http://www.thinkbroadband.com ?
jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Akamai v6 - peering issue?

I had to force IPv6 using the hosts file as the first time it connected using IPv4, but I got:
Quote
Testing packet size: 1500
Result: no PMTUD problems detected
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,887
Thanks: 4,979
Fixes: 316
Registered: ‎04-04-2007

Re: Akamai v6 - peering issue?

Quote from: AndyH
Does http://ipv6-test.com/pmtud/ come up with 1500 for you?
It is interesting you're not affected - I wonder if someone else on fibre can try and replicate setting a 1500 MTU on a AG gateway and then try sites like http://www.thinkbroadband.com ?

I'm connected to ptn-ag03 from a fibre line. Using a 589vn and I chronicled what I did to set it up over here (there's some stuff in that thread concerning MTU that may be of interest).
Using the MTU test URL I get:
Testing packet size: 1500
Result: no PMTUD problems detected

ThinkBroadband loads without problems over IPv6.

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

chrcoluk
Grafter
Posts: 1,990
Thanks: 5
Registered: ‎11-12-2013

Re: Akamai v6 - peering issue?

Quote from: AndyH
Quote from: plexy
Just had another thought - make sure your firewall (on router and local machine) is not blocking ICMP PMTU for ipv6. That can cause a whole host of headaches which can manifest as stalled connections/downloads

Yep, checked this one and it doesn't seem to be an issue my end.
Quote from: chrcoluk
With dnsmasq, windows also bombed out, ipv6 pages simply were waiting forever no throughput occuring beyond the dns lookup.

Is it possible you were on a AG gateway?
For me, I have major issues accessing pretty much everything when on a AG and a MTU of 1500. Again this morning, I've gateway hopped to ptw-ag04 and I've had to disable IPv6 to get pages to work.
Edit: Back on ptn-bng02 and everything is good again (other than Akamai!).

I was on a AG gateway, the bng's cant do ipv6.
Also to point out when I was using 1500 for PPPoE I had pretty much no issues on ipv6, but sadly if this was to ever rollout 99% of people would be using a standard 1492 byte mtu.
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: Akamai v6 - peering issue?

Quote from: chrcoluk
I was on a AG gateway, the bng's cant do ipv6.

BNGs are working fine for IPv6 for me and have been for the past couple of months - although the RADIUS/prefix issue doesn't seem to have been fixed.