topic Re: IPv6 Connectivity Problems in IPv6 Trial

IPv6 Connectivity Problems

MJN — Sun, 08 Jul 2012 11:39:48 GMT

Hi Support/Everyone,
From Mon 2nd July I seem to have lost IPv6 connectivity and whilst I'm pretty sure I didn't change anything that day (I was at work when the failure occured (10-11am)) I cannot rule out something like a DHCP expiry unearthing a latent configuration error at my end.
I am using a Cisco 877 router and can obtain an IPv6 prefix via DHCP:

Cisco877#clear ipv6 dhcp client dial 1
Cisco877#sh ipv6 dhcp int dial 1
Dialer1 is in client mode
  Prefix State is OPEN (0)
  Information refresh timer expires in 23:59:54
  Renew will be sent in 7w0d
  Address State is IDLE
  List of known servers:
    Reachable via address: FE80::90:1A00:3A3:8D54
    DUID: 000200000A4C453332302F373435414333334558322F03
    Preference: 0
    Configuration parameters:
      IA PD: IA ID 0x000D0001, T1 INFINITY, T2 INFINITY
        Prefix: 2A02:16C8:2000:600::/56
                preferred lifetime INFINITY, valid lifetime INFINITY
      DNS server: 2001:470:20::2
      Information refresh time: 0
  Prefix name: MYPLUSNETPREFIX
  Prefix Rapid-Commit: disabled
  Address Rapid-Commit: disabled

The router then sets itself a link-local IPv6 address and a global IPv6 address on VLAN1 with this prefix using auto-configuration:

Vlan1                      [up/up]
    FE80::B6A4:E3FF:FE11:A57A
    2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A

At this point I can ping the DHCP server using (from) my link local address:

Cisco877#ping ipv6 FE80::90:1A00:3A3:8D54
Output Interface: dial 1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to FE80::90:1A00:3A3:8D54, timeout is 2 seconds:
Packet sent with a source address of FE80::B6A4:E3FF:FE11:A57A
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 36/39/40 ms

But I cannot ping anything beyond that (e.g. the DNS server referenced in the DHCP response) from my global address:

Cisco877#ping ipv6 2001:470:20::2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:470:20::2, timeout is 2 seconds:
HHHHH
Success rate is 0 percent (0/5)

Similarly, I cannot ping in from an external IPv6-enabled machine:

mathew@targur:~$ ping6 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A
PING 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A(2a02:16c8:2000:600:b6a4:e3ff:fe11:a57a) 56 data bytes
From 2001:ba8:0:1f1::1 icmp_seq=1 Destination unreachable: No route
From 2001:ba8:0:1f1::1 icmp_seq=2 Destination unreachable: No route
From 2001:ba8:0:1f1::1 icmp_seq=3 Destination unreachable: No route
^C
--- 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2006ms

The 'no route' response is interesting, and if we look at the traceroute it doesn't seem to be getting even as far as Plusnet, nevermind all the way to me:

mathew@targur:~$ traceroute6 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A
traceroute to 2A02:16C8:2000:600:B6A4:E3FF:FE11:A57A (2a02:16c8:2000:600:b6a4:e3ff:fe11:a57a), 30 hops max, 40 byte packets
 1  2001:ba8:1f1:f137::1 (2001:ba8:1f1:f137::1)  0.151 ms  0.194 ms  0.181 ms
 2  jump-gw-1.ipv6.bitfolk.com (2001:ba8:0:1f1::1)  0.906 ms !N  0.990 ms !N  1.367 ms !N

Are you able to check/confirm that my prefix is actually being announced via BGP?
Thanks for reading this far, and if there's anything anyone can suggest it'd be appreciated. It's always frustrating when it has been working for a long time (over a year now) and then one day just stops!! That said, the short traceroute suggests it's a Plusnet issue but I stand to be corrected.
Mathew

Re: IPv6 Connectivity Problems

brueton — Sun, 08 Jul 2012 11:46:03 GMT

I have checked on two connections and both have IPv6 connection problems.
Both are on the trial and have been working OK until recently.
PeteB

Re: IPv6 Connectivity Problems

brueton — Sun, 08 Jul 2012 11:58:59 GMT

thinkbroadband.com quality monitor is showing 100% packet loss for ipv6, ipv4 is OK.
I suspect some change has been made by Plusnet that has caused the ipv6 connectivity to stop working.
PeteB

Re: IPv6 Connectivity Problems

MJN — Sun, 08 Jul 2012 12:00:04 GMT

Thanks Pete. That's quite reassuring in a perhaps-its-not-something-I've-done kind of way!
Mathew

Re: IPv6 Connectivity Problems

brueton — Sun, 08 Jul 2012 12:08:32 GMT

One of the routers is a 7800N and the other a TG582n so the problem is not related to the hardware being used.
Last ipv6 connection on the quality monitor is approx. 10:30 on Monday 2nd July 2012.
Pete

Re: IPv6 Connectivity Problems

MJN — Sun, 08 Jul 2012 12:13:28 GMT

That coincides with when mine went down so thanks for mentioning that.
Mathew

Re: IPv6 Connectivity Problems

MJN — Sun, 08 Jul 2012 21:00:56 GMT

Incidentally, the consequences of this situation - apparent IPv6 capability but no actual offsite connectivity - has caused problems for some devices which might be worthy of consideration for an eventual future rollout.
Firstly, there is the relatively common symptom of browsers (Firefox 13 here) attempting to connect over IPv6 and falling back to IPv4 only after a timeout thus causing sluggish browsing to sites with IPv6 AAAA DNS records. The 'Happy Eyeballs'[1] technique (and similar) may eventually alleviate this problem but for now it is rather frustrating.
Worse than sluggish performance though I am finding that some applications behave very badly. For example, Google's 'Play' (app store) on Android seems to trip over itself if it thinks it has IPv6 connectivity yet fails to get packets delivered - it just waits and waits before eventually giving up (i.e. it doesn't fall back to IPv4). This is occuring on two devices, both running Gingerbread 2.3.5.
This problem is likely to get far worse before it starts to get better as more and more sites are starting to add AAAA records now that they're supporting IPv6.
Mathew
[1] It would appear that Firefox does implement HE but it doesn't seem to be working particularly effectively. To be fair the delay is in the order of a few seconds, and so a lot better than the 10's of seconds that used to occur a few years ago with a broken IPv6 setup, but in today's want-it-now society I'm sure it'll still be too long for many people!

Re: IPv6 Connectivity Problems

w23 — Mon, 09 Jul 2012 06:31:07 GMT

Similar problems here.
I managed to ping your IPv6 address above, MJN, though the ping time was >600ms.
I have disabled IPv6 on my router for now as it's currently a pain.

Re: IPv6 Connectivity Problems

MJN — Tue, 10 Jul 2012 07:27:32 GMT

Hi Support,

Quote from: MJN
Are you able to check/confirm that my prefix is actually being announced via BGP?

Just to follow up on this, your entire prefix has dropped out of the routing table:

route-views.linx.routeviews.org> show ipv6 bgp 2a02:16c8::/32
% Network not in table

Mathew

Re: IPv6 Connectivity Problems

prichardson — Tue, 10 Jul 2012 10:56:17 GMT

Morning folks.
I am doing some digging to see if we can see what may be going on. There are some minor suspicions giving some of the recent network maintenance we have been undertaking, but they are only that initially.
Can I confirm though, does IPv4 connectivity remain unaffected when using the IPv6 test account?

Re: IPv6 Connectivity Problems

MJN — Tue, 10 Jul 2012 11:06:18 GMT

Yes, can confirm that IPv4 is fine on the test account.
It looks to me like it's simply that your BGP announcements to the outside world for IPv6 have stopped. I suppose though that that could be a consequence of something gone awry internally so more of a symptom than the cause?
Mathew

Re: IPv6 Connectivity Problems

brueton — Tue, 10 Jul 2012 11:21:01 GMT

IPv4 also OK on both my and my friend's IPv6 test accounts.
Pete

Re: IPv6 Connectivity Problems

prichardson — Tue, 10 Jul 2012 11:33:35 GMT

MJN,
Our original thoughts on a cause may not be to blame, though until I get a response from our network operations team, we are not ruling out any fringe side-effects.
PeteB,
I cannot locate your main account, thus cannot workout which IPv6 test account you are on.
Can you let me know the main username of the line you are testing IPv6 on, as I can get everything else from this.
Until I have this, I cannot work the rest out.

Re: IPv6 Connectivity Problems

brueton — Tue, 10 Jul 2012 11:40:22 GMT

PM sent with account information.
PeteB

Re: IPv6 Connectivity Problems

prichardson — Tue, 10 Jul 2012 12:09:34 GMT

Thanks.
Checking through that, I thought was had seen a common denominator of you being on PCL gateways. However, the second of the two accounts PeteB quoted in presently on PTN.
One of our newer Network Ops has had a look (I say newer, he's actually a returning member), who is not fully upto speed on the setup that is in for our IPv6 trials.
He has confirms that part of our core does not do IPv6 routing as of yet, but that could ultimately be per present design. His reply also indicates he thinks something is wrong our side, but all elements on balance suggests we could also be looking in the wrong places.
I'm waiting to find out if one of our other core network engineers is around who most certainly knows the IPv6 elements.

Re: IPv6 Connectivity Problems

brueton — Tue, 10 Jul 2012 12:24:39 GMT

Following a bit of gateway hopping I am now on ptn-ag04, still have the problem.
Whatever caused the problem happened at approx. 10:30 am on Monday 02 July 2012?
PeteB

Re: IPv6 Connectivity Problems

prichardson — Wed, 11 Jul 2012 16:28:14 GMT

Well, I now have an explanation, but not the best of news for you.
As per the original technical trial description, the service may have been removed at any time. In this case, we have not specifically removed the trial itself, but it has ended early due to changes we have made within our core network as part of our platform upgrades.
Our core network has been replaced and we have not built the technical trial into the configuration of the gateways concerned.
As a result IPv4 will continue to work unaffected. However those using a IPv6 trial account will see connection delays whilst connection to dual IPv6/IPv4 services. IPv4 only sites are fine and IPv6 only will not work for more obvious reasons.
Due to this, for the better experience, return to your standard account logins.
I will be drafting something better together to go within the forum shortly, but thought I would drop a heads up so it is known. I will put more down within that, but IPv6 is unlikely to return until we commence wider service trials.
What this will entail and when this commenced I don't have the full details on, but we are not hoping this to be too long.

Re: IPv6 Connectivity Problems

MJN — Wed, 11 Jul 2012 18:42:28 GMT

Wow, thanks Phil for that frank revelation - it certainly wasn't what I was expecting! I can't help but feel we've been having the best house party ever and the parent's have come back without warning.... The music has suddenly stopped and everyone's being told to get out sharpish!
It's a shame it had to end like this, particularly not knowing in advance although it is clear from your earlier responses that you and at least some of your colleagues were clearly kept in the dark as much as we were so there's no criticism aimed at you there.
By their very nature trials must always come to an end however deep down I was hoping that at its conclusion we'd be seeing a full IPv6 rollout without any gap inbetween.
Now the party's over I suppose it's back to the tunnels... Speaking of which, may I ask if the 'IPv6 Trial' sub-forum could remain (renamed if deemed appropriate) in order for customers to continue discussion of the protocol, its implementation and configuration etc?
Mathew

Re: IPv6 Connectivity Problems

prichardson — Wed, 11 Jul 2012 19:39:29 GMT

Yes, there are no plans for removal of the forum.
IPv6 is a big big deal. Ignoring the wider Internet and the principals of IPv6 being very much capable over the existing technologies, we have something very new to the majority.
This is obviously a big deal, with not just our network supporting it, our management systems integrating to it, supplying our front line services via it and to the internet, but the biggest is supporting it.
IPv6 is quite a foreign concept, to me as much as anything and I have dabbled in it time a number of times in the past 11 years. It's not that people wont get it, but it's a big ask to take some much on so quickly.
This forum will support that in the early adoption stages of services trials.