Interesting Thread on Reddit - IPv6 Broken Sites/MTU Issue

[#1]

Just seen this post on Reddit:

https://old.reddit.com/r/ipv6/comments/1osr448/rant_about_broken_dual_stack_sites/

The three sites posted all work fine for me, I'm on FTTP using PPPoE and a Plusnet Hub 2 but interested to see if anyone else with different setups has any issues with any of these sites:

https://www.o2.co.uk

https://www.dobbies.com

https://auth0.alaskaair.com

Thanks

[#2]

I saw that post too. All seems fine for me accessing those URLs over IPv6 - I'm on FTTP with PPPoE and a Draytek 2962. I do have my MTU set to 1490 though (autodetected by the router using PMTUD), and I do wonder if PMTUD is helping further with these specific destinations to lower things even further.

[#3]

All work perfectly for me - FTTP on PPOE on ZEN with a Fritz!Box 7530.

[#4]

Same setup as you @dave, and working for me.

[#5]

Interestingly enough with my setup - IP6 MTU consistently advertised at home at 1460 across the home subnets, IP4 MTU at 1500 (baby jumbos on the PPPoE connection), I see currently see timeouts on all 3 sites so I'll do some digging.

The background is that over the last week I have been fiddling around my home network configuration and wiring though, so I could have broken something whilst I'm cleaning up the firewalling and multi-uplink routing and overrides from when I had PlusNet's fttc and Three's 5G home broadband, and two separate HE tunnels over each of those links.  I'm now just running PlusNet FTTP + IPv6 trial.

IP6 MTU of 1460 chosen as that was the maximum value that the HE tunnels used to support, whilst I know the PlusNET side can do higher.  I'm not purposefully blocking ICMPv6 messages.

https://www.rfc-editor.org/rfc/rfc4890 is the RFC for firewalling and ICMPv6 that I'm following.

[#6]

Ok - some tcpdump data with a test to the O2 site.

Client - MacOS.  Terminal opened and using telnet -6 www.o2.co.uk and then when connected, GET / 1.0 enter-enter to fake a http get request.  This is sent as a very short packet, so avoids any mtu issues, and then expects the server response, which is typically large as it sends the web page requested.

On the linux router, I was running tcpdump -i any host www.o2.co.uk and ip6 -n to monitor.

I'm seeing exactly what the reddit thread is talking about.  The tcpdump output is attached and the key part at the end shows my router sending ICMP6, packet too big repeatedly before the connection just stalls out.

[#7]

Further follow up - visiting the path mtu testing page in that thread - http://pmtud.enslaves.us/ gave me the following results.

Results

Direction	Tested Maximum Size Segment	Client Sent MSS	Notes
Server to Client IPv4	1460	1460	OK
Client to Server IPv4	unlimited	(n/a)	OK
Server to Client IPv6	1400	1440	fragile, relies on ICMP or server help
Client to Server IPv6	unlimited	(n/a)	OK

So this does appear to show that my Mac when using IP6 is advertising MSS that's too large initially and is then dying badly if the server doesn't respect the too big message being sent.  I guess I'm going to be deploying 'mss clamping' for IP6 to address this.

Edit:

Ok.  Clamping the MSS to pmtu for IP6 has fixed this, and the O2 and other sites are now fine again.

Direction	Tested Maximum Size Segment	Client Sent MSS	Notes
Server to Client IPv4	1460	1460	OK
Client to Server IPv4	unlimited	(n/a)	OK
Server to Client IPv6	1400	1400	OK
Client to Server IPv6	unlimited	(n/a)	OK

The Firehol tcpmss command in the router6 section of the rules produced the following ip6ables command in the mangle table:

ip6tables -t mangle -I POSTROUTING -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

[#8]

I have taken the fix further as PlusNET supports 1500 byte IP6 and have adjusted my home network settings to run at 1500 not 1460 MTU for IP6.  ( I think this is at the root of the MSS issue as it appears no mainstream network stack supports different MTU for IP4 and IP6 on the same physical network segment.  MacOS appears to be using the IP4 MTU to calculate the initial IP6 MSS rather than the IP6 mtu value I was advertising via radvd. )

The upshot is the IPv6 Server To Client's Client Sent MSS is now 1440 and working, and the results page appears as follows.  The Mac is no longer over-advertising the MSS it was setup to accept:

Results

Direction	Tested Maximum Size Segment	Client Sent MSS	Notes
Server to Client IPv4	1460	1460	OK
Client to Server IPv4	unlimited	(n/a)	OK
Server to Client IPv6	1440	1440	OK
Client to Server IPv6	unlimited	(n/a)	OK

I now only have to clamp when using the tunnel broker path because of the requirement for the 6in4 encapsulation.

The three sites Dave listed still work as expected via PN's IP6.

sudo ping6 -D -s 1452 www.google.co.uk also works, as is a test that 1500 byte MTU IP6 packets are being transmitted and received correctly.

[#9]

@MPC glad that you've got it working. Interesting about the different MTUs for IPv4 and IPv6, did you have a Windows device, wondering if that was the same as the Mac? Not sure I can change MTU on a Hub 2 so might have to go find another device to try out.

[#10]

I do have a number of windows 11 devices but was only testing from the Mac mini I use as my main device.

It's now slightly awkward to rejig the MTU back to 1460 again, but I'll see about giving it a go to confirm directly.

When I was researching this, it looks like OpenWRT have this as a known issue and a bug was fixed around it back in 2023 relating to routing paths with different maximum MSS/MTU values requiring clamping.  Their case was VPN vs non-VPN IP4 traffic but the root cause (router forwarding connections from a client with a higher MTU than the output path's MTU) is effectively the same as I was seeing with IP4 MTU at 1500 and IP6 MTU (attempted) at 1460.

https://github.com/openwrt/openwrt/issues/12112

There's a similar post for Wireguard that covers off the various cases:

https://blog.silvio.cloud/2_WireGuard_and_MTU_MSS

[#11]

Don't worry about putting it back, more curious than anything else.

[#12]

Anecdotally, I was having some websites stall out from the windows systems in the past, so I was probably seeing the same sort of MSS-too-large for the return path issue needing the server to respond to ICMPv6 too big.

Client : Hello, MSS=1440.  Send me stuff.

Server: Great. Have a packet with the first 1440 of data

Client (or an intermediate router on the return path):  Sorry, the client lied.  Only 1400 is allowed.  Here's the IPCMv6 too big telling you that.

then

Good Servers: Whatever.  Here's the first 1400 bytes then.

Bad Servers: You exaggerated?  Have 1440 again.  Naughty client!  Still can't take it?  I don't care!

Read more

[#13]

Had a long time trying to fix this with our HE Tunnel, running through a Ubuntu laptop.
I setup mss clamping using a ufw rule, didn’t seem to work on initial connection - ok after refresh.
Tried changing mtu everywhere, no change with initial connect, except when changing mtu on device connection (using mtu of the tunnel). All other mtu settings seem to be ignored initially, on Linux at least, and uses default interface mtu. macOS may behave the same way, can only check Catalina as the latest available option for me.