Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

[#1]

I've recently had broadband disconnection issues due to my Hub 1 router (specifically the Lantiq chipset) not handling the error correction used by new hardware at the exchange. After months and multiple engineers, plusnet eventually sent me a Hub 2 which appears to fix the disconnections, however it is manifesting another issue.

At times it will become unresponsive for minutes at a time. By that I mean all packets appear to be silently dropped (pings, web interface, DNS, incoming traffic to forwarded ports etc). After a few minutes it will eventually begin responding again and the web interface will report that the connection has been up the whole time.

I have eventually correlated it to when my open Element (the Matrix client), which can trigger my Matrix homeserver (running on my LAN and federated with a bunch of other Matrix homeservers thanks to big rooms like #synapse:matrix.org) to send a flurry of federation requests, in the form of HTTPS requests to .well-known/matrix/server on a bunch of servers (in the order of a thousand or so per minute for only a minute or so). I'm now using a local DNS relay that forwards straight through to plusnet's DNS servers rather than the one on the router, so I know it isn't the DNS server on the router that is triggering it.

It looks like a firmware issue to me. It should surely be able to handle 16 short HTTPS requests (including DNS) per second for a few minutes without becoming unresponsive, but when I phoned plusnet today about it after a week of no response, they recognized that as a home user they weren't likely to give the issue any real priority to investigate or fix, and suggested my idea of using a 3rd party router was probably my best way forward.

So, has anybody else observed these issues?

can anybody suggest a way to escalate the issue further with plusnet to get a proper firmware fix?

can anybody recommend a well supported 3rd party router that works with plusnet, doesn't use Lantiq's broken chipsets/firmware and supports all the usual port forwarding features?

[#2]

I have zero experience with either Matrix Homeserver or OpenElement, and neither have I come across a simlar complaint before.

How easy would your setup/this problem be to replicate, as that's likely to be the quickest route to resolution? Assuming of course the firmware is the cause of the problem.

The router should handle the traffic - and hardware wise - it is undoubtedly more capable compared to the Hub One you were using previously (that you seemingly had no problem with in this regard?)

Are there other clients you could try to determine whether or not the behaviour of the client application is contributing to the issue?

[#3]

I'm going to suggest that the Hub2 places a much higher priority on WiFi traffic to the detriment of wired traffic.

[#4]

@greygit not my experience. I've multiple servers hosted on a wired connection, often using a Hub Two, and also with a high bandwidth Internet link. Nothing my wireless devices ever do is to the detriment of what I've got cabled.

[#5]

Yeh, makes sense that making it easy to reproduce is likely the key. I did attempt to script a bunch of parallel accesses to the same list of domains, but without success. Unfortunately setting up a federating synapse homeserver isn't completely trivial and requires a domain and forwarding etc, though once set up its easy enough to tell it to join one of the huge rooms.

It is also triggered by the mobile client (Element on Android) over wifi, however my desktop is connected by ethernet, as is the homeserver which is where the traffic comes from. Wifi devices are also affected by the unresponsiveness, and I have already tried repurposing my old hub1 as a wifi AP and turning off wifi on the hub2 and it had the same issue.

I'll have to keep trying.

[#6]

@amalon - I'm not adverse to a bit of effort to replicate things and have plenty of domains I can use. As long as it doesn't hurt my pocket too significantly then I'm good 😉
Feel free to share further detail or take the discussion offline by private messaging me.

[#7]

That'd be great if you can spare the effort. The server software is called synapse. There's links to documentation here:

https://matrix.org/docs/guides/installing-synapse

I think they only officially support it on Linux rather than Windows. Do you have a Linux distro installed?

[#8]

So after a fair bit of experimentation, it *appears* that the problem is related to DNS, and seems to go away if I configure my local DNS resolver (Unbound) to only use TCP when forwarding DNS requests. That suggests something about the DNS UDP packets isn't handled right by the router. I did also try turning off DNSSEC verification (with UDP on), but it still failed.

Given the nature of DNS TTL I'll leave it set like that for a while and see if it continues to work, just to be sure.

Hopefully the bug isn't exploitable!

[#9]

Some don't have the luxury (or availability) of a high bandwidth internet link. 720Kbps doesn't go very far when a device is uploading MBs (rather than downloading) on ADSL.

(FTTC doesn't give me much more.)

Hence my enquiry about options/possibilities on local traffic manglement/QoS on the LAN-side capabilities of PN Hub 2.

Has the Hub2 ever been properly tested on a flaky (relatively low bandwidth) ADSL line?

[#10]

---

@amalon wrote:

That'd be great if you can spare the effort. The server software is called synapse. There's links to documentation here:

https://matrix.org/docs/guides/installing-synapse

I think they only officially support it on Linux rather than Windows. Do you have a Linux distro installed?

---

I've a Linix machine on the network I can use. Will have a play around if I get some time.

Interesting observation on the DNS front. I wonder if it might have something to do with the payload. DNS using UDP has a maximum size of 512 bytes and if a response exceeds this, the requesting resolver will often retry using TCP. If you explicitly tell the resolver to use TCP, then in theory you're bypassing this failback mechanism.

How is Unbound configured? To carry out lookups directly, or is it forwarding to an upstream DNS server/the hub?

@greygit wrote:

Has the Hub2 ever been properly tested on a flaky (relatively low bandwidth) ADSL line?

Yes, it has been tested.

[#11]

My Unbound configuration is minimal compared to the default.

To force TCP on all forwarded requests I do this:

tcp-upstream: yes

To forward requests to plusnet I have this:

forward-zone:

    name: "."

    forward-addr: 212.159.6.10

    forward-addr: 212.159.6.9

And this seems to enable DNSSEC verification (this is the default on archlinux):

trust-anchor-file: "/etc/unbound/trusted-key.key"

(The only other changes are irrelevant to this discussion: interface, outgoing-interface, cache-min-ttl, access-control, logging, private-address)

My understanding and intention is that it forwards requests it hasn't cached to plusnet (i've also tried google's ones, and I think I was trying to avoid using the router one since I had issues with the DNS on the hub1 stopping working, and was trying to eliminate causes with hub2), and it should do DNSSEC verification using the signatures etc it gets via the upstream requests and the trust anchor keys. It doesn't have "forward-first" so it shouldn't do the resolution itself. (If I forward to my other unbound installation I can see from the logs that it is correctly caching the requests, and when I set cache-min-ttl it caches them for longer).

[#12]

So, was the problem evident irrespective of destination resolver? i.e. did it happen whether you were using Plusnet's or Google's upstream servers?

'For science', it would also be useful to know if the same occurs when forwarding to the hub.

I have Unbound on my own network but it is configured to query the root servers directly.

I managed to install Synapse/Element etc. earlier and pointed a couple of subdomains to it. I just need to set some time aside to familiarise myself with its workings.

[#13]

Yes, it also definitely happens when set to Google's 8.8.4.4 / 8.8.8.8.

It also definitely happens when no forward is configured (according to tshark it appears to do the full resolution itself to a bunch of different addresses).

However when set to the router IP (192.168.1.254), it did get some packet loss but for <1min, but then its been hard to reproduce... (i haven't seen it happen since even after a synchronised router power cycle to clear the DNS cache, restart of unbound and synapse).

Its definitely easier to reproduce for DNS queries going through the router to the internet (even with cache-max-ttl set low).

[#14]

I always have the option of altering my desktop's IP config to go direct to PN's DNS servers rather than letting the Hub2 act as a local server for the LAN. Done it before. However there is a slight bug in Win10 that ends up not displaying the IP address correctly in one part of Win10 and I've had to resort to using the 'older' style interface for verifying the settings.

I think somewhere in the DHCP protocol there should be an option/setting for the DHCP server to provide a client requesting settings with the address(es) of the DNS server(s) address it has picked up/learned/been given (rather than acting as a DNS server itself). That's my understanding of what DHCP can/should do.

[#15]

So, I think I've replicated this issue.

Installed the Element client on an *nix machine and an Android smartphone earlier and joined a couple of large rooms (#synapse:matrix.org and #raspberrypi:matrix.org). Saw my network become unresponsive, pings timing out, difficulty accessing the hub GUI and hosted servers losing inbound connectivity from the Internet. Happened a few times with the client open.

I'll keep an eye on it and get something raised in due course.

As an aside, Synapse strikes me as an extremely demanding piece of software! I've installed it on a machine with a 5th gen i5 and 8GB of RAM and it's been pegging my CPU/gobbling memory for the best part of an hour! I suppose it could be because I left the database settings at default (which the docs don't advise), but still.

Edit: @greygit assigning DNS using the router's DHCP server (whilst possible) is not going to help here. Unless I'm misinterpreting things, I think we've determined as much.