topic Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver in My Router

Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

amalon — Tue, 24 May 2022 09:36:06 GMT

I've recently had broadband disconnection issues due to my Hub 1 router (specifically the Lantiq chipset) not handling the error correction used by new hardware at the exchange. After months and multiple engineers, plusnet eventually sent me a Hub 2 which appears to fix the disconnections, however it is manifesting another issue.

At times it will become unresponsive for minutes at a time. By that I mean all packets appear to be silently dropped (pings, web interface, DNS, incoming traffic to forwarded ports etc). After a few minutes it will eventually begin responding again and the web interface will report that the connection has been up the whole time.

I have eventually correlated it to when my open Element (the Matrix client), which can trigger my Matrix homeserver (running on my LAN and federated with a bunch of other Matrix homeservers thanks to big rooms like #synapse:matrix.org) to send a flurry of federation requests, in the form of HTTPS requests to .well-known/matrix/server on a bunch of servers (in the order of a thousand or so per minute for only a minute or so). I'm now using a local DNS relay that forwards straight through to plusnet's DNS servers rather than the one on the router, so I know it isn't the DNS server on the router that is triggering it.

It looks like a firmware issue to me. It should surely be able to handle 16 short HTTPS requests (including DNS) per second for a few minutes without becoming unresponsive, but when I phoned plusnet today about it after a week of no response, they recognized that as a home user they weren't likely to give the issue any real priority to investigate or fix, and suggested my idea of using a 3rd party router was probably my best way forward.

So, has anybody else observed these issues?

can anybody suggest a way to escalate the issue further with plusnet to get a proper firmware fix?

can anybody recommend a well supported 3rd party router that works with plusnet, doesn't use Lantiq's broken chipsets/firmware and supports all the usual port forwarding features?

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

bobpullen — Tue, 24 May 2022 10:35:54 GMT

I have zero experience with either Matrix Homeserver or OpenElement, and neither have I come across a simlar complaint before.

How easy would your setup/this problem be to replicate, as that's likely to be the quickest route to resolution? Assuming of course the firmware is the cause of the problem.

The router should handle the traffic - and hardware wise - it is undoubtedly more capable compared to the Hub One you were using previously (that you seemingly had no problem with in this regard?)

Are there other clients you could try to determine whether or not the behaviour of the client application is contributing to the issue?

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

greygit — Wed, 25 May 2022 00:09:08 GMT

I'm going to suggest that the Hub2 places a much higher priority on WiFi traffic to the detriment of wired traffic.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

bobpullen — Wed, 25 May 2022 01:04:47 GMT

@greygit not my experience. I've multiple servers hosted on a wired connection, often using a Hub Two, and also with a high bandwidth Internet link. Nothing my wireless devices ever do is to the detriment of what I've got cabled.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

amalon — Wed, 25 May 2022 07:30:47 GMT

Yeh, makes sense that making it easy to reproduce is likely the key. I did attempt to script a bunch of parallel accesses to the same list of domains, but without success. Unfortunately setting up a federating synapse homeserver isn't completely trivial and requires a domain and forwarding etc, though once set up its easy enough to tell it to join one of the huge rooms.

It is also triggered by the mobile client (Element on Android) over wifi, however my desktop is connected by ethernet, as is the homeserver which is where the traffic comes from. Wifi devices are also affected by the unresponsiveness, and I have already tried repurposing my old hub1 as a wifi AP and turning off wifi on the hub2 and it had the same issue.

I'll have to keep trying.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

bobpullen — Wed, 25 May 2022 08:40:48 GMT

@amalon - I'm not adverse to a bit of effort to replicate things and have plenty of domains I can use. As long as it doesn't hurt my pocket too significantly then I'm good 😉
Feel free to share further detail or take the discussion offline by private messaging me.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

amalon — Wed, 25 May 2022 16:43:55 GMT

That'd be great if you can spare the effort. The server software is called synapse. There's links to documentation here:

https://matrix.org/docs/guides/installing-synapse

I think they only officially support it on Linux rather than Windows. Do you have a Linux distro installed?

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

amalon — Wed, 25 May 2022 22:57:00 GMT

So after a fair bit of experimentation, it *appears* that the problem is related to DNS, and seems to go away if I configure my local DNS resolver (Unbound) to only use TCP when forwarding DNS requests. That suggests something about the DNS UDP packets isn't handled right by the router. I did also try turning off DNSSEC verification (with UDP on), but it still failed.

Given the nature of DNS TTL I'll leave it set like that for a while and see if it continues to work, just to be sure.

Hopefully the bug isn't exploitable!

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

greygit — Wed, 25 May 2022 23:51:37 GMT

Some don't have the luxury (or availability) of a high bandwidth internet link. 720Kbps doesn't go very far when a device is uploading MBs (rather than downloading) on ADSL.

(FTTC doesn't give me much more.)

Hence my enquiry about options/possibilities on local traffic manglement/QoS on the LAN-side capabilities of PN Hub 2.

Has the Hub2 ever been properly tested on a flaky (relatively low bandwidth) ADSL line?

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

bobpullen — Thu, 26 May 2022 13:35:28 GMT

@amalon wrote:

That'd be great if you can spare the effort. The server software is called synapse. There's links to documentation here:

https://matrix.org/docs/guides/installing-synapse

I think they only officially support it on Linux rather than Windows. Do you have a Linux distro installed?

I've a Linix machine on the network I can use. Will have a play around if I get some time.

Interesting observation on the DNS front. I wonder if it might have something to do with the payload. DNS using UDP has a maximum size of 512 bytes and if a response exceeds this, the requesting resolver will often retry using TCP. If you explicitly tell the resolver to use TCP, then in theory you're bypassing this failback mechanism.

How is Unbound configured? To carry out lookups directly, or is it forwarding to an upstream DNS server/the hub?

@greygit wrote:

Has the Hub2 ever been properly tested on a flaky (relatively low bandwidth) ADSL line?

Yes, it has been tested.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

amalon — Thu, 26 May 2022 17:05:43 GMT

My Unbound configuration is minimal compared to the default.

To force TCP on all forwarded requests I do this:

tcp-upstream: yes

To forward requests to plusnet I have this:

forward-zone:

name: "."

forward-addr: 212.159.6.10

forward-addr: 212.159.6.9

And this seems to enable DNSSEC verification (this is the default on archlinux):

trust-anchor-file: "/etc/unbound/trusted-key.key"

(The only other changes are irrelevant to this discussion: interface, outgoing-interface, cache-min-ttl, access-control, logging, private-address)

My understanding and intention is that it forwards requests it hasn't cached to plusnet (i've also tried google's ones, and I think I was trying to avoid using the router one since I had issues with the DNS on the hub1 stopping working, and was trying to eliminate causes with hub2), and it should do DNSSEC verification using the signatures etc it gets via the upstream requests and the trust anchor keys. It doesn't have "forward-first" so it shouldn't do the resolution itself. (If I forward to my other unbound installation I can see from the logs that it is correctly caching the requests, and when I set cache-min-ttl it caches them for longer).

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

bobpullen — Thu, 26 May 2022 17:58:52 GMT

So, was the problem evident irrespective of destination resolver? i.e. did it happen whether you were using Plusnet's or Google's upstream servers?

'For science', it would also be useful to know if the same occurs when forwarding to the hub.

I have Unbound on my own network but it is configured to query the root servers directly.

I managed to install Synapse/Element etc. earlier and pointed a couple of subdomains to it. I just need to set some time aside to familiarise myself with its workings.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

amalon — Thu, 26 May 2022 21:43:42 GMT

Yes, it also definitely happens when set to Google's 8.8.4.4 / 8.8.8.8.

It also definitely happens when no forward is configured (according to tshark it appears to do the full resolution itself to a bunch of different addresses).

However when set to the router IP (192.168.1.254), it did get some packet loss but for <1min, but then its been hard to reproduce... (i haven't seen it happen since even after a synchronised router power cycle to clear the DNS cache, restart of unbound and synapse).

Its definitely easier to reproduce for DNS queries going through the router to the internet (even with cache-max-ttl set low).

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

greygit — Thu, 26 May 2022 22:11:39 GMT

I always have the option of altering my desktop's IP config to go direct to PN's DNS servers rather than letting the Hub2 act as a local server for the LAN. Done it before. However there is a slight bug in Win10 that ends up not displaying the IP address correctly in one part of Win10 and I've had to resort to using the 'older' style interface for verifying the settings.

I think somewhere in the DHCP protocol there should be an option/setting for the DHCP server to provide a client requesting settings with the address(es) of the DNS server(s) address it has picked up/learned/been given (rather than acting as a DNS server itself). That's my understanding of what DHCP can/should do.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

bobpullen — Thu, 26 May 2022 23:05:41 GMT

So, I think I've replicated this issue.

Installed the Element client on an *nix machine and an Android smartphone earlier and joined a couple of large rooms (#synapse:matrix.org and #raspberrypi:matrix.org). Saw my network become unresponsive, pings timing out, difficulty accessing the hub GUI and hosted servers losing inbound connectivity from the Internet. Happened a few times with the client open.

I'll keep an eye on it and get something raised in due course.

As an aside, Synapse strikes me as an extremely demanding piece of software! I've installed it on a machine with a 5th gen i5 and 8GB of RAM and it's been pegging my CPU/gobbling memory for the best part of an hour! I suppose it could be because I left the database settings at default (which the docs don't advise), but still.

Edit: @greygit assigning DNS using the router's DHCP server (whilst possible) is not going to help here. Unless I'm misinterpreting things, I think we've determined as much.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

greygit — Thu, 26 May 2022 23:28:19 GMT

That's "whooshed" me.

The only commonality with what you've posted and what it here is an android smartphone.

I think my original comment on this subject became entangled/crossed.

Although I'm happy to sit and watch this. It is certainly better/more intriguing that much of what is on broadcast (and streaming)TV!

(P.S. I still think Hub2 has undiscovered problems, and there may yet be a connection).

P.P.S.

(tried to post this and got an error code 7509B6FE.)

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

amalon — Fri, 27 May 2022 07:45:35 GMT

Awesome, thanks bobpullen.

Yeh, mine uses a fair bit of CPU too, running in a smallish VM (they're making a second gen server called dendrite which is claimed to be less resource intensive than synapse). I should look into switching to postgresql as I never did get around to it.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

bobpullen — Fri, 27 May 2022 09:20:11 GMT

@amalon - checking again this morning, I've a little more work to do it seems. I have my Unbound DNS requests running through DNSMasq/Pihole and it seems simply opening the element client causes Pihole to start rejecting DNS requests due to the default rate limits (1000 DNS requests per minute per client).

That could be clouding my symptoms, so I'll up the limit later and carry out some more testing.

I've only joined two Matrix groups and it generates thousands of DNS requests in an extremely short period of time! I understand why it's doing this - but it's somewhat questionable when you consider the fact that an extremely popular ad-blocking solution has out of the box config that cripples it. The maintainers of Pihole clearly don't consider it 'typical' for a single client to be generating this much DNS traffic

That said, I guess it doesn't detract from the fact that it 'works' with other hubs, and not the Hub Two.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

amalon — Fri, 27 May 2022 15:07:26 GMT

Yeh. There are rate limiting options in synapse, such as the read receipt rate limit which can slow it down a bit when you just look at a room, but any sending of events still happens at top speed so it doesn't really work around it.

I did wonder if it was just a case of having a randomised TTL for each federated domain, but then if you leave it and come back to it, it wants to do them all at once anyway. Unbound's cache-min-ttl helps a bit for domains with unreasonably short TTLs.

Re: Plusnet Hub 2 becomes unresponsive with Matrix Homeserver

bobpullen — Fri, 27 May 2022 15:12:29 GMT

Do we know yet if it happens without Unbound in the equation? i.e. with the Hub dishing out 192.168.1.254 or a public DNS server addresses directly to the connected clients over DHCP?

If not, it's another scenario that will need testing.