MQTT connection for IoT device not working

[#1]

Hello, our company sells an IoT product that is unable to connect to our AWS-based IoT service via some Plusnet routers (Hub one, 2704n, 582n) - and it may be specific to certain firmware versions. No other ISP in the UK (or 26 other countries) is having this issue. If the customer swaps out their Plusnet router for their own router (of any brand) the problem completely goes away. Our connectivity is very simple, just a TCP connection to port 8883 in the AWS cloud (specifically MQTT over TLS).

We worked with one of your customers that was willing to try virtually all router configuration options possible to get it working (including CLI commands to try and fully disable the firewall functionality). In the end they had to buy a different router, and everything then worked as expected. A different customer claimed that it was working until a firmware update.

We would greatly appreciate any assistance or insight you can provide on this.

Thank you.

[#2]

@monitor-io Hi, I'd like to try and get a liitle more information about the problem if I may, so that I can push it in the direction of the relevant PlusNet  staff if necessary.

I've taken a look at what I assume is your website and the technical description of your test methodolgy here http://www.monitor-io.com/test-methodology.html . TBH it all seems pretty straightforward.

So a bit more information if possible:-

1) Do you know which of the PN routers exhibits the problem ? If you're not sure, are all the connections FTTC ( in which case that limits the routers being used to the Hub One )

2) According to the tech doc you use DHCP to get an IP address. Are you able to confirm if the device is actually getting an IP address ?

[#3]

Well, thank you in advance for your time. And yes, our test methodology and connectivity is designed to be very straightforward. As to your questions...

1) We have seen at least one issue with each of the 3 routers we listed, an FTTC via the Hub One and a couple with the other 2 models.

2) We do use DHCP and we have confirmed that we are getting an address for the local LAN because the customer is able to access our device locally via a browser (our unit displays its local address periodically). In addition, we’ve also confirmed that we are getting names resolved via the DNS address supplied with DHCP and the unit is able to reach public NTP servers (x.debian.pool.ntp.org via standard UDP port 123) and synchronize. A reasonably accurate clock is of course important for TLS.

We also know that our unit can successfully reach the home page of various well-known web sites (e.g., Google, YouTube, Netflix, etc.) because when the user initiates one of our local connectivity tests it passes just fine. This corresponds with their own experience of being able to surf the web with no problem from their local computers.

Given all this, we feel our issue has been narrowed down to the outgoing TCP connection to port 8883 in AWS (US east region) for our needed MQTT over TLS connectivity. For some reason, the PN routers (and maybe only those with certain firmware versions) have a problem allowing this connection to get established.

In addition, we have verified that from a laptop on the local LAN a user could resolve the specific DNS name we use for MQTT and was able to ping it – so basic reachability seemed ok. The problem looked like it was with how the router was treating the specific TCP connection to 8883 carrying TLS. Also, a customer provided us the output from “connection applist” and “connection bindlist” command on a 582n and we verified that we did not see any obvious port entry or range that would impact 8883. BTW, a full factory reset of the router was also done just to be sure.

[#4]

Also, a customer provided us the output from “connection applist” and “connection bindlist” command on a 582n and we verified that we did not see any obvious port entry or range that would impact 8883

I was going to suggest that if the TG582n was one of the routers showing the problem, but you beat me to it. TBH I'm surprised that the TG582 gives a problem as it's a fairly generic router used by many ISP's (maybe not currently) and AFAIK the PN firmware is little different to the generic one.

In addition, we have verified that from a laptop on the local LAN a user could resolve the specific DNS name we use for MQTT and was able to ping it – so basic reachability seemed ok.

Is there any way to test the MQTT TLS connection from a laptop ?

[#5]

We do think that the PN firmware difference may be the key. A direct quote from one customer was "I’ve had to change router to basic isp one (Plusnet). Monitor-io working fine on old router until router firmware update issue. Now it displays no internet & red display".

Unfortunately, we don't have a way of testing our MQTT over TLS connection from a laptop - we'd have to create a new client app (which is technically possible but hasn't been required for anything else). However, we would be happy to ship a unit to you (or whoever) if there was a willingness to assist us more directly - especially since you guys would have easy access to the various router models and firmware versions. Please let us know.

[#6]

There's little consistency across the three routers you've mentioned, plus there's two different manufacturers at play.

Neither do I see any reason why 8883 would be blocked. It's not reserved for anything on any of the models you've mentioned AFAIK.

Extremely curious that trying a non-Plusnet router fixes things. Exactly how many of these examples have you logged?

Few things...

1. On a Windows machine, your customers could download tracetcp and run a TCP trace to your AWS instance on the affected port. Happy to provide more detail if needed.

2. Anyone with a *nix machine can try initiating an SSL connection from command line using openssl. In fact, there will be ways to do this on Windows too by downloading an executable. Again, can provide instructions, it's just tricky at the mo as on mobile.

3. Feel free to ship a unit to me and I'll do my best to help when I've some free time:
FAO: Bob Pullen
Plusnet
The Balance
2 Pinfold Street
Sheffield
S1 2GU

[#7]

We are definitely in agreement that it doesn't quite make sense and is unexpected. And in fairness, we've only encountered about 5 of these. But after several with the same symptoms and solution, and having done a fair bit of troubleshooting (or what we could talk the customers through), we felt it was time to reach out to you.

And your suggestions are well taken - we would look to incorporate that type functionality into a test client (for our non-tech users) if it comes to that.

But we will take you up on your offer to receive one - we'll ship it out in the next 24 hours. It will be all setup so you'll just need to plug it in. If it goes red and says INTERNET OFFLINE that will be the issue. And even if it goes green and says INTERNET ONLINE we'll know what models and firmware it definitely works with. In either case, we really appreciate it - Thank you...

[#8]

2. Anyone with a *nix machine can try initiating an SSL connection from command line using openssl. In fact, there will be ways to do this on Windows too by downloading an executable. Again, can provide instructions, it's just tricky at the mo as on mobile.

I have a 'nix' machine at home if there's anything I can do to help. At present I don't use the PN supplied router, I have a Draytek 2830 with a VDSL modem but I'm pretty sure I've got a PN TG582n lying about somewhere I could resurrect for testing.

[#9]

You'd need to know the destination IP, but these would help prove connectivity:

openssl s_client -connect <aws_ip>:8883

tcptraceroute <aws_ip> 8883

[#10]

I'm intrigued to find out what this issue turns out to be?

Neat looking device - but a tad pricy for what it is?

[#11]

We just wanted to let you know that tracking says the unit was delivered - hopefully you'll see it in the near future.

And thanks again for your help...

[#12]

Wanted to mention that if you'd like to look at the network information and test data for that unit (although it's not really needed to work this issue), we created an account for you...

https://my.monitor-io.com

Username: www.plus.net

Password: <Unit ID (XX-XXXX-XXXX)>

(The ID is periodically shown on the LCD as well as the menu bar of the local web interface...available via the local IP also shown on the LCD)

[#13]

Thanks, unit received.

I've been unable to replicate the issue with a Plusnet Hub One though, regardless of firmware version. In all circumstances the monitor reports an active Internet connection and the LED lights up Green:

4.7.5.1.83.8.217.1.1 - Working from 46.208.79.226/FTTC
4.7.5.1.83.8.218 - Working from 80.189.10.163/FTTC
4.7.5.1.83.8.226 - Working from 146.200.0.132/FTTC
4.7.5.1.83.8.237.2.2 - Working from 146.199.145.25/FTTC
4.7.5.1.83.8.259 - Working from 146.199.30.227/FTTC
4.7.5.1.83.8.259.1.1 - Working from 80.189.10.163/FTTC

I've also tried the following firmware versions of the Hub Zero and Technicolor 582n, and they work OK too:

10.2.5.2.FO - Working from 146.199.145.11/FTTC
7.275.12_F2704N_Plusnet - Working from 84.92.178.196/ADSL2+

The next time one of your customers reports problems, it would be useful to collect the following info from them (or direct them to this thread):

• WAN IP address assigned to the router.
• Confirmation that the server side firewall here is switched off.
• Make, model and firmware version of the router in use.
• When the problem began, if applicable.

As an aside (and perhaps intentional?), I don't seem to be able to register the device online. It just returns an Invalid ID error:

Edit: Ignore the bit above about the invalid ID error. I'd overlooked your last post. I'm now logged in successfully

[#14]

That's very good info.

I will check with an existing customer that is still having an issue and see if he is ok with us getting some more details. When we worked with him last he had said that "SafeGuard is / was not enabled. The firewall is / was already on default settings." - but maybe not the server side firewall, so I will have him check again.

[#15]

SafeGuard is / was not enabled. The firewall is / was already on default settings." - but maybe not the server side firewall, so I will have him check again.

Just a note that If the Firewall settings are changed, they won't take effect until the PPPoE connection is dropped and reconnected. Usually that means having to reboot the router...