MQTT connection for IoT device not working

[#1]

We are still trying to get in contact with our customer to check this additional setting and get more info. I will be traveling for a few days, so my interaction will be sparse, but thanks again for helping us with this. Hopefully we can crack this nut.

[#2]

Hi,

I'm new to both Plusnet and monitor-io and I have exactly the problem being described in this thread. Happy to help in any way I can.

Chris

[#3]

Bob,

I wanted to explicitly reply given your generous offer of "The next time one of your customers reports problems..." -- as they have already posted to this thread (as chrisking) about the same issue with a Hub One. The only additional info to what was posted was the WAN IP of [Removed] as of a few hours ago. Once again I thank you in advance for your help and assistance.

Moderator's note by Dick (Strat): IP removed as requested. 

[#4]

I'd prefer it if you didn't publish my IP address on a public forum!

Message me privately if you want more info.

[#5]

@chrisking 

I see you've posted the router model and firmware info. Understandable that you don't wish your IP public!

Whilst we're waiting for him to spot the update to this thread, could you confirm the other information that Bob requested earlier in the thread

The next time one of your customers reports problems, it would be useful to collect the following info from them (or direct them to this thread)

• Confirmation that the server side firewall here is switched off.

[#6]

Yes, the server-side firewall is off. Safeguard is off.

[#7]

Ok, I have an update.

1) I can use MQTT over TLS from my MacBook, connecting either to the Plusnet WiFi, or via a Thunderbolt ethernet adapter.

2) Connect the monitor-io box to a MacBook Pro Thunderbolt ethernet adapter, and share the macbook's wifi connection with that adapter, then it works.

So.. it's something to do with the way the hub is handling traffic from the ethernet ports.

If I can find an ethernet hub, I'll a look at the traffic (I only have a switch at the moment)

[#8]

@chrisking that sounds very strange!.

If I can find an ethernet hub, I'll a look at the traffic (I only have a switch at the moment)

That would be useful.

[#9]

Connecting it through the MacBook as a test was an excellent idea.

Could we be dealing with an MTU issue either through the Plusnet Hub One router (i.e., via its configuration) or some portion of the Plusnet network. For example, if some ICMP messages are being blocked somewhere and the Packet Too Big messages are not making it back Path MTU Discovery will break. And, there are more than a few occurrences online of protocols like TLS having an issue when MTU is not being handled as expected.

Also, I found this on the Plusnet forum which was interesting (and there may be other examples): https://community.plus.net/t5/Fibre-Broadband/MTU-Setting/m-p/1331399#M37878

The fact that a couple other customers were able to get their I/O Monitor working when they used a different router may also support this theory.

[#10]

Interesting thought, but why did @bobpullen s tests all work

[#11]

Well, it could be because either there is a difference between their router settings (not sure how likely that is, maybe a low probability) OR it has to do with the access router they're serviced by, the path through Plusnet, or the peering router they're using (i.e., something impacting PMTU Discovery). And if the customer changes their router, the local MTU handling is different and the device starts working (as was mentioned in that other forum thread).

Although a theory, it's the only one so far that seems to allow for all the quirks we've seen.

[#12]

So, I think it's safe to say we're not dealing solely with an issue with the CPE/router, else none of my tests would have worked. The Hub One does have a questionable MTU configuration of 1488, however the same does not apply to any of the other routers that were reported as failing. I can send a modified Hub One with altered config, but the evidence doesn't suggest that would be worthwhile just yet.

The devices I used to test were running default configs with no alterations.

I have an unmanaged switch I could grab some captures with, however it's unlikely to show anything of merit because I'm unable to replicate the problem.

@chrisking, I've made some routing changes to your connection (the line will have dropped briefly, so apologies if you were in the middle of anything). Be useful to know if the situation has changed?

[#13]

@chrisking was able to do a packet capture of when the device is unable to connect (I wish all our customers were this resourceful). Anyway, after looking at the capture it appears the MTU size is not the issue. It's looking like a problem related to the DNS resolution of the MQTT destination. Therefore, the new theory is that the difference between your working unit and the ones that don't work is the DNS they are using. It appears that the customer is unable to change the DNS configuration on the Hub One...but are you able to home it to another primary/secondary DNS - maybe the same one's your router is using as a test?

Any thoughts or ideas?

[#14]

Right then. @monitor-io set the DNS on my box to 8.8.8.8, and it went green immediately.

So it's a DNS issue.

Chris

[#15]

*WARNING: Not for the technically faint of heart*

OK, so the routing change hasn't worked.

Thanks to @chrisking though, we now have some captures from the working setup using the MAC as the access point, and the non-working setup whilst connected directly to the Hub.

The results are interesting.

Looking at the working capture, we can see a DNS request gets sent from the monitor-io box (192.168.2.3 in this instance, highlighted packet), to the Mac (192.168.2.1) for an amazon AWS address associated with the monitor-io service. The subsequent chatter is the rest of this DNS transaction as one would expect:

Now look what happens in the non-working capture...

We can see the same DNS requests sent from the monitor-io box (192.168.1.77 this time), to the Hub One (192.168.1.254), however there's something interesting with the behaviour of the monitor-io and the responses from the Hub's DNS proxy:

Firstly the payload of the IPv4 DNS response is a lot bigger compared to the MAC. In fact, it's almost at the 512 'limit' at which DNS starts favouring TCP over UDP (see here if you're so inclined). Not my area of expertise, but I'm guessing with some additional headers considered, the monitor-io box probably is seeing a packet > 512 bytes.

This seems to be leading to the subsequent DNS request from the monitor-io to attempt the transaction using TCP rather than UDP.

What happens next is interesting. The hub replies with a TCP reset packet and potentially isn't forwarding the DNS request upstream (or if it is, it's doing something odd).

I think this is then causing the monitor-io to try again, and what we then see is the Hub One attempting to query the same AWS address, but this time suffixed with .lan - Again, I'm no DNS proxying expert, but my guess here is that it's doing this because it's already determined the domain can't be resolved, and therefore starts looking locally for other devices connected to the router that might have this name.

Unsurprisingly, what happens next, is a response is received advising that the domain (suffixed with .lan) cannot be found.

So in summary, there's a few questions here.

1. Why is the payload of the Hub's DNS response greater than that returned by the Mac?
2. Why does the router send a TCP reset in response to a valid (?) TCP DNS packet?
3. Why couldn't I replicate this problem when I tried?

For next steps, few things I'd suggest:

1. Can the monitor-io config be amended to use hardcoded DNS resolvers e.g. Google's rather than the router proxy that's assigned by DHCP? If we're looking in the right place, then I'd expect things to work using this set-up.

2. I should try grabbing a capture from my test lab, and see what it shows.

3. We should also look to grab a capture from the WAN interface of the router when this is happening. I can do that remotely with @chrisking's assistance, I'd first need the serial number of the Hub One in use though.

Apologies for the wall of technical gubbins, but I'd probably forget all of the above tomorrow if I didn't write it down somewhere

Edit: I see we've now seen to item 1 above