MQTT connection for IoT device not working
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Plusnet Community
- :
- Forum
- :
- Help with my Plusnet services
- :
- My Router
- :
- Re: MQTT connection for IoT device not working
Re: MQTT connection for IoT device not working
26-07-2018 5:51 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
We are still trying to get in contact with our customer to check this additional setting and get more info. I will be traveling for a few days, so my interaction will be sparse, but thanks again for helping us with this. Hopefully we can crack this nut.
Re: MQTT connection for IoT device not working
04-09-2018 10:23 AM - edited 04-09-2018 10:24 AM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
Hi,
I'm new to both Plusnet and monitor-io and I have exactly the problem being described in this thread. Happy to help in any way I can.
Chris
Re: MQTT connection for IoT device not working
on 04-09-2018 5:26 PM - last edited on 04-09-2018 5:47 PM by Strat
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
Bob,
I wanted to explicitly reply given your generous offer of "The next time one of your customers reports problems..." -- as they have already posted to this thread (as chrisking) about the same issue with a Hub One. The only additional info to what was posted was the WAN IP of [Removed] as of a few hours ago. Once again I thank you in advance for your help and assistance.
Moderator's note by Dick (Strat): IP removed as requested.
Re: MQTT connection for IoT device not working
04-09-2018 5:28 PM - edited 04-09-2018 5:49 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
I'd prefer it if you didn't publish my IP address on a public forum!
Message me privately if you want more info.
Re: MQTT connection for IoT device not working
04-09-2018 6:01 PM - edited 04-09-2018 6:03 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
I see you've posted the router model and firmware info. Understandable that you don't wish your IP public!
Whilst we're waiting for him to spot the update to this thread, could you confirm the other information that Bob requested earlier in the thread
The next time one of your customers reports problems, it would be useful to collect the following info from them (or direct them to this thread)
- Confirmation that the server side firewall here is switched off.
Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.
Re: MQTT connection for IoT device not working
04-09-2018 7:03 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
Yes, the server-side firewall is off. Safeguard is off.
Re: MQTT connection for IoT device not working
11-09-2018 10:43 AM - edited 11-09-2018 10:47 AM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
Ok, I have an update.
1) I can use MQTT over TLS from my MacBook, connecting either to the Plusnet WiFi, or via a Thunderbolt ethernet adapter.
2) Connect the monitor-io box to a MacBook Pro Thunderbolt ethernet adapter, and share the macbook's wifi connection with that adapter, then it works.
So.. it's something to do with the way the hub is handling traffic from the ethernet ports.
If I can find an ethernet hub, I'll a look at the traffic (I only have a switch at the moment)
Re: MQTT connection for IoT device not working
11-09-2018 11:37 AM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
@chrisking that sounds very strange!.
If I can find an ethernet hub, I'll a look at the traffic (I only have a switch at the moment)
That would be useful.
Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.
Re: MQTT connection for IoT device not working
11-09-2018 11:38 AM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
Connecting it through the MacBook as a test was an excellent idea.
Re: MQTT connection for IoT device not working
11-09-2018 11:51 AM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
Interesting thought, but why did @bobpullen s tests all work
Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.
Re: MQTT connection for IoT device not working
11-09-2018 12:34 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
Well, it could be because either there is a difference between their router settings (not sure how likely that is, maybe a low probability) OR it has to do with the access router they're serviced by, the path through Plusnet, or the peering router they're using (i.e., something impacting PMTU Discovery). And if the customer changes their router, the local MTU handling is different and the device starts working (as was mentioned in that other forum thread).
Although a theory, it's the only one so far that seems to allow for all the quirks we've seen.
Re: MQTT connection for IoT device not working
12-09-2018 10:48 AM - edited 12-09-2018 10:50 AM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
So, I think it's safe to say we're not dealing solely with an issue with the CPE/router, else none of my tests would have worked. The Hub One does have a questionable MTU configuration of 1488, however the same does not apply to any of the other routers that were reported as failing. I can send a modified Hub One with altered config, but the evidence doesn't suggest that would be worthwhile just yet.
The devices I used to test were running default configs with no alterations.
I have an unmanaged switch I could grab some captures with, however it's unlikely to show anything of merit because I'm unable to replicate the problem.
@chrisking, I've made some routing changes to your connection (the line will have dropped briefly, so apologies if you were in the middle of anything). Be useful to know if the situation has changed?
Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵
Re: MQTT connection for IoT device not working
13-09-2018 6:34 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
@chrisking was able to do a packet capture of when the device is unable to connect (I wish all our customers were this resourceful). Anyway, after looking at the capture it appears the MTU size is not the issue. It's looking like a problem related to the DNS resolution of the MQTT destination. Therefore, the new theory is that the difference between your working unit and the ones that don't work is the DNS they are using. It appears that the customer is unable to change the DNS configuration on the Hub One...but are you able to home it to another primary/secondary DNS - maybe the same one's your router is using as a test?
Any thoughts or ideas?
Re: MQTT connection for IoT device not working
13-09-2018 6:54 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
Right then. @monitor-io set the DNS on my box to 8.8.8.8, and it went green immediately.
So it's a DNS issue.
Chris
Re: MQTT connection for IoT device not working
13-09-2018 7:31 PM - edited 13-09-2018 7:41 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
*WARNING: Not for the technically faint of heart*
OK, so the routing change hasn't worked.
Thanks to @chrisking though, we now have some captures from the working setup using the MAC as the access point, and the non-working setup whilst connected directly to the Hub.
The results are interesting.
Looking at the working capture, we can see a DNS request gets sent from the monitor-io box (192.168.2.3 in this instance, highlighted packet), to the Mac (192.168.2.1) for an amazon AWS address associated with the monitor-io service. The subsequent chatter is the rest of this DNS transaction as one would expect:
Now look what happens in the non-working capture...
We can see the same DNS requests sent from the monitor-io box (192.168.1.77 this time), to the Hub One (192.168.1.254), however there's something interesting with the behaviour of the monitor-io and the responses from the Hub's DNS proxy:
Firstly the payload of the IPv4 DNS response is a lot bigger compared to the MAC. In fact, it's almost at the 512 'limit' at which DNS starts favouring TCP over UDP (see here if you're so inclined). Not my area of expertise, but I'm guessing with some additional headers considered, the monitor-io box probably is seeing a packet > 512 bytes.
This seems to be leading to the subsequent DNS request from the monitor-io to attempt the transaction using TCP rather than UDP.
What happens next is interesting. The hub replies with a TCP reset packet and potentially isn't forwarding the DNS request upstream (or if it is, it's doing something odd).
I think this is then causing the monitor-io to try again, and what we then see is the Hub One attempting to query the same AWS address, but this time suffixed with .lan - Again, I'm no DNS proxying expert, but my guess here is that it's doing this because it's already determined the domain can't be resolved, and therefore starts looking locally for other devices connected to the router that might have this name.
Unsurprisingly, what happens next, is a response is received advising that the domain (suffixed with .lan) cannot be found.
So in summary, there's a few questions here.
1. Why is the payload of the Hub's DNS response greater than that returned by the Mac?
2. Why does the router send a TCP reset in response to a valid (?) TCP DNS packet?
3. Why couldn't I replicate this problem when I tried?
For next steps, few things I'd suggest:
1. Can the monitor-io config be amended to use hardcoded DNS resolvers e.g. Google's rather than the router proxy that's assigned by DHCP? If we're looking in the right place, then I'd expect things to work using this set-up.
2. I should try grabbing a capture from my test lab, and see what it shows.
3. We should also look to grab a capture from the WAN interface of the router when this is happening. I can do that remotely with @chrisking's assistance, I'd first need the serial number of the Hub One in use though.
Apologies for the wall of technical gubbins, but I'd probably forget all of the above tomorrow if I didn't write it down somewhere
Edit: I see we've now seen to item 1 above
Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Plusnet Community
- :
- Forum
- :
- Help with my Plusnet services
- :
- My Router
- :
- Re: MQTT connection for IoT device not working