cancel
Showing results for 
Search instead for 
Did you mean: 

MQTT connection for IoT device not working

bobpullen
Community Gaffer
Community Gaffer
Posts: 16,869
Thanks: 4,950
Fixes: 315
Registered: ‎04-04-2007

Re: MQTT connection for IoT device not working

@chrisking, as an afterthought, what DNS resolvers is your router using?

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

chrisking
Dabbler
Posts: 15
Thanks: 12
Registered: ‎04-09-2018

Re: MQTT connection for IoT device not working

@bobpullen

 

Primary DNS: 212.159.6.9  
Secondary DNS: 212.159.6.10
chrisking
Dabbler
Posts: 15
Thanks: 12
Registered: ‎04-09-2018

Re: MQTT connection for IoT device not working

1. Can the monitor-io config be amended to use hardcoded DNS resolvers e.g. Google's rather than the router proxy that's assigned by DHCP? If we're looking in the right place, then I'd expect things to work using this set-up.

 

We did that and it worked instantly.

 

2. I should try grabbing a capture from my test lab, and see what it shows.

3. We should also look to grab a capture from the WAN interface of the router when this is happening. I can do that remotely with @chrisking's assistance, I'd first need the serial number of the Hub One in use though.

 

Happy to help - I'll DM you the serial number. @monitor-io will need to reset the DNS entry.

 

Edit: a word

 

chrisking
Dabbler
Posts: 15
Thanks: 12
Registered: ‎04-09-2018

Re: MQTT connection for IoT device not working

It looks like the size difference comes from the reply from the Hub having a list of authoritative nameservers in addition to the answers to the query.

OS X might be stripping those out.

 

Chris

bobpullen
Community Gaffer
Community Gaffer
Posts: 16,869
Thanks: 4,950
Fixes: 315
Registered: ‎04-04-2007

Re: MQTT connection for IoT device not working


@chrisking wrote:

It looks like the size difference comes from the reply from the Hub having a list of authoritative nameservers in addition to the answers to the query.

Looks like you're right Thumbs_Up

This from ours:

~ $ dig a3n046iw3l65aq.iot.us-east-1.amazonaws.com @212.159.6.10

; <<>> DiG 9.9.5-9+deb8u16-Raspbian <<>> a3n046iw3l65aq.iot.us-east-1.amazonaws.com @212.159.6.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4459
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;a3n046iw3l65aq.iot.us-east-1.amazonaws.com. IN A

;; ANSWER SECTION:
a3n046iw3l65aq.iot.us-east-1.amazonaws.com. 300 IN CNAME iotmoonraker.us-east-1.prod.us-east-1.pb.iot.amazonaws.com.
iotmoonraker.us-east-1.prod.us-east-1.pb.iot.amazonaws.com. 295 IN CNAME dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com.
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 11 IN A 34.201.86.33
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 11 IN A 34.199.71.15
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 11 IN A 54.208.138.178
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 11 IN A 34.230.169.77
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 11 IN A 34.236.229.54
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 11 IN A 52.0.31.133
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 11 IN A 34.234.106.18
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 11 IN A 34.193.201.46

;; AUTHORITY SECTION:
us-east-1.elb.amazonaws.com. 122 IN     NS      ns-1119.awsdns-11.org.
us-east-1.elb.amazonaws.com. 122 IN     NS      ns-1793.awsdns-32.co.uk.
us-east-1.elb.amazonaws.com. 122 IN     NS      ns-235.awsdns-29.com.
us-east-1.elb.amazonaws.com. 122 IN     NS      ns-934.awsdns-52.net.

;; Query time: 158 msec
;; SERVER: 212.159.6.10#53(212.159.6.10)
;; WHEN: Thu Sep 13 20:40:58 BST 2018
;; MSG SIZE  rcvd: 477

This from Google:

~ $ dig a3n046iw3l65aq.iot.us-east-1.amazonaws.com @8.8.8.8

; <<>> DiG 9.9.5-9+deb8u16-Raspbian <<>> a3n046iw3l65aq.iot.us-east-1.amazonaws.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33234
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;a3n046iw3l65aq.iot.us-east-1.amazonaws.com. IN A

;; ANSWER SECTION:
a3n046iw3l65aq.iot.us-east-1.amazonaws.com. 299 IN CNAME iotmoonraker.us-east-1.prod.us-east-1.pb.iot.amazonaws.com.
iotmoonraker.us-east-1.prod.us-east-1.pb.iot.amazonaws.com. 852 IN CNAME dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com.
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 29 IN A 52.207.117.234
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 29 IN A 52.1.97.174
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 29 IN A 52.206.140.104
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 29 IN A 52.200.108.124
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 29 IN A 52.204.235.24
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 29 IN A 52.201.108.255
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 29 IN A 52.20.205.12
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 29 IN A 52.21.111.93

;; Query time: 220 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Sep 13 20:41:07 BST 2018
;; MSG SIZE  rcvd: 340

So, this leads me to a potential workaround that I suspect will help some of your other customers @monitor-io

If a customer enables our Safeguard parental control filter, waits a while, and then disables it. They should be assigned a different pair of resolvers following a PPP disconnect/reconnect or router reboot:

213.120.234.42
213.120.234.38

These resolvers don't return the authoritative nameservers either, so I'd put my money on them 'fixing' the problem:

~ $ dig a3n046iw3l65aq.iot.us-east-1.amazonaws.com @213.120.234.42

; <<>> DiG 9.9.5-9+deb8u16-Raspbian <<>> a3n046iw3l65aq.iot.us-east-1.amazonaws.com @213.120.234.42
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38067
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;a3n046iw3l65aq.iot.us-east-1.amazonaws.com. IN A

;; ANSWER SECTION:
a3n046iw3l65aq.iot.us-east-1.amazonaws.com. 300 IN CNAME iotmoonraker.us-east-1.prod.us-east-1.pb.iot.amazonaws.com.
iotmoonraker.us-east-1.prod.us-east-1.pb.iot.amazonaws.com. 543 IN CNAME dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com.
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 19 IN A 34.201.86.33
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 19 IN A 34.236.229.54
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 19 IN A 52.0.31.133
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 19 IN A 54.208.138.178
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 19 IN A 34.206.140.74
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 19 IN A 34.202.141.67
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 19 IN A 34.193.201.46
dualstack.iotmoonraker-u-elb-1bkszq0bvnbyv-1001897703.us-east-1.elb.amazonaws.com. 19 IN A 34.199.71.15

;; Query time: 120 msec
;; SERVER: 213.120.234.42#53(213.120.234.42)
;; WHEN: Thu Sep 13 20:42:13 BST 2018
;; MSG SIZE  rcvd: 340

It also provides the answer to the third of my questions:

3. Why couldn't I replicate this problem when I tried?

The broadband account I was testing from has previously had Safeguard enabled.

Going back to the fact that the original reports cited various Plusnet routers, I've a gut feeling there's quite a few devices out there whose DNS proxies don't handle this situation very gracefully. I've a fair bit of kit at my disposal, so I might test a few devices when I get the chance - with the 212.159.x.x resolvers assigned of course.

I'll also ask the vendor for confirmation of how the Hub One handles this.

We could probably change the behaviour of our DNS caches but I'm fairly certain we're adhering to RFC's so shouldn't have to.

Edit: @chrisking, please don't try turning on Safeguard, because I won't be able to revert your DNS assignment afterwards!

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

chrisking
Dabbler
Posts: 15
Thanks: 12
Registered: ‎04-09-2018

Re: MQTT connection for IoT device not working

@bobpullen Don't worry - I understand. I'm not going to do that

MisterW
Superuser
Superuser
Posts: 14,575
Thanks: 5,411
Fixes: 385
Registered: ‎30-07-2007

Re: MQTT connection for IoT device not working

@bobpullen @chrisking Looks like you might be getting to the bottom of thisSmiley

Not sure there's anything I can do to to help but shout up if there is.

I use a Draytek router on my PN connection and can change DNS servers easily, normally I use OpenDNS.

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

bobpullen
Community Gaffer
Community Gaffer
Posts: 16,869
Thanks: 4,950
Fixes: 315
Registered: ‎04-04-2007

Re: MQTT connection for IoT device not working

3. We should also look to grab a capture from the WAN interface of the router when this is happening. I can do that remotely with @chrisking's assistance...

So, we've now managed to do this and we can see that the TCP DNS request, and the subsequent queries that are suffixed with .lan are not being forwarded upstream by the DNS proxy.

This gives me sufficient evidence to query the situation with the router vendor. I'll also look to test some other devices if I get the chance next week...

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

bobpullen
Community Gaffer
Community Gaffer
Posts: 16,869
Thanks: 4,950
Fixes: 315
Registered: ‎04-04-2007

Re: MQTT connection for IoT device not working

@monitor-io, would it be possible for you to configure @chrisking's device so it's using hardcoded DNS again, but this time use one of our resolvers? Either 212.159.6.9 or 212.159.6.10.

This should help prove that it's specifically the Hub's DNS proxy that's contributing to the problem, rather than a wider issue handling the larger responses from our DNS servers.

I'd like this detail before I approach the vendor.

Thanks.

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

MisterW
Superuser
Superuser
Posts: 14,575
Thanks: 5,411
Fixes: 385
Registered: ‎30-07-2007

Re: MQTT connection for IoT device not working

@bobpullen I was just thinking about the first post on this thread which quotes the problems as being seen with the hub one, the hub zero and the TG582n. I could possibly see how the hub zero & one would possibly have similar dns proxy traits as they are the same supplier (sagemcom) but the TG582n is a completely different supplier (Techicolor ) isn't it ?

So I thought I'd try a little experiment with my Draytek 2830, using dig with the +tcp option to force TCP rather than UDP.

Currently the WAN connection has the PN 212.159.6.9/10 DNS servers assigned, although the LAN is configured to supply OpenDNS 208.67.220.220 & 208.67.222.222.

A dig forcing TCP works fine directly to both the PN DNS and OpenDNS servers but using the DNS proxy on the Draytek (192.168.5.1 ) it fails ( just sits there and sulks ).

So it would appear that router DNS proxies failing to forward TCP DNS requests is perhaps a slightly more common problem.

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

bobpullen
Community Gaffer
Community Gaffer
Posts: 16,869
Thanks: 4,950
Fixes: 315
Registered: ‎04-04-2007

Re: MQTT connection for IoT device not working

Thanks, that's useful to know. Does it fail when doing a lookup on just the Amazon AWS address, or does it fail for every hostname?

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

chrisking
Dabbler
Posts: 15
Thanks: 12
Registered: ‎04-09-2018

Re: MQTT connection for IoT device not working

The box is back online via the MacBook, so @monitor-io should be able to update the DNS entry.

 

Chris

MisterW
Superuser
Superuser
Posts: 14,575
Thanks: 5,411
Fixes: 385
Registered: ‎30-07-2007

Re: MQTT connection for IoT device not working

@bobpullen

Does it fail when doing a lookup on just the Amazon AWS address, or does it fail for every hostname?

Fails for everything

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

monitor-io
Dabbler
Posts: 15
Thanks: 12
Registered: ‎16-07-2018

Re: MQTT connection for IoT device not working

@bobpullen, the device is now setup to use 212.159.6.9 for DNS

@chrisking, you can move it back to the router

monitor-io
Dabbler
Posts: 15
Thanks: 12
Registered: ‎16-07-2018

Re: MQTT connection for IoT device not working

FYI @bobpullen - We just had a customer in Australia with a NetComm NF18ACV that was also unable to connect. So, based on our experience here we advised them to try turning off the router's DNS Proxy function (as it's configurable). And boom, we immediately connected...didn't even require a router reboot.