cancel
Showing results for 
Search instead for 
Did you mean: 

We don't compromise on value or service - HA! (SSH inbound issue)

redtela
Grafter
Posts: 28
Thanks: 2
Registered: ‎28-05-2015

We don't compromise on value or service - HA! (SSH inbound issue)

I'm hopeful that this forum, and particularly the PN staff that frequent it, will be able to assist (and then I'll happily eat my words).

Until approx. a week ago, I used to be able to SSH from remote locations to my home PN account. Then it stopped working (simply times out when connecting).

So tonight, I started poking. Before today, the only configuration change applied to the PN supplied router was to add my Raspberry Pi to the DMZ. That worked beautifully, until about a week ago. It had been working like this for over a year.

Diagnosis attempts

Remote nmap to my PN IP address, with ping scan disabled. No port open. Also doesn't send RST as it doesn't actively report closed.
Remote ssh attempt, using valid account details, from remote location to my PN IP address. Timeout (hardly surprising considering the above).
Analysis of the Pi's auth.log, no inbound connection attempts seen.
Analysis of the Pi's firewall (iptables). No packet counters increased during the diagnosis (either accept or deny/drop rules - literally no packet counts changed).
Analysis of the PN router logs. No information provided.
Cable checks - the Pi is connected to the ethernet port on the back of the PN router. I can connect to the Pi from wifi, ergo, eth0 on the Pi is fine.
Still experiencing SSH timeouts on inbound connections.

All of that done, I modified the advanced settings on the PN router - specifically, disabling the firewall, keeping the Raspberry Pi in the DMZ (I'm aware of the risks, more on that later).

So, with that looked at, I decided to talk to PN support (via chat). Spoke to "[CSA Removed]" (and I have emailed the transcript to myself). [CSA Removed] has informed me that PN do not offer support on this, and are not trained on it, due to it being advanced features of the PN router that they supply. I find that odd, that there is NO provision for advanced support (heck, bill me for the advanced feature support, I'll pay!).

I specifically asked [CSA Removed] if there really is no Network Operations team at PN that are trained in these matters, and could Ash log a ticket for them to investigate (I'm happy to wait for an investigation to happen). No, [CSA Removed] could not log a ticket for someone to investigate.

 

I'm sorry, don't PN pride themselves on customer service? This was working, and with absolutely no changes from me, it's stopped working. All analysis points to the issue likely being packet filtering of some description on the PN network. I even see a similar thread, that was resolved by PN: https://community.plus.net/t5/Fibre-Broadband/No-traffic-over-SSH/m-p/1348861#M41326 (that discusses port forwarding, whereas I can't do any port forwarding, because I can't connect!).

As I told [CSA Removed], I'm aware that I'm outside my minimum contract term with PN, and while their service has been great, this is a required feature for me. If PN can no longer supply this service, I have absolutely no quarms about migrating to a provider who can.

A little background info: I'm a Software Developer/DevOps. I setup networking systems as part of my work life. I'm aware I could request a static IP etc, but I don't see why I should pay for something I was receiving for free. I know how to configure the PN router so that it should work, but seemingly, if the PN router is the fault here, it's not obeying the configuration applied (when it once was!). I'm aware I should probably reboot the PN router, but reluctant to do that at the moment as the SO is watching Netflix.

 

Apologies for the rant, but if I told a customer "we have no-one trained to do that, and I can't log a ticket for someone to investigate" - I'd be getting my P45. Incensed at the lack of Customer Service is an understatement. Especially as this is either my first or second interaction with PN Customer Services.

Moderator's note by Mike (Mav): CSA name removed as per Forum rules.

EDIT: Didn't know about the forum rules, no problem. A cursory glance tells me that several ISPs have deals for around half the cost of PN, and moving to those may bring other benefits. Time to do a little shopping.

20 REPLIES 20
ScottStorey
Pro
Posts: 410
Thanks: 130
Fixes: 1
Registered: ‎21-02-2013

Re: We don't compromise on value or service - HA! (SSH inbound issue)

Have you checked the broadband firewall in the member centre?

Details here: https://www.plus.net/help/broadband/about-plusnets-broadband-firewall/

It's probably set to some level of on and blocking the SSH traffic. The broadband firewall is set at a network level inside the plusnet network so no amount of port forwarding on your router would help.
redtela
Grafter
Posts: 28
Thanks: 2
Registered: ‎28-05-2015

Re: We don't compromise on value or service - HA! (SSH inbound issue)

Thank you for the reply.

I either wasn't aware of the Broadband Firewall, or I don't remember setting it. Something the CSA could have (should have) pointed out. (Should'a, Could'a, Would'a...)

But the Broadband Firewall is "OFF" (in both settings pages).

rongtw
Seasoned Hero
Posts: 6,973
Thanks: 1,541
Fixes: 12
Registered: ‎01-12-2010

Re: We don't compromise on value or service - HA! (SSH inbound issue)

I am no expert , But could you try a Non PN router ?

Asus ROG Hero Vii Z97 , Intel i5 4690k ,ROG Asus Strix 1070,
samsung 850evo 250gig , WD black 2 TB . Asus Phoebus sound ,
16 gig Avexir ram 2400 , water cooling Corsair H100i gtx ,
Corsair 750HXI Psu , Phanteks Enthoo pro case .
redtela
Grafter
Posts: 28
Thanks: 2
Registered: ‎28-05-2015

Re: We don't compromise on value or service - HA! (SSH inbound issue)

Again, thanks for the thought.

The fact remains, that the PN router was doing the job fine until about a week ago, and without changes to settings, the setup stopped working.

I have no reason to consider the PN router is the problem.

Much more likely, is that PN have changed their network for security reasons. Even with the Pi configured to limit access, I used to see bots probing SSH many times a day. So from a malware perspective, that security precaution is sensible. But providing means to opt out for experienced customers would be nice, if this is the cause of my problems.

Tonight, I'll try changing the port that SSH listens on - not ideal but might provide an interim solution.

Failing that, I can SSH outbound, so can setup a tunnel from a remote location to the PN IP. That gives me two endpoints to secure & monitor though.

It would be great if PN had someone trained in such matters to check their ACLs. CSA say that such a team doesn't exist, and frankly, that's enough for me to start shopping around.
spraxyt
Resting Legend
Posts: 10,063
Thanks: 674
Fixes: 75
Registered: ‎06-04-2007

Re: We don't compromise on value or service - HA! (SSH inbound issue)

I've no reason to think such a change might be significant but was your connection changed to the new network at the time this problem started?

David
redtela
Grafter
Posts: 28
Thanks: 2
Registered: ‎28-05-2015

Re: We don't compromise on value or service - HA! (SSH inbound issue)

Any ideas how, as a consumer, I'd check this?

Different network implies different ACLs, and the other thread that my OP links to mentions this network change, and a static IP as resolving the issue.

Edit: the other thread implies 146.0.0.0/8 is part of this "new network" - my dynamic IP has certainly been in that range, while the problem has been happening. IP is currently in the 194.0.0.0/8 range.

Thanks.
spraxyt
Resting Legend
Posts: 10,063
Thanks: 674
Fixes: 75
Registered: ‎06-04-2007

Re: We don't compromise on value or service - HA! (SSH inbound issue)

With a dynamic IP, if the result from the Gateway Checker http://usertools.plus.net/@gateway/ mentions an Internet Router (pxx-irxx) your current connection is on the new network. However unless you know when the switch occurred this might not be significant.

David
redtela
Grafter
Posts: 28
Thanks: 2
Registered: ‎28-05-2015

Re: We don't compromise on value or service - HA! (SSH inbound issue)

Thanks for that, I'll check tonight when I'm home.

The IDS I configured for the aforementioned SSH malware bots will tell me when SSH stopped working, with an accuracy in the order of a few hours.

The thing I won't be able to tell, is when my connection moved to the new network, if indeed, I have been migrated.

Still, it's a start...
redtela
Grafter
Posts: 28
Thanks: 2
Registered: ‎28-05-2015

Re: We don't compromise on value or service - HA! (SSH inbound issue)

Well, the UserTools Gateway checker apparently cannot determine my gateway: "Sorry, the tool failed to identify your gateway; this does not mean that anything is wrong with your connection."

I'll make a comment in the relevant thread about that tool.

Traceroute tells me that the likely gateway is be3-3104.psb-ir01.plus.net (this is the last hop on the PN network) - this seems to fit with the previous poster's information about the net network, and that provides a more concrete link to the thread in my OP.

I've also seen traceroutes that appear to show be3-3106.psb-ir02.plus.net as the gateway. Still fitting the naming convention.

My IDS stopped reporting intrusion attempts on SSH on 02 November. Specifically, the last attempt logged was 01 November 18:08:19. SSH logs verify this.

Changing the port that SSH listens on (I've tried 4 separate ports), while the PN router firewall is disabled and the Pi is within DMZ makes absolutely no difference.

So, questions to PN:

  1. Why can CSA not raise tickets on behalf of customers for further investigation of network routing? (Or, why can they not advise customers on how to raise such tickets themselves?)
  2. Can you confirm the date/time (even if only approx.) that I was migrated to this "new network" infrastructure?
  3. Are you performing any packet filtering for inbound connections to residential customers on this "new network"?
  4. Do you have anyone trained, capable of investigating this issue?
  5. Why are CSA not informed about the problems with routing, as highlighted in this thread (which claims PN staff are aware of this problem): https://community.plus.net/t5/Fibre-Broadband/Gateway-Check-Issues/td-p/1335003/page/5

Finally, thank you to the other posters (whom I assume are fellow PN customers themselves) for providing input so far.

 

EDIT: @spraxyt - I concur with your observations of late July. I've done some mild network probing (nothing that would upset anyone with brains at PN). It appears the gateway I'm connected to doesn't fully have ACLs configured. More often than not, hops 3 & 4 don't announce properly - I'm yet to see hop 3 announce. Hop 2 is my PN router at 192.168.1.254. Probably an over zealous ICMP configuration. Occasionally I see BGP routers on other networks (particularly *.msn.net) fail to announce too, but I suspect that's due to Microsoft's incessant requirement for multiple redirects.

ejs
Aspiring Hero
Posts: 5,442
Thanks: 631
Fixes: 25
Registered: ‎10-06-2010

Re: We don't compromise on value or service - HA! (SSH inbound issue)

How is the Pi obtaining its IP address? Is it from the router by DHCP? Or is it set statically on the Pi itself? I recall port forwarding seems to randomly stop working after some time for at least one of the Plusnet routers, they seem to have problems with devices that end up named as Unknown in the router interface. That problem might apply to the DMZ function also.

As far as I can see, the packets aren't reaching your Pi, but there's nothing to determine whether they are reaching your router or not. I think it's more likely to be a problem with the router not doing the forwarding / DMZ as it should be doing.

redtela
Grafter
Posts: 28
Thanks: 2
Registered: ‎28-05-2015

Re: We don't compromise on value or service - HA! (SSH inbound issue)

@ejs - certainly worth looking into.

The Pi gets its IP from the PN router, in fact, it's the only device on my home network that actually uses the PN DHCP, and therefore always gets assigned 192.168.1.201.

However, I did note last night that it was showing as "Unknown" - even though the MAC was correct (since port forwarding and DMZ assignment on the PN Thomson router is via MAC address, rather than IP).

I'll have a play & report back...

spraxyt
Resting Legend
Posts: 10,063
Thanks: 674
Fixes: 75
Registered: ‎06-04-2007

Re: We don't compromise on value or service - HA! (SSH inbound issue)


@redtela wrote:

Well, the UserTools Gateway checker apparently cannot determine my gateway: "Sorry, the tool failed to identify your gateway; this does not mean that anything is wrong with your connection."

I'll make a comment in the relevant thread about that tool.

Traceroute tells me that the likely gateway is be3-3104.psb-ir01.plus.net (this is the last hop on the PN network) - this seems to fit with the previous poster's information about the net network, and that provides a more concrete link to the thread in my OP.

I've also seen traceroutes that appear to show be3-3106.psb-ir02.plus.net as the gateway. Still fitting the naming convention.

My IDS stopped reporting intrusion attempts on SSH on 02 November. Specifically, the last attempt logged was 01 November 18:08:19. SSH logs verify this.


The problem with the gateway checker failing to identify the gateway is a well known one. The Internet Router which follows the hop shown doesn't identify itself (the tool suppresses trailing asterisks from the trace). Plusnet are aware of this but investigating is low priority since it does not prevent the router serving its primary role. Based on your outbound trace it will be in the psb-ir01 cluster in Southbank Point of Presence. However you are on the new network.

During the night preceding the last intrusion log entry Plusnet Service Status had announced Planned Network Maintenance - Tuesday 1st November 01:00 - 06:00. Are you aware of any transient disconnections in that period or in the 24 hours following that work?

David
redtela
Grafter
Posts: 28
Thanks: 2
Registered: ‎28-05-2015

Re: We don't compromise on value or service - HA! (SSH inbound issue)

I'm not aware of any connection drops, but I only require adhoc connections - there's nothing persistent.

I'm not concerned about the tool, and the routers that fail to announce properly. That's for folks like you to worry about. Smiley As the tool says, the tool output has no baring on my connection status.

 

However, hopefully a final update from me...

Possibly a combination of rebooting the Pi and the PN router, but heypresto, everything works as it should again. Again, thank you to the posters that have been trying hard to provide constructive input.

Unfortunately, this whole debacle occurred because I didn't want to incur the wrath of the SO while she was watching Netflix - if not for that, I'd have followed "Standard Fix Procedure #1 - If in doubt, reboot."

 

Such a shame that PN CSA can't even recommend SFP#1, even when a customer starts talking about "advanced" router features. Anyway, I'm out of minimum terms, so able to vote with my feet.

 

Again, thanks to the posters here - maybe PN should pay you! Smiley

redtela
Grafter
Posts: 28
Thanks: 2
Registered: ‎28-05-2015

Re: We don't compromise on value or service - HA! (SSH inbound issue)

OK, so I'm back. Sad

It seems I was lucky with the restart of everything on the network, as it no longer accepted inbound connections this morning. It's still that way, despite restarting everything again.

Further investigation into PN network routing implies that the gateway/router issues (that are well known about) could well be linked to my problems.

Occasionally today, I've seen "no route to host" when trying to communicate from remote locations to my PN IP (Pi still in DMZ, firewall off, port forwarding enabled). So a few traceroutes, all of which return the following:

2 vl-1960.gw-distp-a.kw.nbz.fr.oneandone.net (195.20.243.93) 0.285 ms 0.298 ms *
3 ae-4-0.bb-a.bap.rhr.de.oneandone.net (195.20.243.39) 0.681 ms 0.680 ms 0.659 ms
4 ae-4.bb-d.bv.crb.fr.oneandone.net (212.227.120.41) 8.177 ms 8.272 ms 8.358 ms
5 ae-5-0.bb-a.ba.slo.gb.oneandone.net (212.227.120.29) 15.414 ms 15.409 ms 15.455 ms
6 linx4.ukcore.bt.net (195.66.236.11) 23.956 ms 16.461 ms 16.476 ms
7 core1-hu0-9-0-1.southbank.ukcore.bt.net (195.99.127.2) 17.813 ms 17.846 ms 17.878 ms
8 * 195.99.125.139 (195.99.125.139) 17.201 ms be10.psb-ir01.plusnet.bt.net (195.99.125.131) 17.702 ms
9 * * *
10 * * *

Hop 1 omitted for obvious reasons. From psb-ir01 onwards, I cannot perform ICMP traces, this backs up the "no route to host."

So, given CSA tells me PN don't have a team to look at this, and that they can't raise a ticket, can anyone advise how I open a ticket with PN support please? I'm suitably convinced that the problem I have is on the PN network, outside my house (especially, as the Pi allows all ICMP).

Thanks in advance. Smiley

EDIT: The CSA did open a "question" - #137623063 with regard to my request to the CSA to "put a note on my account" detailing that I'd be talking to customer retentions. That's a start, but I guess PN are happy to lose customers, based on the CSA approach to problem solving so far.