Evening VPN performance issue (IPsec, not L2TP)

[#1]

That is not very good and could indicate exchange congestion.
Can you post a tracert to ntp.plus.net please. I'd like to see confirmation that that traffic is going direct and not via the VPN (I'm presuming that the router only routes traffic for the LAN via VPN).
Alternatively can you post the results of "pathping ntp.plus.net" which will show where the packet loss occurs.

[#2]

Tracing route to ntp.plus.net [212.159.6.9]
over a maximum of 30 hops:
  0  MAXDATA [192.168.3.100]
  1  home.gateway [192.168.3.254]
  2  lo0-plusnet.pcl-ag01.plus.net [195.166.128.27]
  3  gi1-2-803.pcl-gw1.plus.net [84.92.5.57]
  4  vl63.pcl-lb01.plus.net [212.159.2.252]
  5  ptn-cdns01.plus.net [212.159.6.9]
Computing statistics for 125 seconds...
            Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address
  0                                           MAXDATA [192.168.3.100]
                                0/ 100 =  0%   |
  1    1ms     0/ 100 =  0%     0/ 100 =  0%  home.gateway [192.168.3.254]
                                0/ 100 =  0%   |
  2   25ms     0/ 100 =  0%     0/ 100 =  0%  lo0-plusnet.pcl-ag01.plus.net [195
.166.128.27]
                                1/ 100 =  1%   |
  3   30ms     2/ 100 =  2%     1/ 100 =  1%  gi1-2-803.pcl-gw1.plus.net [84.92.
5.57]
                                0/ 100 =  0%   |
  4   24ms     2/ 100 =  2%     1/ 100 =  1%  vl63.pcl-lb01.plus.net [212.159.2.
252]
                                0/ 100 =  0%   |
  5   24ms     1/ 100 =  1%     0/ 100 =  0%  ptn-cdns01.plus.net [212.159.6.9]

Trace complete.

Right now the server response is better, directory browse in under 10s, mostly. It still seems sluggish though.

[#3]

That looks OK, can you repeat the test when the VPN is slow and you are getting bad pings for us to compare please.