Link 403 problem

[#1]

I have a website hosted on the CGI site, with an integral blog. I've just added to the blog on the site and then tried to post an update on LinkedIn with a link to the blog.
However, LinkedIn just displays a "403 Forbidden" error.
The last time I used a link on LinkedIn (or anywhere) was before the CGI hosting change earlier this year.
Note that this error occurs with any link to my website, even the home page.
Do I need to change any settings in my PlusNet host?
I've just spent days creating content so it's a real problem not to be able to use it!

[#2]

The site is accessible for me using Firefox or Internet Explorer. Does Linkedin handle embedded Shockwave video?

[#3]

Hi Spraxyt
Thanks for responding.
There are no problems accessing the site from browsers. It's only when I post a link within LinkedIn. Usually LinkedIn displays a thumbnail/image from the target side a a bit of text. I can post a link from another site, say the BBC site and it works fine. By posting a link from my site gives the 403 error.
Not clear on your comment about Shockwave - I don't have any video on my site - it's very simple.
Thanks

[#4]

Hi AlFox,
I've been having a bit of a dig into this one for you and can't seem to pull anything up, even my Google-Fu has failed me on this one!
There only seems to be material on the development side of the support site that outline a possible api issue with regards to throttling however I don't think this would link back to your issue in any way so could be best to check with someone here.
Aside from this I can't think there would be any permissions locking down your ccgi space that would cause it 

[#5]

Hi Acassim
Thanks for digging.
It's not LinkedIn only because I can post third-party links and the work correctly. Must be something to do with LinkedIn and my hosting.
There is a thread on LinkedIn Help here http://community.linkedin.com/questions/188702/adding-a-link-on-status-update-and-getting-forbidd.ht... that raises the same issue but not solutions as yet...
Regards
Al

[#6]

Does linkedin try to connect to the site using https?

[#7]

Hi Spraxyt
Unfortunately I don't know how LinkedIn accesses the PlusNet hosting.
However, I saw the following response in a LinkedIn thread about this problem:

I spoke to the support guys at our web hosting company (Clook) and they identified that the UserAgent that Linkedin uses is "LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +[link removed])"
Apparenty Jakarta is mostly used in HTTP flood attacks, and for this reason they had it disabled when attacks can occur.
So they removed the block on Jakarta and hey presto, no more error messages.
Note that it took 24 hours for the fix to work as we had to wait for LinkedIn to clear its cache.

This doesn't mean much to me, but perhaps someone familiar with PluNet's set-up can let me know if this is likely to be a source of the problem, and whether they can offer a solution?
BR
Alistair

[#8]

I think Plusnet need to advise if LinkedInBot/1.0 or Jakarta Commons-HttpClient/3.1 is blocked on the Hostopia server. I assume you don't block either of them yourself in .htaccess?
David

[#9]

Thanks spraxyt,
I think this seems like one of the most likely options, unfortunately my go-to guru on hosting is sunning himself on holiday at the moment so I'll have a dig around and see who can grab me some answers on these.
@AlFox - From looking at your Used/Free webspace on your account I'm assuming that this is where the site files are hosted. Can you confirm before I start looking into this?

Thanks,
Adam C

[#10]

Hi Acassim
Not sure I understand your question.
The files are on the new Hostopia CGI site. Nothing special. I think all the files are in the Public folder.
I've not changed any of the standard settings nor htaccess etc.
If it's a hosting setting then I would have thought other PlusNet users might have the same problem, or perhaps they don't post links to LinkedIn?
Best regards
Alistair

[#11]

Thanks Alistair,
I just needed to know if this was on Hostopia.
I'll have a quick dig around and see if I can get some answers for you, fingers crossed we get this resolved for you!

Cheers,
Adam C

[#12]

Yep, definitely seems to be something to do with the user-agent:
This works:

~$ telnet ccgi.bobpullen.plus.com 80
Trying 91.136.8.9...
Connected to ccgi.bobpullen.plus.com
Escape character is '^]'.
GET / HTTP/1.1
Host: ccgi.bobpullen.plus.com
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) <-- Firefox web browser
HTTP/1.1 200 OK
Date: Wed, 11 Jun 2014 15:03:01 GMT
Server: Apache
Last-Modified: Fri, 11 Apr 2014 16:09:23 GMT
Content-Length: 767
Content-Type: text/html
<html>
<head>
<title>Yay!</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
   ...

This doesn't:

~$ telnet ccgi.bobpullen.plus.com 80
Trying 91.136.8.9...
Connected to ccgi.bobpullen.plus.com
Escape character is '^]'.
GET / HTTP/1.1
Host: ccgi.bobpullen.plus.com
User-Agent: LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com) <-- LinkedIn's web crawler.
HTTP/1.1 403 Forbidden
Date: Wed, 11 Jun 2014 15:14:13 GMT
Server: Apache
Content-Length: 202
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.</p>
</body></html>
Connection closed by foreign host.

I've asked our hosting partner to do something about this and there's a support ticket raised against your account to track the request (ref: 86942737).

[#13]

This should now be fixed if you fancy testing again?

[#14]

Bob
Thanks.
Yes, this is working now. Thanks for raising the issue. It may not have appeared important but it's a key channel for my blog posts!
BR
Alistair
PS: Is it possible to post what Hostpedia changed so I can relay that the LinkedIn help in case others get the same problem?

[#15]

Quote from: AlFox

PS: Is it possible to post what Hostpedia changed so I can relay that the LinkedIn help in case others get the same problem?

The latest response I received from them was that they were unable to replicate the problem. It didn't imply they'd changed anything although they must have. I'm fairly confident there will have been Apache directives in place to restrict access based on the user agent string. LinkedIn has it's own user agent and I don't think it was being allowed. To fix the problem a host would have to ensure that they're accepting connections from the LinkedIn user agent.
To test that it is this causing the issue you can telnet to the website in question as per the evidence in one of my previous replies or use something like curl from a *nix machine, e.g.
If this returns the source code for your website:

~$ curl -A "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)" [address_of_your_website]

But this doesn't:

~$ curl -A "LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)" [address_of_your_website]

Then the LinkedIn user agent is probably being blocked by the host.