cancel
Showing results for 
Search instead for 
Did you mean: 

Web Browsers and their http requests 'on the wire'.....

pawhe955
Grafter
Posts: 106
Registered: ‎31-07-2007

Web Browsers and their http requests 'on the wire'.....

Not sure if this is the right forum, but a techie question on http protocol requests......
Have set up a discrete ('guest's usage') Wireless LAN at my workplace - DHCP being served from the Wireless Switch Module, serving OpenDNS Servers for Internet name resolution, and providing a default gateway IP which is the address of a discrete NIC on an ISA 2006 Server, which is running both an explicit proxy for http/s, and I think (as I'm not an MS Server Administrator... - more of an Data Network/IP Network engineer...) a transparent proxy for http, and which is configured to proxy-chain to our 'URL filtering service' provider's proxy.....  Devices on the 'other' ("corporate", non-guest) network are set by enforced group policy to explicitly proxy via (a different NIC/interface on) the ISA - and always work for Internet browsing o.k.
I have the issue that when the 'guest network' Wireless device's browser is configured explicitly to proxy via the ISA server, Web Browsing is o.k.. But if I clear all proxy settings (emulating a 3rd party bringing in their own laptop/Netbook/iPhone/AndroidPhone, etc.), then I get an http 404 "page not found" error when attempting connection to most public Internet servers. Further testing has found that (with cleared proxy settings) I can connect to dedicated webservers (e.g. one web server running per piece of hardware/IP address - like an Internet Camera applicance - or in fact Google's home page), but cannot connect to virtual web servers (e.g. one of multiple web-servers running behind 1 IP - for example, the bit of web space that I get from Plusnet, accessed by "www.<plusnetname>plus,com"). I know (believe?) that direction to a virtual web server's index page requires that the GET request from the client includes the full URL/URI<?> (DNS name and path to file) of the resource being requested (i.e. html page or graphic, etc. - e.g. 'GET http://www.google.com/ HTTP/1.0'), so initially blamed our filtering service proxy for somehow interfering with (corrupting or otherwise changing) the GET request 'on route' to the target Internet Webserver, hence causing a 404 error....
However....  Wireshark traces at the client (Windows XP, IECool reveal that when IE8 is configured with an explicit proxy setting, the GET request includes the full URL - but when the proxy settings in IE8 are cleared, the GET in a simple (truncated) "GET /" - which of course explains why the connection works to single-instance web servers, but not on multiple-instance virtual web-servers.... (as the latter don't know which virtual server/index page to route the request to).
Can anyone confirm that this is "normal" behaviour for IE8 - or any other Web-Browser? (haven't had time to test with others at the moment) - I'm just really surprised to see the simple "GET /" go out on the wire when I've entered the full URL into the browser's address bar, and whilst IE's proxy settings are confgured like probably 90% of end-user's IE installations [e.g. *not* configured]....??  I can/will also do some Wireshark tests at home - but that'll be later, and was just wondering if anyone could point me in the right direction whist still at work looking at the issue...?Huh
Thanks,
P.
Edit: Sorry - after reviewing the other forums, this should obvioulsy not be in Plusnet (specific) Help and Support Forum - probably more like one of the Technology ones. I can't see a way of moving it myself - could a mod do so - thanks.
2 REPLIES
Community Veteran
Posts: 3,789
Registered: ‎08-06-2007

Re: Web Browsers and their http requests 'on the wire'.....

Kinda happy for it to live here...
I am theorising that the problem may still be to do with your proxy.
The feature that you talk of which allows multiple websites per IP address is a feature of HTTP 1.1 (which all browsers should support by default).  This means that when negotiating with the webserver the browser should send the 'Host:' Header specifying the host as entered on the browser address bar.
It seems to me that the browser in this case is actually falling back on HTTP 1.0 which didn't support the Host header and the only reason I can think of is that the browser is aware of your proxy and seems to believe that it won't pass on HTTP 1.1.  Alternatively, disabling the proxy settings may disable the HTTP 1.1 'extensions' in the browser (I've seen MS software do plenty of 'interesting' things, believe me)
Hope those nuggets help you find a solution
Cheers
B.
pawhe955
Grafter
Posts: 106
Registered: ‎31-07-2007

Re: Web Browsers and their http requests 'on the wire'.....

Thanks for the reply Barry - pointed me in the right direction to look further....
At home, my "no proxy settings" browser connects to remote websites with no issues, and Wireshark shows that the GET is a plain and simple "GET /" - but (as per your points), there is a "host:" header further down the request data, with the full DNS name of the destination Web-Server. Research (including what looks like a good website for plain speaking about HTTP 1.0/HTTP 1.1 - "http://www.jmarshall.com/easy/http/") suggests that HTTP1.1 (only) requires that when a browser is configured for a proxy, it sends the full URL/URI (DNS host name / path to resource) in the GET line (e.g. "GET http://www.bloggs.com/index.htm HTTP/1.1"), but when no proxy is configured, it sends the plain "GET / HTTP/1.1", but MUST then include the "host:" header - and all HTTP 1.1 proxies and servers must be able to deal with these requests, when the specified protocol is HTTP 1.1....
Back at work today, looked at the Wireshark trace from yesterday for my "no proxy settings" browser (failing) connection - and indeed, the request is HTTP 1.1, and (as referred to yesterday) is a plain "GET /" - but the request does in fact include the (correct) "host:" header data further down..... (something, until your posting and the reference to the other website, I wasn't looking for yesterday). So my IE is in fact behaving as it should. Which I guess means that something further along (tranparent proxy in ISA, or upstream filtering proxy) is failing, perhaps in not supporting HTTP 1.1/changing the chained request to HTTP 1.0, etc. Further investigation required....  but the main question as to whether IE is doing (or appears to be doing) as it should, is answered - so thanks.
P.