topic Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki in Tech Help - Software/Hardware etc

My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki.pl

dougrice — Fri, 19 May 2017 19:14:33 GMT

http://ccgi.dougrice.plus.com/cgi-bin/wiki.pl and my other perl scripts gets

"...

The requested URL was rejected. If you think this is an error, please contact the webmaster.

Your support ID is: 18036279169211782779

..."

If I use Private Browsing, it renders the page, but I cannot get the Wiki into Update mode!

If I delete all cookies it works first time, but Edit mode needs cookies.

These scripts have worked for years.

I deleted the older logs. I have loads of spare space.

As webmaster where do I use the support ID?

It is the same on different computers and networks.

This started about a moth ago, and getting worse.

Any Ideas?

regards, Doug

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Sat, 20 May 2017 07:32:45 GMT

A colleague said it might be to do with cookies. Does the CCGI server like cookies with & in them.

I stripped own another perl script to make it as simple as possible. It prints out the environment.

I have a wireshark capture. First one is not working, seconds one is working
=========================================
GET /cgi-bin/tt.pl HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, */*
Accept-Language: en-GB
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393
Accept-Encoding: gzip, deflate
Host: ccgi.dougrice.plus.com
DNT: 1
Connection: Keep-Alive
Cookie: Wiki=rev&1&id&3989&randkey&286140683

HTTP/1.1 200 OK
Connection: close
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Content-Length: 211

<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. If you think this is an error, please contact the webmaster. <br><br>Your support ID is: 18036279169218871260</body></html>

==========================================
GET /cgi-bin/tt.pl HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, */*
Accept-Language: en-GB
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393
Accept-Encoding: gzip, deflate
Host: ccgi.dougrice.plus.com
DNT: 1
Connection: Keep-Alive
Cookie: TS0194eee0=01a7d14f7fbd8ed2d93430826b98f227643c4c8933d30032c53e6ae7a0651d644e522f2cb9

HTTP/1.1 200 OK
Date: Sat, 20 May 2017 07:21:21 GMT
Keep-Alive: timeout=10, max=100
Connection: Keep-Alive
Content-Type: text/html
Set-Cookie: TS0194eee0=01a7d14f7fd89ac60e76d845ad9a2901104d9fae067996172d18ddd00d03531e030fd4d8b0; Path=/
Transfer-Encoding: chunked

<html>
<head>
<title>Test CGI output</title>
</head>
<body>
<h1>Test CGI output</h1>
<HR>
ARGS = <BR>
<H2>Environment variables</H2>
<pre>set DOCUMENT_ROOT=/services/webpages/c/c/ccgi.dougrice.plus.com/public
set GATEWAY_INTERFACE=CGI/1.1
set GDFONTPATH=/services/share/fonts
set HTTP_ACCEPT=text/html, application/xhtml+xml, image/jxr, */*
set HTTP_ACCEPT_ENCODING=gzip, deflate
set HTTP_ACCEPT_LANGUAGE=en-GB
set HTTP_CONNECTION=Keep-Alive
set HTTP_COOKIE=TS0194eee0=01a7d14f7fbd8ed2d93430826b98f227643c4c8933d30032c53e6ae7a0651d644e522f2cb9
set HTTP_DNT=1
set HTTP_HOST=ccgi.dougrice.plus.com
set HTTP_USER_AGENT=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393
set MvCONFIG_LIBRARY=/services/websoftware/miva/Empresa/cgi-bin/libmivaconfig.so
set PATH=/sbin:/usr/sbin:/bin:/usr/bin
set QUERY_STRING=
set REMOTE_ADDR=212.56.108.219
set REMOTE_HOST=dougrice.plus.com
set REMOTE_PORT=55815
set REQUEST_METHOD=GET
set REQUEST_URI=/cgi-bin/tt.pl
set SCRIPT_FILENAME=/services/webpages/c/c/ccgi.dougrice.plus.com/cgi-bin/tt.pl
set SCRIPT_NAME=/cgi-bin/tt.pl
set SCRIPT_URI=http://ccgi.dougrice.plus.com/cgi-bin/tt.pl
set SCRIPT_URL=/cgi-bin/tt.pl
set SERVER_ADDR=91.136.8.9
set SERVER_ADMIN=or webmaster
set SERVER_DOMAIN=ccgi.dougrice.plus.com
set SERVER_NAME=ccgi.dougrice.plus.com
set SERVER_PORT=80
set SERVER_PROTOCOL=HTTP/1.1
set SERVER_SIGNATURE=
set SERVER_SOFTWARE=Apache
set no_gzip=1
</pre><HR>
</body>
</html>

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

Anonymous — Sat, 20 May 2017 07:41:06 GMT

Isn't the cookie separator a semi-colon (;) this may be the issue you have here as I notice the other entry uses these.

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Sat, 20 May 2017 08:01:39 GMT

If I clear the browser cache, it works, for a few goes.

It breaks when I do a post.with my password in it.

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Sat, 20 May 2017 08:16:10 GMT

I am not sure. I think it is a website issue. The main modiwiki works.

My Instance on PlusNet seem to be the ones breaking. They have worked for years.

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Mon, 22 May 2017 09:57:57 GMT

I spent some time clearing the cache and looking at the access logs on ccgi site.

As soon as a web page has the cookie below attached, it stops the CCGI server serving my PERL cgi scripts.

Cookie: Wiki=rev&1&id&3989&randkey&286140683

I can access .php and .htm pages okay. CGI scripts ending in .pl break, but work again after clearing the cache.

The access logs do not show the GET when it is rejecting the URL.

I tried using CARP to report any errors to browser, but this did help much.

The cgi does not have error log files, only access log files.

I am confused.

Using the preference link on the Wiki seems to be the trigger.

Clear the cache and the perl scripts start working. As soon as the wiki attaches the cookie scripts seem to be rejected.

The Wiki has worked for a long time, and this behaviour has only started about a month or two ago.

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

spraxyt — Mon, 22 May 2017 17:46:45 GMT

PHP was updated to version 5.6 a few weeks ago, but this should have no effect on Perl scripts, and didn't in my experience. I think Apache might have been updated around that time too, though that is just a guess. However if this was done it changed nothing for me.

The obscure error message you mentioned in the opening post means an error of some sort did occur but printing it on screen has been suppressed for security purposes. It might even mean there was a Server Error 500 which could result from something in .htaccess. Whilst the message suggests finding out the underlying cause is easy - just ask; unfortunately I think the system log is inundated with error messages like that so it isn't; Doing your own diagnostics is likely to be far quicker.

To this end you mentioned using Carp to get a clue. Was this by adding

use CGI::Carp qw/fatalsToBrowser/;

in the preface at the top of the Perl script?

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Mon, 22 May 2017 20:18:37 GMT

Hi David,

I have tried adding the CARP code. Unfortunately, it does not show anything.

It does seem related to when I open the preference link.

http://ccgi.dougrice.plus.com/cgi-bin/wiki_doug.pl?action=editprefs

I will need to add my own debug code and disable functionality then add it back until it breaks!

Once the cookie is attached, this and other perl scripts stop working. This suggests the script is left in a state that overloads the CGI module for the next few GETS.

regards,

Doug

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

spraxyt — Tue, 23 May 2017 09:45:48 GMT

After visiting that linked page then trying to go elsewhere on the site I've experienced the "invalid URL" problem. Deleting cookie dhrWiki makes it work again.

I think the problem is that cookies being transmitted as part of the headers means the cookie value needs to be URL encoded (which would encode the ampersands).. Doing Perl diagnostics is unlikely to show anything because the failure occurs before Perl is entered.

I haven't used cookies in Perl but my impression is that the CGI::Cookie Perl module does URL encoding for you. Do you use this or roll your own?

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Thu, 25 May 2017 06:17:55 GMT

Many thanks,

It seems that & in the cookie value breaks the web server.

I consider this to be a bug in the webserver, as the RFCs make no mention of avoiding & in the value. The RFC only talk about = and ;

One workaround is change the wiki.pl script to use a different value separator.

I changed the cookie to use _ instead of & as a value separator.

Cookie: Wiki=rev&1&id&3989&randkey&286140683

becomes:

Cookie: Wiki=rev_1_id_3989_randkey_286140683

Now when I try Microsoft Edge, it is not breaking the web server.

I have another workaround for updating the wiki which works.

spraxyt

The Wiki I use was downloaded from http://www.usemod.com/cgi-bin/wiki.pl?UseModWiki a very long time ago.

So may I thank all who have helped get a solution. spraxyt's last comment was very helpful.

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

Anonymous — Thu, 25 May 2017 06:49:02 GMT

I did say this 5 days ago.

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Thu, 25 May 2017 09:02:25 GMT

Hi Mook,

Thanks a well. I had a look on Wikipedia and the RFC and it mentioned = and ; and spaces.

So having & in the value between = and ; has not been a problem except in this case.

For this wiki, the value is a list.

I changed & to _ this morning and it did not trigger the problem.

Once the cookie is attached to a perl script served by the PlusNet CCGI server, the server is breaking, until the cookie is deleted.

By changing the cookie to use a different list separator, it now get around the CCGI server defect.

The expiry date in the CGI server was hard wired to a date in 2013. I am not sure who is suppose to deleted expired cookies!

There is probably an issue here to look into.

Thanks

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

spraxyt — Tue, 30 May 2017 16:57:39 GMT

I extracted the cookie creation and setting code from the Wiki source into a small script and reproduced the "lockout" problem with this script.

However a test using a forms example from the CGI Perl Module distribution creates a cookie containing

Name: preferences
Value: text&black&background&silver&name&David&size&3

and that works OK; returns the values and updates if preferences are changed. The obvious difference is that it contains text items - I'll pass on why that might be significant.

The expiry date set in the Wiki is '+3y' (3 years hence) and that's what my test showed. That date would normally be extended by the script on each access. To delete a cookie one supplies a date in the past. Browsers should remove expired cookies from their store. However the cookie store is regarded as a finite space and browsers may choose to remove cookies before they expire. users can also remove them.

Scripts are expected to cope with absence of state information due to missing cookies.

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Tue, 30 May 2017 17:30:08 GMT

Thanks for the reply. Interesting that you get the lock out. Without the error log, this is almost impossible to fault find.

I have another perl CGI-script tt.pl which makes no attempt to decode the cookie. It works until I use the wiki and it attaches this cookie.

It breaks until I delete the cookies. These GET requests are not in the access.log whil it is broken.

I have other perl scripts that also work until this cookie is attached.

When I delete the cookies by deleting the browser history they work again. I use Midori, Firefox, Edge and IE11. ( IE11 has a feature that when you tick all the boxes, you do not delete cookies for your favorates! )

I have a work around. I have another copy of wiki.pl renamed but pointing at the same database. I enabled edits by anyone without a password. Using FTP I can enable and disable execute permissions for this script. I can update my wiki. My wiki got "handbagged" without a password back in 2011 by trollers putting links to products ( mostly handbags ) on my wiki and guest books.

This script has worked for years, but has just started to break the PlusNet CCGI server recently, after visiting the Preferences page and providing the correct password. It works on the Raspberry Pi still. Even the latest version breaks and the cookies code is different.

Many thanks, Doug

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

spraxyt — Tue, 30 May 2017 18:59:33 GMT

Because the path in the problem cookie is set to "/" (which means all the site) nothing at all will work, irrespective of language until that cookie is removed.

Using a PHP script I've listed the response headers passed by my form script (which works). These are

HTTP/1.1 200 OK
Date: Tue, 30 May 2017 18:16:14 GMT
Server: Apache
Set-Cookie: preferences=text&black&name&&background&silver&size&;
   path=/; expires=Thu, 29-Jun-2017 18:16:14 GMT
Content-Length: 2596
Connection: close
Content-Type: text/html; charset=ISO-8859-1

(The Set-Cookie header is on a single line, I split it across two line above for readability.)

The values shown are default settings, my PHP script doesn't understand cookies.

If I do the same check with the wiki extract script I get

HTTP/1.1 200 OK
Date: Tue, 30 May 2017 18:47:13 GMT
Server: Apache
Set-Cookie: hashcookie=randkey&56074534&rev&1&id&1357;
   path=/; expires=Thu, 01-Jun-2017 18:47:13 GMT
Content-Length: 91
Connection: close
Content-Type: text/html; charset=ISO-8859-1
X-Pad: avoid browser bug

Apart the addition of an X-Pad header (i'm not sure what that is about) the structure of the two Set-Cookie lines seems the same to me.

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

spraxyt — Wed, 31 May 2017 01:18:51 GMT

Courtesy of Google I discovered the X-Pad header is a fudge Apache adds to bypass a bug in a very ancient version of Netscape triggered when header character 256 or 257 is a new-line character. It gets added for all user agents since doing this was quicker than working out the actual one being used. So the presence of that header was not significant.

By experiment with my wiki extract test I've discovered how to make this work. It seems any cookie containing '&' characters will fail if

all settings are numerical, or
it contains a variable called 'id'.

When I replace the value of randkey with a random string of 9 letters and change 'id' to 'idnum' it works..

This gives a cookie like

Name: wikipreferences
Value: randkey&VKCvwhRCz&rev&7&idnum&1347

In that the value of 'rev' is 7 which means 6 update cycles have been done after initialisation.

Clearly my testing is far from exhaustive, and you discovered avoiding ampersands also works.

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Mon, 05 Jun 2017 07:31:31 GMT

Hi David,

I modified tt2.pl to send the cookie.

I reproduced the problem if I include &id&

I changed it to &ie& - still works

I changed it to &ib& - still works

#http://ccgi.dougrice.plus.com/cgi-bin/tt2.pl
sub PrintHeader {
#breakes
#print "Set-Cookie:Wiki=rev&1&id&3989&randkey&286140683";
#okay if id is changed to ib or ie
print "Set-Cookie:Wiki=rev&1&ib&3989&randkey&286140683";
#okay if id is changed to ib or ie
print "Set-Cookie:Wiki=rev&1&ie&3989&randkey&286140683";
#okay
# print "Set-Cookie:Wiki=rev_1_ie_3989_randkey_286140683";
#okay
#print "Set-Cookie:Wiki=rev&1&ie_3989_randkey_286140683";
#okay
#print "Set-Cookie:Wiki=rev&1&ie_3989_randkey_286140683";
return "Content-type: text/html\n\n";
}

Can PlusNet raise a ticket to the CCGI team?

I do have a work around that works.

regards and many thanks,

Doug

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Mon, 05 Jun 2017 20:01:22 GMT

Hi there must be another filter script in the web server that does not like id in the cookie. I modified tt2.pl to Set-Cookie and found it had to have "id".

I tried some different combinations and positions and capitalisations.

it breaks if it finds id, but not for idi,idd,Id,iD.iid,ib,ic,ie.

It works on apache on my raspberry Pi and has worked for years.

I have a working workaround, so this is now just for curiousity.

Another cookie is reported. TS0194eee0. My script does not add this.

HTTP_COOKIE=TS0194eee0=01a7d14f7faf7bea938c3ab082e4d9fd3eadc2caea13d8a27b3bbb37cebaf6dc93bcb086d3aa88c3e7599da28bab15614b4169c1381c0ae75fabac8cf727cb23fcfd8b3e21; Wiki=& ie&3989rev&1&3989&randkey&286140683

# PrintHeader
# Returns the magic line which tells WWW that we're an HTML document

#http://ccgi.dougrice.plus.com/cgi-bin/tt2.pl
sub PrintHeader {
#breakes
#print "Set-Cookie:Wiki=rev&1&id&3989&randkey&286140683\n";
#okay if id is changed to ib or ie
#print "Set-Cookie:Wiki=rev&1&ib&3989&randkey&286140683\n";
#okay if id is changed to ib or ie
#print "Set-Cookie:Wiki=rev&1&ie&3989&randkey&286140683\n";
#okay if id is changed to ib or ie
#print "Set-Cookie:Wiki=rev&1&ic&3989&randkey&286140683\n";

#okay
#print "Set-Cookie:Wiki=rev_1_ie_3989_randkey_286140683\n";
#okay
#print "Set-Cookie:Wiki=rev&1&ie_3989_randkey_286140683\n";
#okay
#print "Set-Cookie:Wiki=rev&1&ie_3989_randkey_286140683\n";

#breaks
#print "Set-Cookie:Wiki=rev&1&3989&randkey&286140683&id&3989\n";
#breaks
#print "Set-Cookie:Wiki=&id&3989rev&1&3989&randkey&286140683\n";
#okay id is set to ID
#print "Set-Cookie:Wiki=&ID&3989rev&1&3989&randkey&286140683\n";

#okay id is set to iD
#print "Set-Cookie:Wiki=&iD&3989rev&1&3989&randkey&286140683\n";
#okay id is set to iD
#print "Set-Cookie:Wiki=&Id&3989rev&1&3989&randkey&286140683\n";
#okay id is set to Id
#print "Set-Cookie:Wiki=&iid&3989rev&1&3989&randkey&286140683\n";
#okay id is set to idi
#print "Set-Cookie:Wiki=&idi&3989rev&1&3989&randkey&286140683\n";
#breaks if okay id is set to id
#print "Set-Cookie:Wiki=&idd&3989rev&1&3989&randkey&286140683\n";
#breaks id is set to 'id '
#print "Set-Cookie:Wiki=&id &3989rev&1&3989&randkey&286140683; Thu, 01 Jan 1970 00:00:01 GMT; Path=/\n";
#breaks id is set to ' id'
print "Set-Cookie:Wiki=& id&3989rev&1&3989&randkey&286140683; Thu, 01 Jan 1970 00:00:01 GMT; Path=/\n";
#okay id is set to ' ie'
print "Set-Cookie:Wiki=& ie&3989rev&1&3989&randkey&286140683; Thu, 01 Jan 1970 00:00:01 GMT; Path=/\n";

#set HTTP_COOKIE=TS0194eee0=01a7d14f7faf7bea938c3ab082e4d9fd3eadc2caea13d8a27b3bbb37cebaf6dc93bcb086d3aa88c3e7599da28bab15614b4169c1381c0ae75fabac8cf727cb23fcfd8b3e21; Wiki=& ie&3989rev&1&3989&randkey&286140683
return "Content-type: text/html\n\n";
}

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

dougrice — Mon, 05 Jun 2017 20:11:56 GMT

If I have id& at the start of the cookie it works! e.g

set HTTP_COOKIE=Wiki=id&3989rev&1&3989&randkey&286140683; TS0194eee0=01a7d14f7f22f5883bb444e95972bb6b672b7c05f0e1a2149131c5ee7273bec2622e7be9a05db116eba2c031b743dcae530343cab7

Have &id& in the cookie and it breaks!

I have a working workaround!

Re: My Perl cgi perl scrips have stopped working and get: http://ccgi.dougrice.plus.com/cgi-bin/wiki

spraxyt — Tue, 06 Jun 2017 16:16:18 GMT

I don't think that "TS" cookie is anything to do with the current scripts. I have a similarly named one and it's a session cookie for the whole site but I don't know what creates it. I have several "TS" cookies associated with different parts of my site, some for WordPress, some for phpMyAdmin, some for other apps.

I've confirmed that a cookie with all numeric values works for me as long as I don't have an "id" parameter. I created the cookie using standard CGI.pm method calls. I've tried to get that to put 'id' first but the software won't play ball.