Trend Micro bots
Is anyone else having trouble with Trend Micro's bots? I don't really want to have to block them, because they're probably well intentioned, but I think I'm going to have to because they're just so stupid. 
These are the bots that follow you round if you're running a Trend Micro safe-browsing product. Any time you visit a site, a couple of minutes later the Trend Micro bots come trotting along in your exact footsteps, like a pair of little dogs, presumably baiting sites to push them malware. Cute.
One problem I have with them is that they have your query string, so one of my search interfaces is getting hit two or three times for each real search a Trend Micro user conducts. (Incidentally, I can't use POST, because users need to be able to link to their search results.) Search stats are used in development and the Trend Micro bots are skewing them.
Another problem I have with them is that they hit parts of my sites to which they have no access, so they're just getting a login form returned, and that just wastes bandwidth. They don't respect robots.txt.
Another problem is one of informed consent. A Trend Micro user with whom I discussed this said she had no idea she was being followed round like this and wasn't entirely comfortable with the idea. When I asked Trend Micro if one could switch this feature off, they replied
followed, a few minutes later, by
So you have to opt out at installation, without really knowing what information you would be sharing with Trend Micro. The user mentioned above had had the software pre-installed on her new PC as a free trial. Looking at the blurb, all she could find was
which she didn't think conveyed the fact that her browsing history was routinely being shared with robots in Japan and the US.
At this point I should probably plug ZoneAlarm and Little Snitch (outbound firewalls).
I also asked Trend Micro why their bots didn't respect robots.txt and was told
Sure look like robots to me. They present as "MSIE 6.0; Windows NT 5.1", i.e. stupidly vulnerable - presumably to solicit attacks. But malware sites would choose old-fashioned methods to attack that sort of profile (why waste effort?), so all this trawling round is presumably missing more recent techniques. It's probably well-intentioned, but I wouldn't call it smart.
Consign to bad-bot category, I guess.
Gabe
These are the bots that follow you round if you're running a Trend Micro safe-browsing product. Any time you visit a site, a couple of minutes later the Trend Micro bots come trotting along in your exact footsteps, like a pair of little dogs, presumably baiting sites to push them malware. Cute.
One problem I have with them is that they have your query string, so one of my search interfaces is getting hit two or three times for each real search a Trend Micro user conducts. (Incidentally, I can't use POST, because users need to be able to link to their search results.) Search stats are used in development and the Trend Micro bots are skewing them.
Another problem I have with them is that they hit parts of my sites to which they have no access, so they're just getting a login form returned, and that just wastes bandwidth. They don't respect robots.txt.
Another problem is one of informed consent. A Trend Micro user with whom I discussed this said she had no idea she was being followed round like this and wasn't entirely comfortable with the idea. When I asked Trend Micro if one could switch this feature off, they replied
Quote You can disable Smart Surfing
followed, a few minutes later, by
Quote Sorry for my mistake. There's an option during the installation. You can tick off the box "share information with Trend Micro".
So you have to opt out at installation, without really knowing what information you would be sharing with Trend Micro. The user mentioned above had had the software pre-installed on her new PC as a free trial. Looking at the blurb, all she could find was
Quote Smart Feedback is a type of mechanism that detects the latest threats and reports them to the Trend Micro Smart Protection Network.
which she didn't think conveyed the fact that her browsing history was routinely being shared with robots in Japan and the US.
At this point I should probably plug ZoneAlarm and Little Snitch (outbound firewalls).
I also asked Trend Micro why their bots didn't respect robots.txt and was told
Quote This is not a robot. This is the way the software works.
Sure look like robots to me. They present as "MSIE 6.0; Windows NT 5.1", i.e. stupidly vulnerable - presumably to solicit attacks. But malware sites would choose old-fashioned methods to attack that sort of profile (why waste effort?), so all this trawling round is presumably missing more recent techniques. It's probably well-intentioned, but I wouldn't call it smart.
Consign to bad-bot category, I guess.
Gabe