Like, I know a lot of webmasters would prefer their sites be indexed by search engines, but doesn't having web crawlers around give anyone else the creeps? I'm always worried about what personally-identifiable information they may be collecting, or the possibility that they're harvesting e-mails for spam-lists. I don't like the idea of advertising agencies collecting information to profile my browsing habits with. Maybe I'm just paranoid.

Of course, if they're not actually wanted and they're just wasting the site's bandwidth with their requests, Kevin could always consider a crawler trap to throw them off. It's something like a fake page that the bot would get "stuck" in.

Or robots.txt...

It is good for bots that respect it.

It is not totally paranoid. Forums are excellent to collect e-mails. And I don't like the idea of crawlers seeing my profile, too.

Since most of the calc stuff is viewable by public, we don't need web crawlers, right?

Me too.  I don't like the idea of bots could be going through our emails (or more) to spam us.  ;.;

Don't worry about it. I get about 500-1000 spam a month in my gmail folder and still put my email address everywhere. As long as you don't put your home address/phone number anywhere online (except maybe Ebay/Paypal, as long as you use a different email address) you're safe. You need to be careful to warez sites, though.

Also, in case anyone didn't notice, every member contact info on Omnimaga is invisible to guests and members have the ability to hide their email address from everyone except Eeems and myself too. So basically when bots crawl your profile, all they'll find is your website URL, your location and your age, if you made them public by your own.

Also bots cannot see member IP addresses either. The only way they can know is if you connect to an IRC channel they have access to, but even then it's not like someone's gonna find you and hunt you down.

As for the bots that displays on home page, they are for search page indexing. They won't cause harm to the site. In fact, these bots have been present on Omnimaga since half a decade. The only reason why they showed as guests before is because Invisionfree and SMF1.1.x doesn't recognize them by default. A mod was installed on this SMF install to show them. The only bots I dislike are spambots, in which case I ban them.

Btw, on SMF, sending e-mails to other members is impossible by members. As for spamming via PMs, bots cannot do so either because sending PMs on Omnimaga requires you to have 5 posts minimum. On the old site before we had such protections a bot sent nigerian scams to 40 members or so.

In conclusion, don't worry about these bots. If they were that harmful, no one would ever make their e-mail address public on web forums, same for their AIM/Yahoo/MSN addresses anyway and no one would ever use IRC at all.

Yes, for some reason all those bots creep me out too. Remember, Big brother is watching...