Author Topic: Web crawlers  (Read 3885 times)

0 Members and 1 Guest are viewing this topic.

Offline Zera

  • LV8 Addict (Next: 1000)
  • ********
  • Posts: 737
  • Rating: +82/-7
  • Monochrome Artisan
    • View Profile
Web crawlers
« on: February 21, 2010, 03:45:56 pm »


Like, I know a lot of webmasters would prefer their sites be indexed by search engines, but doesn't having web crawlers around give anyone else the creeps? I'm always worried about what personally-identifiable information they may be collecting, or the possibility that they're harvesting e-mails for spam-lists. I don't like the idea of advertising agencies collecting information to profile my browsing habits with. Maybe I'm just paranoid.

Of course, if they're not actually wanted and they're just wasting the site's bandwidth with their requests, Kevin could always consider a crawler trap to throw them off. It's something like a fake page that the bot would get "stuck" in.

Offline jbr

  • LV1 Newcomer (Next: 20)
  • *
  • Posts: 11
  • Rating: +1/-1
    • View Profile
Re: Web crawlers
« Reply #1 on: February 21, 2010, 04:03:08 pm »
Or robots.txt...

Offline Galandros

  • LV9 Veteran (Next: 1337)
  • *********
  • Posts: 1140
  • Rating: +42/-10
    • View Profile
Re: Web crawlers
« Reply #2 on: February 21, 2010, 04:23:53 pm »
Or robots.txt...
It is good for bots that respect it.

It is not totally paranoid. Forums are excellent to collect e-mails. And I don't like the idea of crawlers seeing my profile, too.

Since most of the calc stuff is viewable by public, we don't need web crawlers, right?
Hobbing in calculator projects.

Offline Geekboy1011

  • The Oneironaut
  • Donator
  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2031
  • Rating: +119/-2
  • Dream that Awakening dream
    • View Profile
Re: Web crawlers
« Reply #3 on: February 21, 2010, 05:17:14 pm »
agreed i dont like the idea of tha eaither as i get enough spam with colledges atm so i defietily would like somthing against bots

Offline ztrumpet

  • The Rarely Active One
  • CoT Emeritus
  • LV13 Extreme Addict (Next: 9001)
  • *
  • Posts: 5712
  • Rating: +364/-4
  • If you see this, send me a PM. Just for fun.
    • View Profile
Re: Web crawlers
« Reply #4 on: February 21, 2010, 08:15:53 pm »
Me too.  I don't like the idea of bots could be going through our emails (or more) to spam us.  ;.;

Offline JoeyBelgier

  • LV7 Elite (Next: 700)
  • *******
  • Posts: 545
  • Rating: +25/-21
  • Joey
    • View Profile
Re: Web crawlers
« Reply #5 on: February 21, 2010, 08:35:34 pm »
I .. I'm one of them :'[

Offline DJ Omnimaga

  • Clacualters are teh gr33t
  • CoT Emeritus
  • LV15 Omnimagician (Next: --)
  • *
  • Posts: 55943
  • Rating: +3154/-232
  • CodeWalrus founder & retired Omnimaga founder
    • View Profile
    • Dream of Omnimaga Music
Re: Web crawlers
« Reply #6 on: February 21, 2010, 09:21:55 pm »
Don't worry about it. I get about 500-1000 spam a month in my gmail folder and still put my email address everywhere. As long as you don't put your home address/phone number anywhere online (except maybe Ebay/Paypal, as long as you use a different email address) you're safe. You need to be careful to warez sites, though.

Also, in case anyone didn't notice, every member contact info on Omnimaga is invisible to guests and members have the ability to hide their email address from everyone except Eeems and myself too. So basically when bots crawl your profile, all they'll find is your website URL, your location and your age, if you made them public by your own.

Also bots cannot see member IP addresses either. The only way they can know is if you connect to an IRC channel they have access to, but even then it's not like someone's gonna find you and hunt you down.

As for the bots that displays on home page, they are for search page indexing. They won't cause harm to the site. In fact, these bots have been present on Omnimaga since half a decade. The only reason why they showed as guests before is because Invisionfree and SMF1.1.x doesn't recognize them by default. A mod was installed on this SMF install to show them. The only bots I dislike are spambots, in which case I ban them.

Btw, on SMF, sending e-mails to other members is impossible by members. As for spamming via PMs, bots cannot do so either because sending PMs on Omnimaga requires you to have 5 posts minimum. On the old site before we had such protections a bot sent nigerian scams to 40 members or so.

In conclusion, don't worry about these bots. If they were that harmful, no one would ever make their e-mail address public on web forums, same for their AIM/Yahoo/MSN addresses anyway and no one would ever use IRC at all.
« Last Edit: February 21, 2010, 09:25:25 pm by DJ Omnimaga »

Offline tifreak

  • LV11 Super Veteran (Next: 3000)
  • ***********
  • Posts: 2708
  • Rating: +82/-3
  • My Kung Fu IS strong...
    • View Profile
    • TI-Freakware
Re: Web crawlers
« Reply #7 on: February 21, 2010, 10:18:51 pm »
I am more concerned they may learn the location of the Rebel's secret base. Then we WILL be in trouble. D=
Projects: AOD Series: 75% | FFME: 80% | Pokemon: 18% | RPGSK: 60% | Star Trek: 70% | Star Trek 83+: 40% | TI-City: 5%

Offline TIfanx1999

  • ಠ_ಠ ( ͡° ͜ʖ ͡°)
  • CoT Emeritus
  • LV13 Extreme Addict (Next: 9001)
  • *
  • Posts: 6173
  • Rating: +191/-9
    • View Profile
Re: Web crawlers
« Reply #8 on: February 22, 2010, 12:03:23 am »
Yes, for some reason all those bots creep me out too. Remember, Big brother is watching...