Re: please use robots.txt for your gopher apps

[Cameron Kaiser <spectre@floodgap.com>]

> > 1) If there are parts that you don't want crawled, then use a robots.txt
> >    to define those,
> >
> > That's what Cameron is asking.
> 
> > I'm considering a policy requirement that sites to be accepted to the
> > new servers page must have some sort of robots.txt
> 
> Then I misread it I guess.

I've got a middle ground asking for this to be done voluntarily on the new
list, and we'll see how well that works.

-- 
------------------------------------ personal: http://www.cameronkaiser.com/ --
  Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser@floodgap.com
-- Seen on hand dryer: "Push button for a message from your congressman." -----