# This file politely asks web robots to cooperate. # # Please do not edit or remove this file without permission # of the wiki webmaster. Thank you. #User-agent: Googlebot #Disallow: /bin/view/Quality #Disallow: /bin/view/Dynamic #Crawl-delay: 60 #User-agent: gsa-crawler #Disallow: /cgi-bin/ #Disallow: /cgi-internal/ User-agent: * Disallow: / #Crawl-delay (in seconds) is a Slurp-specific command and is not universal.