# Revision: 18 # Modtime: 24/04/2010 11:08 # # Define BANNED web Robots/Spiders # User-agent: 192.comAgent User-agent: DTAAgent User-agent: ia_archiver User-agent: linksmanager_bot User-agent: MSIECrawler User-agent: NetMechanic User-agent: Stress-Agent User-agent: SurveyBot User-agent: WebVulnCrawl Disallow: / # # Make changes for all other web Robots/Spiders # User-agent: * # # Ignore FrontPage and system generated files/directories # Disallow: _ Disallow: /aspnet_client/ Disallow: /bin/ # # My directories to ignore # Disallow: /Cal/ Disallow: /Classes/ Disallow: /Common/Error/ Disallow: /Common/Legal/ Disallow: /Email/ Disallow: /Img/ Disallow: /JS/ Disallow: /Payment/ Disallow: /SMS/ Disallow: /Style/ Sitemap: /SiteMap.xml