Web site owners use the / robots. txt file to give instructions about their site to web robots; this is The " Disallow: /" tells the robot that it should not visit any pages on the site. The short answer: in the top-level directory of your web server. Can I block just bad robots? · tag · Frequently Asked Questions. Is there a line of code in robots. txt that will disallow a folder and everything beneath it? My example that I am thinking of would be the star. A given robots. txt file applies only to the exact subdomain it was loaded from. In other words, the following robots. txt file. Remember to use all lower case for the filename: For instance, to block an URLs that end. We will not rent or sell your email address. There are two historical descriptions:. It finds the file and reads it. Sign up or log in to customize your list.