Posted on


The Robots.txt protocol, also known as the “Robot Exclusion Standard,” is designed to prevent robots from accessing certain parts of a website. This is a security or privacy measure similar to putting a “No Trespassing” sign on your door.

This protocol is used by website administrators when there are sections or files that the rest of the world does not want access to. This may include employee lists or files circulating internally. For example, the White House website uses the robots.txt file to block requests for the Vice President’s speeches, the First Lady’s photo shoots, and the profiles of panic victims.

How does the protocol work? Lists files that should not be scanned and places them in the top directory of the site. Robot.txt was created in June 1994 by consensus of members of the Robots mailing list ( There is no official standards body or RFC for the protocol, so it is difficult to issue regulations or require compliance with the protocol. This is because the file is considered purely a recommendation and does not provide an absolute guarantee that its contents will not be read.

This is because robot.txt requires the cooperation of a web spider and even a reader, since everything uploaded to the Internet becomes publicly available. You don’t stop them from accessing these websites, you just make it harder for them to access them. But it doesn’t take long before they ignore these instructions. Hackers can also easily break into files and extract information. As a general rule, if it’s such a sensitive topic, it shouldn’t be covered on your website.