robots.txt
robots.txt is a plain-text file that a website places at the root of its address space (for example to tell automated clients, known as crawlers or robots, which parts of the site they are asked not to fetch.
Explore AI Policy & Regulation through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of AI Policy & Regulation.
Showing 1-1 of 1 article
robots.txt is a plain-text file that a website places at the root of its address space (for example to tell automated clients, known as crawlers or robots, which parts of the site they are asked not to fetch.