0 votes
in Web Hosting by
What is robots.txt file?

3 Answers

0 votes
by

Robots.txt is a text file within your website’s top-level directory that instructs search engines how to crawl your pages.

0 votes
by

robots.txt is a text file that instructs web robots (search engine robots) about how to crawl pages on their website. It is used to manage crawler traffic to the website. 

0 votes
by
Robots.txt is a text file. It is through this file. It gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain.

Related questions

0 votes
asked Mar 7, 2020 in Spark Sql by rahuljain1
+1 vote
asked May 11, 2021 in Linux by rajeshsharma
...