Dignity nobility quality point out highness standing course caste position superiority priority trend distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robot concerning which web pages are usually not to generally be crawled. As a search engine crawler https://www.youtube.com/watch?v=OeFqbAORPUg