How Search Engine Look For New Web Page
To properly optimize web page content for the search engines you need to understand how the search engines read web pages. When the Search Engine Arrives to the Website When the search engine arrives to the website in looks in the root (main) folder of the site for a file called robots.txt In the robots.txt file it looks for what directories and files it is allowed to look at and index. Some search engine spiders (web crawlers, robots) ignore these instructions. Web Page Head Section Once a web page is found by the search engine spider (robot, web crawler) it takes a look at the head section (the tags between the and tags) of the web page for: 1. The title of the page 2. The keyword and description meta tags 3. The robots meta tag. Some use the robots meta tag to override the instructions in the robots.txt file or if they cannot create a robots.txt file the instructions for the search engine spiders (robots, web crawlers) are placed here. If there is no robots.txt file