This post is a translation of the material What are crawl errors? , originally published on the Yoast blog. The content was authorized for publication on the HostGator blog.
Crawl errors occur when a search engine attempts to access a page on your website, but fails. First, let’s clarify a little more about crawling.
Crawling is the process by which a search japan cell phone number list engine attempts to visit all the pages on your website via a bot. A search engine bot finds a link to your site and starts searching all your public pages. It crawls the pages, indexes all the content for use on Google, and adds all the links from those pages to the pages it still needs to crawl.
Your main goal as a site owner is to ensure that every page on your website can be accessed by the search engine bot . Failure in this process results in what we call crawl errors.
Your goal is to ensure that every link on your website leads to a real page. This can be done using a 301 redirect , but the page at the end of that link must always return a 200 OK server response.
Site errors – You don’t want these errors as they mean your entire site can’t be crawled.
URL Errors – You don’t want these errors either, but since they refer to a specific URL in error, they are easier to maintain and fix.
Let's go into more detail on this.