These Errors and Mistakes Could Be Blocking Google from Indexing You
The only way that your website can be discovered by the masses – short of a massive paid marketing campaign – is through search engines. When someone wants to find information or products that relate to what you offer, they can enter a query and find you in no time at all. The only way to be listed in search engines, however, is to have your site consistently indexed by Google and other search engines. Since the dawn of the internet, search engines have used a pretty rudimentary process involving search engine “crawlers” and “bots” to find all the content on the web. If these crawlers and bots cannot find your pages, then they won’t be discoverable through search engines. We’ll talk about some errors below that could be preventing search engines from finding your pages.
Check Your Robots File
Every website (should) have a file called robots.txt in its directory. This file can be edited to allow only certain directories to be indexed, or to disallow all directories from being indexed. Unfortunately, errors in your robots.txt file can be rather common if you have ever edited it in the past. This in turn may be preventing your pages from being indexed properly. The command for excluding directories will display “user-agent: *” followed by “Disallow:” and then the directory being blocked. Check your robots.txt file to ensure that no directories you need listed on search engines are classified this way. The exclusion of pages via this file does not make them invisible, but instead prevents search engines from finding/listing them.
Your Pages Have Passwords
If some of your pages have password protection, then you may be unaware that these sites will not be pinging to Google. Some people have created password-protected pages for show or as a way to test the feature out; unfortunately for the owners, this means that no one will be able to find it. If the pages need to be found by search engines, then we highly recommend removing password protection or instead, provide a secondary link or landing page that will explain what the page is about, subsequently linking to it from there.