A good rule of thumb when building a website is: keep the spiders happy and you will be happy, too. With all jokes and pop culture title references put aside, it is vital that you understand the basics of how search engine spiders work and what you can do to maximize their efficiency at reading and indexing your website. An ill-formatted page in the eyes of a search engine can lead to terrible PR, while irrelevant content can have the same impact on the potential of the site in question. We will review some of the basic practices of search engine spiders along with some of the mistakes you should avoid.

How Search Engine Spiders Work

Search engine spiders are specific pieces of software that are used by search engines predominantly to collect information about a website along with any page data that is provided in the source code. Search engine spiders do not look at a web page in the same way that you or I do; instead, the search engine spider dissects the information on the website into two distinct categories: plain text and HTML. It then evaluates each based on formatting and validity and makes a decision on how to classify the website. This is why concepts such as headers and titles on web pages are so important. With that covered, let’s move on to some of the biggest no-nos when it comes to search engine spiders reading your site.

Irrelevant Content

Through a series of algorithms and analysis, search engine spiders will determine what specific category your site caters to and will assign it to search rankings accordingly. Many people – in the old days of the internet – would cram their sites with as many keywords as possible in order to garner more traffic. Today, however, that type of practice can kill your website and lead to it being a permanent failure. Using keywords and content is vital to your site’s success, but you should strive to have your site’s content as uniform as possible. Separate websites and blogs should be used for various categories, if you are trying to attract people from various walks of life.

Redirect Misdirect

Some people try to get away with using redirects as a means to stuff their site full of contents. Any given site may use a redirect as a way to link to a page with loads of keyword and content stuffing which will then redirect to the page that is meant for human viewing. In case you didn’t read the above paragraph, keyword and content stuffing no longer works like it used to work; you will end up with massive penalties on your PR.

See Your Site Through A Spider’s Eyes

If you want to know what your website looks like to a spider, then use Pingler’s Spider Viewer as a way to inspect your site on a different level. This will show you the pure text of what is on your website, which in this displayed form can help you review each page and find any keyword redundancies that may be more harmful than helpful.