{"id":590,"date":"2012-02-13T09:00:36","date_gmt":"2012-02-13T09:00:36","guid":{"rendered":"http:\/\/pingler.com\/blog\/?p=590"},"modified":"2014-08-19T21:33:15","modified_gmt":"2014-08-19T21:33:15","slug":"what-is-a-search-engine-spider","status":"publish","type":"post","link":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/","title":{"rendered":"What is a Search Engine Spider?"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-medium wp-image-5842\" src=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s-300x200.jpg\" alt=\"Indexes\" width=\"300\" height=\"200\" srcset=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s-300x200.jpg 300w, https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg 450w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/>We\u2019ve spoken a lot on this blog about how it is important for your website to be fully searchable by the search engine spiders so now it\u2019s time to get some in depth knowledge about what a search engine spider actually is.\u00a0 A spider isn\u2019t a cute little arachnid like you may have been led to believe; rather it is an automated software agent that seeks out the content on each and every one of your webpages.\u00a0 The findings of the spider are then relayed back to the search engines enabling them to correctly position your webpage in the search engine results.<\/p>\n<p><strong>How do Spiders Work?<\/strong><\/p>\n<p>Before a search engine can correctly bring up results for your search terms it needs to know that webpages for your search terms exist and where to find them.\u00a0 This is the job of the search engine spiders that literally crawl through webpages looking for keywords and indexing them.\u00a0 As a spider moves through websites looking for keywords the process is called web crawling (yes this is one of the disadvantages of calling the internet the World Wide Web, we end up with a lot of spider centred names).\u00a0 In order for the search engine results to be beneficial to the searcher, the spiders must crawl through millions of pages and index them correctly.<\/p>\n<p><strong>Where Does a Spider\u2019s Job Begin?<\/strong><\/p>\n<p>A search engine spider will generally begin its search on a popular website or on a heavily used server.\u00a0 As it begins to crawl through the webpages and index them, the spider will continue on to follow any links that are found on the webpage too.\u00a0 By following links the search engine spiders find themselves indexing large parts of the internet, and in particular the most popular websites.\u00a0 When Google was first invented, inventors Lawrence Page and Sergey Brin announced that when operating at peak performance the spiders could crawl up to 100 pages per second.<\/p>\n<p><strong>What do the Spider\u2019s Look For?<\/strong><\/p>\n<p>The answer to this question varies depending on the search engine.\u00a0 For example the Google spiders take into account the titles, subtitles, meta tags and keywords on a page whilst ignoring words like \u201ca, an, of, in\u201d etc.\u00a0 Other spiders\u2019 crawl the webpages differently \u2013 some may even still pay attention to keyword tags it all depends on the search engine they are working for.\u00a0 Additionally the search engines are continually updating the way they regulate their searches and so whilst the Google spiders may take certain things into account now, this may all change in a few months time.<\/p>\n<p><strong>A Spider\u2019s Job is Never Finished<\/strong><\/p>\n<p>With the World Wide Web changing on a daily basis, a spider\u2019s job is never finished.\u00a0 After indexing webpages, the search engines will use a variety of different algorithms to determine which webpage is the most useful for the searcher when they enter a search term.\u00a0 Each page is given a different amount of \u201cweight\u201d by the search engine and so when a search is conducted by a search engine user they should, in theory, be presented with a list of the most useful webpages for their search term.\u00a0 As the amount of webpages on the internet increases on a daily basis, search engine spiders need to continue to crawl the web (both new and existing pages) in order to ensure they are still presenting the most relevant search results to users.<\/p>\n<p>If you want to see how your website looks to a search engine you might want to check out the <a href=\"http:\/\/pingler.com\/seo-tools\/spider-viewer\/\" data-wpel-link=\"external\">Pingler Spider Viewer Tool<\/a>.\u00a0 Whilst the results aren\u2019t intended to be seen by human eyes it can help you to determine what the spiders see and whether they can crawl your page effectively.<\/p>\n<p><i><html><br \/>\n\t<head><br \/>\n\t\t<title><\/title><br \/>\n\t<\/head><br \/>\n\t<body><\/p>\n<p style=\"text-align: center;\">\n\t\t\t<a href=\"http:\/\/www.pingler.com\/account\/register\/\" data-wpel-link=\"external\"><img decoding=\"async\" alt=\"\" src=\"http:\/\/www.pingler.com\/affiliates\/banners\/Pingler-468x60-01A.gif\" style=\"width: 468px; height: 60px;\" \/><\/a><\/p>\n<p>\t<\/body><br \/>\n<\/html><\/i><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We\u2019ve spoken a lot on this blog about how it is important for your website to be fully searchable by the search engine spiders so now it\u2019s time to get some in depth knowledge about what a search engine spider actually is.\u00a0 A spider isn\u2019t a cute little arachnid like you may have been led [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":5842,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16,25],"tags":[175],"class_list":["post-590","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-google","category-plugins-and-tools","tag-search-engine-spider"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.1.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>What is a Search Engine Spider? - Pingler Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is a Search Engine Spider? - Pingler Blog\" \/>\n<meta property=\"og:description\" content=\"We\u2019ve spoken a lot on this blog about how it is important for your website to be fully searchable by the search engine spiders so now it\u2019s time to get some in depth knowledge about what a search engine spider actually is.\u00a0 A spider isn\u2019t a cute little arachnid like you may have been led [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/\" \/>\n<meta property=\"og:site_name\" content=\"Pingler Blog\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/facebook.com\/pinglerdotcom\" \/>\n<meta property=\"article:published_time\" content=\"2012-02-13T09:00:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-08-19T21:33:15+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"450\" \/>\n\t<meta property=\"og:image:height\" content=\"300\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Pingler\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Pingler\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/\"},\"author\":{\"name\":\"Pingler\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9\"},\"headline\":\"What is a Search Engine Spider?\",\"datePublished\":\"2012-02-13T09:00:36+00:00\",\"dateModified\":\"2014-08-19T21:33:15+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/\"},\"wordCount\":631,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/pingler.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg\",\"keywords\":[\"Search Engine Spider\"],\"articleSection\":[\"Google\",\"Plugins and Tools\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/\",\"url\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/\",\"name\":\"What is a Search Engine Spider? - Pingler Blog\",\"isPartOf\":{\"@id\":\"https:\/\/pingler.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg\",\"datePublished\":\"2012-02-13T09:00:36+00:00\",\"dateModified\":\"2014-08-19T21:33:15+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#primaryimage\",\"url\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg\",\"contentUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg\",\"width\":450,\"height\":300},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/pingler.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is a Search Engine Spider?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/pingler.com\/blog\/#website\",\"url\":\"https:\/\/pingler.com\/blog\/\",\"name\":\"Pingler.com\",\"description\":\"Ping\",\"publisher\":{\"@id\":\"https:\/\/pingler.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/pingler.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/pingler.com\/blog\/#organization\",\"name\":\"Pingler.com\",\"url\":\"https:\/\/pingler.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg\",\"contentUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg\",\"width\":300,\"height\":160,\"caption\":\"Pingler.com\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"http:\/\/facebook.com\/pinglerdotcom\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9\",\"name\":\"Pingler\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g\",\"caption\":\"Pingler\"},\"sameAs\":[\"http:\/\/www.pingler.com\"],\"url\":\"https:\/\/pingler.com\/blog\/author\/pingler\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What is a Search Engine Spider? - Pingler Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/","og_locale":"en_US","og_type":"article","og_title":"What is a Search Engine Spider? - Pingler Blog","og_description":"We\u2019ve spoken a lot on this blog about how it is important for your website to be fully searchable by the search engine spiders so now it\u2019s time to get some in depth knowledge about what a search engine spider actually is.\u00a0 A spider isn\u2019t a cute little arachnid like you may have been led [&hellip;]","og_url":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/","og_site_name":"Pingler Blog","article_publisher":"http:\/\/facebook.com\/pinglerdotcom","article_published_time":"2012-02-13T09:00:36+00:00","article_modified_time":"2014-08-19T21:33:15+00:00","og_image":[{"width":450,"height":300,"url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg","type":"image\/jpeg"}],"author":"Pingler","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Pingler","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#article","isPartOf":{"@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/"},"author":{"name":"Pingler","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9"},"headline":"What is a Search Engine Spider?","datePublished":"2012-02-13T09:00:36+00:00","dateModified":"2014-08-19T21:33:15+00:00","mainEntityOfPage":{"@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/"},"wordCount":631,"commentCount":0,"publisher":{"@id":"https:\/\/pingler.com\/blog\/#organization"},"image":{"@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#primaryimage"},"thumbnailUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg","keywords":["Search Engine Spider"],"articleSection":["Google","Plugins and Tools"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/","url":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/","name":"What is a Search Engine Spider? - Pingler Blog","isPartOf":{"@id":"https:\/\/pingler.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#primaryimage"},"image":{"@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#primaryimage"},"thumbnailUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg","datePublished":"2012-02-13T09:00:36+00:00","dateModified":"2014-08-19T21:33:15+00:00","breadcrumb":{"@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#primaryimage","url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg","contentUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/28653592_s.jpg","width":450,"height":300},{"@type":"BreadcrumbList","@id":"https:\/\/pingler.com\/blog\/what-is-a-search-engine-spider\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/pingler.com\/blog\/"},{"@type":"ListItem","position":2,"name":"What is a Search Engine Spider?"}]},{"@type":"WebSite","@id":"https:\/\/pingler.com\/blog\/#website","url":"https:\/\/pingler.com\/blog\/","name":"Pingler.com","description":"Ping","publisher":{"@id":"https:\/\/pingler.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/pingler.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/pingler.com\/blog\/#organization","name":"Pingler.com","url":"https:\/\/pingler.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg","contentUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg","width":300,"height":160,"caption":"Pingler.com"},"image":{"@id":"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["http:\/\/facebook.com\/pinglerdotcom"]},{"@type":"Person","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9","name":"Pingler","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g","caption":"Pingler"},"sameAs":["http:\/\/www.pingler.com"],"url":"https:\/\/pingler.com\/blog\/author\/pingler\/"}]}},"_links":{"self":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/590","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/comments?post=590"}],"version-history":[{"count":5,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/590\/revisions"}],"predecessor-version":[{"id":5843,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/590\/revisions\/5843"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/media\/5842"}],"wp:attachment":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/media?parent=590"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/categories?post=590"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/tags?post=590"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}