{"id":675,"date":"2012-03-26T09:00:36","date_gmt":"2012-03-26T09:00:36","guid":{"rendered":"http:\/\/pingler.com\/blog\/?p=675"},"modified":"2014-08-19T19:13:26","modified_gmt":"2014-08-19T19:13:26","slug":"the-basics-of-search-engine-spiders","status":"publish","type":"post","link":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/","title":{"rendered":"The Basics of Search Engine Spiders"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-medium wp-image-5792\" src=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s-300x200.jpg\" alt=\"Search Engine Web\" width=\"300\" height=\"200\" srcset=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s-300x200.jpg 300w, https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg 450w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/>Search engine spiders, also known as crawlers, consist of scripts or programs that when used by search engines, go out and collect information about publicly-accessible web pages. \u00a0These spiders are solely responsible for collecting information \u2013 including content, keywords and meta data \u2013 and use this information to index your site and rank it against other web sites for any given search query. \u00a0There are multiple types of spiders that are designed to scan and engage in specific functions and acts, each contributing to how any given search engine views your site. \u00a0By understanding what these spiders are doing when they happen upon your website, you will be better equipped to know how to use these spiders to your advantage.<\/p>\n<p><strong>Selection Spiders<\/strong><\/p>\n<p>Selection spiders are the most basic form of search engine crawler and are typically what we think of when the word &#8216;search engine spider&#8217; comes to mind. \u00a0As these spiders crawl the web, they are primarily looking for web pages that have not been indexed yet and from there, proceed to add these sites&#8217; data and information to the specified search engine. \u00a0These types of spiders will also look for any major changes to already-indexed pages and may apply the update, depending on the rank of the page and the exact commands of the spider.<\/p>\n<p><strong>Re-Visitation Spiders<\/strong><\/p>\n<p>This form of spider is tasked with the exclusive job of doing what some selection spiders do: visiting already-indexed pages to determine if any major updates or changes need to be applied to the search algorithms. \u00a0Re-visitation spiders and crawlers are predominantly useful for larger sites and blogs that have constant content being added and need continuous updates in order for that content to be viewable by users of search engines. \u00a0Re-visitation spiders will check any given site at a frequency that is calculated based on factors pertaining both to the specific search engine and the site itself.<\/p>\n<p><strong>Polite Spiders<\/strong><\/p>\n<p>These form of spiders work in many of the same ways as other spiders do, with the main difference being that the spider alternates data gathering between multiple websites in order to prevent excessive bandwidth consumption on behalf of the bot and to prevent the bot from crashing due to certain scripts being ran on particular sites. \u00a0\u201cImpolite\u201d bots will often ignore specifications made by the site (robots.txt file is one example) and will end up wasting bandwidth or crashing the site (the bots can also be crashed by ignoring the commands of the site and being tricked into a \u201cspider trap\u201d). \u00a0Polite bots gain their name from being considerate when it comes to bandwidth and an individual website&#8217;s specifications.<\/p>\n<p><strong>What Are Spiders Seeing?<\/strong><\/p>\n<p>If you are curious as to what data search engine spiders are collecting about your website, then you should try out <a href=\"..\/..\/seo-tools\/site-spider-viewer\/\" data-wpel-link=\"internal\">Pingler&#8217;s Site Spider Viewer<\/a>, which allows you to insert the URL of any primary, secondary or tertiary webpage and see the same information that spiders are collecting about your website. \u00a0This tool is perfect for anyone who wants to see or revise the information being collected about their website.<\/p>\n<p><i><html><br \/>\n\t<head><br \/>\n\t\t<title><\/title><br \/>\n\t<\/head><br \/>\n\t<body><\/p>\n<p style=\"text-align: center;\">\n\t\t\t<a href=\"http:\/\/www.pingler.com\/account\/register\/\" data-wpel-link=\"external\"><img decoding=\"async\" alt=\"\" src=\"http:\/\/www.pingler.com\/affiliates\/banners\/Pingler-468x60-01A.gif\" style=\"width: 468px; height: 60px;\" \/><\/a><\/p>\n<p>\t<\/body><br \/>\n<\/html><\/i><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Search engine spiders, also known as crawlers, consist of scripts or programs that when used by search engines, go out and collect information about publicly-accessible web pages. \u00a0These spiders are solely responsible for collecting information \u2013 including content, keywords and meta data \u2013 and use this information to index your site and rank it against [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":5792,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[135,25,4],"tags":[175],"class_list":["post-675","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-how-to-use-pingler","category-plugins-and-tools","category-seo","tag-search-engine-spider"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.1.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>The Basics of Search Engine Spiders - Pingler Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The Basics of Search Engine Spiders - Pingler Blog\" \/>\n<meta property=\"og:description\" content=\"Search engine spiders, also known as crawlers, consist of scripts or programs that when used by search engines, go out and collect information about publicly-accessible web pages. \u00a0These spiders are solely responsible for collecting information \u2013 including content, keywords and meta data \u2013 and use this information to index your site and rank it against [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/\" \/>\n<meta property=\"og:site_name\" content=\"Pingler Blog\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/facebook.com\/pinglerdotcom\" \/>\n<meta property=\"article:published_time\" content=\"2012-03-26T09:00:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-08-19T19:13:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"450\" \/>\n\t<meta property=\"og:image:height\" content=\"300\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Pingler\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Pingler\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/\"},\"author\":{\"name\":\"Pingler\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9\"},\"headline\":\"The Basics of Search Engine Spiders\",\"datePublished\":\"2012-03-26T09:00:36+00:00\",\"dateModified\":\"2014-08-19T19:13:26+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/\"},\"wordCount\":518,\"commentCount\":5,\"publisher\":{\"@id\":\"https:\/\/pingler.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg\",\"keywords\":[\"Search Engine Spider\"],\"articleSection\":[\"How To Use Pingler\",\"Plugins and Tools\",\"SEO\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/\",\"url\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/\",\"name\":\"The Basics of Search Engine Spiders - Pingler Blog\",\"isPartOf\":{\"@id\":\"https:\/\/pingler.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg\",\"datePublished\":\"2012-03-26T09:00:36+00:00\",\"dateModified\":\"2014-08-19T19:13:26+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#primaryimage\",\"url\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg\",\"contentUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg\",\"width\":450,\"height\":300},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/pingler.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The Basics of Search Engine Spiders\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/pingler.com\/blog\/#website\",\"url\":\"https:\/\/pingler.com\/blog\/\",\"name\":\"Pingler.com\",\"description\":\"Ping\",\"publisher\":{\"@id\":\"https:\/\/pingler.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/pingler.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/pingler.com\/blog\/#organization\",\"name\":\"Pingler.com\",\"url\":\"https:\/\/pingler.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg\",\"contentUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg\",\"width\":300,\"height\":160,\"caption\":\"Pingler.com\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"http:\/\/facebook.com\/pinglerdotcom\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9\",\"name\":\"Pingler\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g\",\"caption\":\"Pingler\"},\"sameAs\":[\"http:\/\/www.pingler.com\"],\"url\":\"https:\/\/pingler.com\/blog\/author\/pingler\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The Basics of Search Engine Spiders - Pingler Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/","og_locale":"en_US","og_type":"article","og_title":"The Basics of Search Engine Spiders - Pingler Blog","og_description":"Search engine spiders, also known as crawlers, consist of scripts or programs that when used by search engines, go out and collect information about publicly-accessible web pages. \u00a0These spiders are solely responsible for collecting information \u2013 including content, keywords and meta data \u2013 and use this information to index your site and rank it against [&hellip;]","og_url":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/","og_site_name":"Pingler Blog","article_publisher":"http:\/\/facebook.com\/pinglerdotcom","article_published_time":"2012-03-26T09:00:36+00:00","article_modified_time":"2014-08-19T19:13:26+00:00","og_image":[{"width":450,"height":300,"url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg","type":"image\/jpeg"}],"author":"Pingler","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Pingler","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#article","isPartOf":{"@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/"},"author":{"name":"Pingler","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9"},"headline":"The Basics of Search Engine Spiders","datePublished":"2012-03-26T09:00:36+00:00","dateModified":"2014-08-19T19:13:26+00:00","mainEntityOfPage":{"@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/"},"wordCount":518,"commentCount":5,"publisher":{"@id":"https:\/\/pingler.com\/blog\/#organization"},"image":{"@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#primaryimage"},"thumbnailUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg","keywords":["Search Engine Spider"],"articleSection":["How To Use Pingler","Plugins and Tools","SEO"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/","url":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/","name":"The Basics of Search Engine Spiders - Pingler Blog","isPartOf":{"@id":"https:\/\/pingler.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#primaryimage"},"image":{"@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#primaryimage"},"thumbnailUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg","datePublished":"2012-03-26T09:00:36+00:00","dateModified":"2014-08-19T19:13:26+00:00","breadcrumb":{"@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#primaryimage","url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg","contentUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/03\/22954711_s.jpg","width":450,"height":300},{"@type":"BreadcrumbList","@id":"https:\/\/pingler.com\/blog\/the-basics-of-search-engine-spiders\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/pingler.com\/blog\/"},{"@type":"ListItem","position":2,"name":"The Basics of Search Engine Spiders"}]},{"@type":"WebSite","@id":"https:\/\/pingler.com\/blog\/#website","url":"https:\/\/pingler.com\/blog\/","name":"Pingler.com","description":"Ping","publisher":{"@id":"https:\/\/pingler.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/pingler.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/pingler.com\/blog\/#organization","name":"Pingler.com","url":"https:\/\/pingler.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg","contentUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg","width":300,"height":160,"caption":"Pingler.com"},"image":{"@id":"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["http:\/\/facebook.com\/pinglerdotcom"]},{"@type":"Person","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9","name":"Pingler","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g","caption":"Pingler"},"sameAs":["http:\/\/www.pingler.com"],"url":"https:\/\/pingler.com\/blog\/author\/pingler\/"}]}},"_links":{"self":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/675","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/comments?post=675"}],"version-history":[{"count":4,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/675\/revisions"}],"predecessor-version":[{"id":5793,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/675\/revisions\/5793"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/media\/5792"}],"wp:attachment":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/media?parent=675"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/categories?post=675"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/tags?post=675"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}