{"id":750,"date":"2012-05-07T09:00:36","date_gmt":"2012-05-07T09:00:36","guid":{"rendered":"http:\/\/pingler.com\/blog\/?p=750"},"modified":"2014-08-20T10:56:22","modified_gmt":"2014-08-20T10:56:22","slug":"accommodating-bots-crawlers-and-spiders-on-your-website","status":"publish","type":"post","link":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/","title":{"rendered":"Accommodating Bots, Crawlers and Spiders On Your Website"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-medium wp-image-5944\" src=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s-300x225.jpg\" alt=\"Robot Crawler\" width=\"300\" height=\"225\" srcset=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s-300x225.jpg 300w, https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg 450w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/>Every single day, there are countless bots, spiders and crawlers perusing the internet on behalf of major search engines that are collecting information about each site and page they come across. \u00a0Your site \u2013 especially if it has many backlinks \u2013 may be indexed several times per day by bots that are looking for any changes that have been made to the website. \u00a0If you want to take advantage of these bots and their constant patrolling, you will want to make sure that your site is optimized for their presence. These tips will provide you with ways to augment your site&#8217;s indexing potential and allow you to get properly indexed.<\/p>\n<p><strong>The First Three Sentences Are Crucial<\/strong><\/p>\n<p>On each page, you should have a page title, article title and page content. \u00a0One of the best ways to make sure bots, spiders and crawlers are comfortable on your pages is to make sure that you use titles and headings that are relevant to the rest of your site. \u00a0Spiders and bots have become increasingly intuitive and look for matching content sections on each page. \u00a0You will also want to make sure that all of your keywords (or rather, the first instance of each) appears in the first two or three sentences of the page. \u00a0Your article or page title should have also emphasis on it (&lt;em&gt;, &lt;h5&gt;, etc) so that search engines properly index it along with the rest of the page.<\/p>\n<p><strong>Have A Site Map<\/strong><\/p>\n<p>Especially useful if your site is new, site maps give search bots and crawlers the ability to find all pages on your website. \u00a0Some of your pages at first may not be locatable by spiders and crawlers due to the lack of backlinks and internal linking, so a site map incorporates links to all of your pages in one easy to find section. \u00a0Based on the formatting of the site map, you can also indicate to the bots which sites are primary pages, which are secondary and which are tertiary. \u00a0Site maps work well on websites that have 50 pages or less.<\/p>\n<p><strong>Small Edits<\/strong><\/p>\n<p>Your site is completely evaluated by these crawlers, so it is vital to make sure that your site doesn&#8217;t have any aspects to it that could lead to a downgraded PageRank or other rating. \u00a0Some of the most common ways to avoid being perceived as a spammer is to reduce the instances of dashes in your URL (common in blog posts) and to use your keywords at a rate of approximately 2% density. \u00a0When creating anchor links on your site for example, use keywords in the hyperlink instead of \u201cclick here\u201d or \u201cvisit this page\u201d. \u00a0Finally, be sure to spin the descriptions on any products or services offered; if you are selling a product that a thousand other websites sell, do not use the generic item description.<\/p>\n<p><strong>See Your Site Through Their Eyes<\/strong><\/p>\n<p>Pingler has released its <a href=\"..\/..\/seo-tools\/spider-viewer\/\" data-wpel-link=\"internal\">Spider Viewer Tool<\/a> that gives you first-hand access to what search engine spiders are seeing. This strips away all of the code such as HTML and allows you to see what the spider does in relation to content and keywords. \u00a0This will let you know the extent of your keyword usage and whether or not excess and unnecessary content is being indexed by major search engines.<\/p>\n<p><i><html><br \/>\n\t<head><br \/>\n\t\t<title><\/title><br \/>\n\t<\/head><br \/>\n\t<body><\/p>\n<p style=\"text-align: center;\">\n\t\t\t<a href=\"http:\/\/www.pingler.com\/account\/register\/\" data-wpel-link=\"external\"><img decoding=\"async\" alt=\"\" src=\"http:\/\/www.pingler.com\/affiliates\/banners\/Pingler-468x60-01A.gif\" style=\"width: 468px; height: 60px;\" \/><\/a><\/p>\n<p>\t<\/body><br \/>\n<\/html><\/i><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Every single day, there are countless bots, spiders and crawlers perusing the internet on behalf of major search engines that are collecting information about each site and page they come across. \u00a0Your site \u2013 especially if it has many backlinks \u2013 may be indexed several times per day by bots that are looking for any [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":5944,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16,25,4],"tags":[250,248,251,249],"class_list":["post-750","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-google","category-plugins-and-tools","category-seo","tag-bots","tag-spider","tag-spider-viewer-tool","tag-web-crawler"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.1.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Accommodating Bots, Crawlers and Spiders On Your Website - Pingler Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Accommodating Bots, Crawlers and Spiders On Your Website - Pingler Blog\" \/>\n<meta property=\"og:description\" content=\"Every single day, there are countless bots, spiders and crawlers perusing the internet on behalf of major search engines that are collecting information about each site and page they come across. \u00a0Your site \u2013 especially if it has many backlinks \u2013 may be indexed several times per day by bots that are looking for any [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/\" \/>\n<meta property=\"og:site_name\" content=\"Pingler Blog\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/facebook.com\/pinglerdotcom\" \/>\n<meta property=\"article:published_time\" content=\"2012-05-07T09:00:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-08-20T10:56:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"450\" \/>\n\t<meta property=\"og:image:height\" content=\"338\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Pingler\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Pingler\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/\"},\"author\":{\"name\":\"Pingler\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9\"},\"headline\":\"Accommodating Bots, Crawlers and Spiders On Your Website\",\"datePublished\":\"2012-05-07T09:00:36+00:00\",\"dateModified\":\"2014-08-20T10:56:22+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/\"},\"wordCount\":559,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\/\/pingler.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg\",\"keywords\":[\"Bots\",\"Spider\",\"Spider Viewer Tool\",\"Web Crawler\"],\"articleSection\":[\"Google\",\"Plugins and Tools\",\"SEO\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/\",\"url\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/\",\"name\":\"Accommodating Bots, Crawlers and Spiders On Your Website - Pingler Blog\",\"isPartOf\":{\"@id\":\"https:\/\/pingler.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg\",\"datePublished\":\"2012-05-07T09:00:36+00:00\",\"dateModified\":\"2014-08-20T10:56:22+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#primaryimage\",\"url\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg\",\"contentUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg\",\"width\":450,\"height\":338},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/pingler.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Accommodating Bots, Crawlers and Spiders On Your Website\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/pingler.com\/blog\/#website\",\"url\":\"https:\/\/pingler.com\/blog\/\",\"name\":\"Pingler.com\",\"description\":\"Ping\",\"publisher\":{\"@id\":\"https:\/\/pingler.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/pingler.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/pingler.com\/blog\/#organization\",\"name\":\"Pingler.com\",\"url\":\"https:\/\/pingler.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg\",\"contentUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg\",\"width\":300,\"height\":160,\"caption\":\"Pingler.com\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"http:\/\/facebook.com\/pinglerdotcom\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9\",\"name\":\"Pingler\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g\",\"caption\":\"Pingler\"},\"sameAs\":[\"http:\/\/www.pingler.com\"],\"url\":\"https:\/\/pingler.com\/blog\/author\/pingler\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Accommodating Bots, Crawlers and Spiders On Your Website - Pingler Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/","og_locale":"en_US","og_type":"article","og_title":"Accommodating Bots, Crawlers and Spiders On Your Website - Pingler Blog","og_description":"Every single day, there are countless bots, spiders and crawlers perusing the internet on behalf of major search engines that are collecting information about each site and page they come across. \u00a0Your site \u2013 especially if it has many backlinks \u2013 may be indexed several times per day by bots that are looking for any [&hellip;]","og_url":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/","og_site_name":"Pingler Blog","article_publisher":"http:\/\/facebook.com\/pinglerdotcom","article_published_time":"2012-05-07T09:00:36+00:00","article_modified_time":"2014-08-20T10:56:22+00:00","og_image":[{"width":450,"height":338,"url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg","type":"image\/jpeg"}],"author":"Pingler","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Pingler","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#article","isPartOf":{"@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/"},"author":{"name":"Pingler","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9"},"headline":"Accommodating Bots, Crawlers and Spiders On Your Website","datePublished":"2012-05-07T09:00:36+00:00","dateModified":"2014-08-20T10:56:22+00:00","mainEntityOfPage":{"@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/"},"wordCount":559,"commentCount":1,"publisher":{"@id":"https:\/\/pingler.com\/blog\/#organization"},"image":{"@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#primaryimage"},"thumbnailUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg","keywords":["Bots","Spider","Spider Viewer Tool","Web Crawler"],"articleSection":["Google","Plugins and Tools","SEO"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/","url":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/","name":"Accommodating Bots, Crawlers and Spiders On Your Website - Pingler Blog","isPartOf":{"@id":"https:\/\/pingler.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#primaryimage"},"image":{"@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#primaryimage"},"thumbnailUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg","datePublished":"2012-05-07T09:00:36+00:00","dateModified":"2014-08-20T10:56:22+00:00","breadcrumb":{"@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#primaryimage","url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg","contentUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/05\/8710955_s.jpg","width":450,"height":338},{"@type":"BreadcrumbList","@id":"https:\/\/pingler.com\/blog\/accommodating-bots-crawlers-and-spiders-on-your-website\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/pingler.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Accommodating Bots, Crawlers and Spiders On Your Website"}]},{"@type":"WebSite","@id":"https:\/\/pingler.com\/blog\/#website","url":"https:\/\/pingler.com\/blog\/","name":"Pingler.com","description":"Ping","publisher":{"@id":"https:\/\/pingler.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/pingler.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/pingler.com\/blog\/#organization","name":"Pingler.com","url":"https:\/\/pingler.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg","contentUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg","width":300,"height":160,"caption":"Pingler.com"},"image":{"@id":"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["http:\/\/facebook.com\/pinglerdotcom"]},{"@type":"Person","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9","name":"Pingler","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g","caption":"Pingler"},"sameAs":["http:\/\/www.pingler.com"],"url":"https:\/\/pingler.com\/blog\/author\/pingler\/"}]}},"_links":{"self":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/750","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/comments?post=750"}],"version-history":[{"count":3,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/750\/revisions"}],"predecessor-version":[{"id":5946,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/750\/revisions\/5946"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/media\/5944"}],"wp:attachment":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/media?parent=750"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/categories?post=750"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/tags?post=750"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}