{"id":596,"date":"2012-02-15T09:00:02","date_gmt":"2012-02-15T09:00:02","guid":{"rendered":"http:\/\/pingler.com\/blog\/?p=596"},"modified":"2014-08-19T21:29:56","modified_gmt":"2014-08-19T21:29:56","slug":"everything-you-need-to-know-about-robots-txt","status":"publish","type":"post","link":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/","title":{"rendered":"Everything You Need to Know about Robots.txt"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-medium wp-image-5838\" src=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s-300x240.jpg\" alt=\"Robot Laptop\" width=\"300\" height=\"240\" srcset=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s-300x240.jpg 300w, https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg 450w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/>Invented in 1994 by Martijn Koster, the robots.txt protocol, also known as the Robots Exclusion Protocol or the Robot Exclusion Standard is used to prevent the web crawlers from search engines as well as other types of web robots from being able to access and index parts of or the entirety of a website that otherwise would be accessible by the public.\u00a0 Robots.txt is used by search engines to better categorise as well as archive web sites, and webmasters will use it as a way to proofread source code.\u00a0 While robots.txt can be very helpful, since it is a de facto standard and was reached on consensus, it does rely on search engines voluntarily choosing to obey the instructions within.\u00a0 This means that web crawling robots can be instructed to ignore your robots.txt file completely.\u00a0 This can lead to security issues for your website including the following:<\/p>\n<ul>\n<li>Pages listed in the robots.txt file could provide unintended access to parts of your website<\/li>\n<li>Email addresses can be harvested and used by spammers<\/li>\n<li>Anyone can see what parts of your site you are trying to hide since robots.txt is a publicly available file.<\/li>\n<\/ul>\n<p><strong>Ways to Create a robots.txt File<\/strong><\/p>\n<p>A robots.txt file can be created by any application that is capable of producing a text file, as long as said file is in plain text, or the .txt format.\u00a0 For Microsoft Windows users, this can be done using either the notepad or wordpad.exe applications.\u00a0 Macintosh users can use TextEdit to create robots.txt, and users on a Unix or Linux system can use vi or emacs. \u00a0Once you have created a robots.txt file, it should be placed in the top level directory on your web server.<\/p>\n<p><strong>Why is robots.txt So Important?<\/strong><\/p>\n<p>The biggest advantage is that robots.txt can help you avoid wasting precious server resources.\u00a0 This is because when a web crawler crawls your site to index it, any applications that you have, such as a contact form, search box, etc. will ask for access just as a browser would if a person was using it, which can put stress on your server.\u00a0 A way around this would be to instruct robots.txt to ignore the directory with your scripts in it, since there is no need for your scripts to be indexed by a search engine.<\/p>\n<p>You can also use a robots.txt file to cut down on bandwidth usage, since you will be able to stop web crawlers from requesting files that would use a lot of bandwidth being constantly accessed by web crawlers such as image files.\u00a0 You can use robots.txt to restrict access to your image directory, so that directory and the files within will be restricted.<\/p>\n<p><strong>Pingler robots.txt Checker<\/strong><\/p>\n<p>By using web based tools, such as the <a href=\"http:\/\/pingler.com\/seo-tools\/robots-txt-checker\/\" data-wpel-link=\"external\">Pingler robots.txt checker<\/a>, you can access the data listed in a robots.txt file from any website.\u00a0 This can be an incredibly useful SEO tool, since you can use it to alter what search engines see as well as check the robots.txt file of other websites to see what they have listed.<\/p>\n<p><i><html><br \/>\n\t<head><br \/>\n\t\t<title><\/title><br \/>\n\t<\/head><br \/>\n\t<body><\/p>\n<p style=\"text-align: center;\">\n\t\t\t<a href=\"http:\/\/www.pingler.com\/account\/register\/\" data-wpel-link=\"external\"><img decoding=\"async\" alt=\"\" src=\"http:\/\/www.pingler.com\/affiliates\/banners\/Pingler-468x60-01A.gif\" style=\"width: 468px; height: 60px;\" \/><\/a><\/p>\n<p>\t<\/body><br \/>\n<\/html><\/i><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Invented in 1994 by Martijn Koster, the robots.txt protocol, also known as the Robots Exclusion Protocol or the Robot Exclusion Standard is used to prevent the web crawlers from search engines as well as other types of web robots from being able to access and index parts of or the entirety of a website that [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":5838,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,25],"tags":[176],"class_list":["post-596","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-general","category-plugins-and-tools","tag-robots-txt"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.1.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Everything You Need to Know about Robots.txt - Pingler Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Everything You Need to Know about Robots.txt - Pingler Blog\" \/>\n<meta property=\"og:description\" content=\"Invented in 1994 by Martijn Koster, the robots.txt protocol, also known as the Robots Exclusion Protocol or the Robot Exclusion Standard is used to prevent the web crawlers from search engines as well as other types of web robots from being able to access and index parts of or the entirety of a website that [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"Pingler Blog\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/facebook.com\/pinglerdotcom\" \/>\n<meta property=\"article:published_time\" content=\"2012-02-15T09:00:02+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-08-19T21:29:56+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"450\" \/>\n\t<meta property=\"og:image:height\" content=\"360\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Pingler\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Pingler\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/\"},\"author\":{\"name\":\"Pingler\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9\"},\"headline\":\"Everything You Need to Know about Robots.txt\",\"datePublished\":\"2012-02-15T09:00:02+00:00\",\"dateModified\":\"2014-08-19T21:29:56+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/\"},\"wordCount\":535,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\/\/pingler.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg\",\"keywords\":[\"robots.txt\"],\"articleSection\":[\"General\",\"Plugins and Tools\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/\",\"url\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/\",\"name\":\"Everything You Need to Know about Robots.txt - Pingler Blog\",\"isPartOf\":{\"@id\":\"https:\/\/pingler.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg\",\"datePublished\":\"2012-02-15T09:00:02+00:00\",\"dateModified\":\"2014-08-19T21:29:56+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#primaryimage\",\"url\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg\",\"contentUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg\",\"width\":450,\"height\":360},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/pingler.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Everything You Need to Know about Robots.txt\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/pingler.com\/blog\/#website\",\"url\":\"https:\/\/pingler.com\/blog\/\",\"name\":\"Pingler.com\",\"description\":\"Ping\",\"publisher\":{\"@id\":\"https:\/\/pingler.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/pingler.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/pingler.com\/blog\/#organization\",\"name\":\"Pingler.com\",\"url\":\"https:\/\/pingler.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg\",\"contentUrl\":\"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg\",\"width\":300,\"height\":160,\"caption\":\"Pingler.com\"},\"image\":{\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"http:\/\/facebook.com\/pinglerdotcom\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9\",\"name\":\"Pingler\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/pingler.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g\",\"caption\":\"Pingler\"},\"sameAs\":[\"http:\/\/www.pingler.com\"],\"url\":\"https:\/\/pingler.com\/blog\/author\/pingler\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Everything You Need to Know about Robots.txt - Pingler Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/","og_locale":"en_US","og_type":"article","og_title":"Everything You Need to Know about Robots.txt - Pingler Blog","og_description":"Invented in 1994 by Martijn Koster, the robots.txt protocol, also known as the Robots Exclusion Protocol or the Robot Exclusion Standard is used to prevent the web crawlers from search engines as well as other types of web robots from being able to access and index parts of or the entirety of a website that [&hellip;]","og_url":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/","og_site_name":"Pingler Blog","article_publisher":"http:\/\/facebook.com\/pinglerdotcom","article_published_time":"2012-02-15T09:00:02+00:00","article_modified_time":"2014-08-19T21:29:56+00:00","og_image":[{"width":450,"height":360,"url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg","type":"image\/jpeg"}],"author":"Pingler","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Pingler","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#article","isPartOf":{"@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/"},"author":{"name":"Pingler","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9"},"headline":"Everything You Need to Know about Robots.txt","datePublished":"2012-02-15T09:00:02+00:00","dateModified":"2014-08-19T21:29:56+00:00","mainEntityOfPage":{"@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/"},"wordCount":535,"commentCount":1,"publisher":{"@id":"https:\/\/pingler.com\/blog\/#organization"},"image":{"@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg","keywords":["robots.txt"],"articleSection":["General","Plugins and Tools"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/","url":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/","name":"Everything You Need to Know about Robots.txt - Pingler Blog","isPartOf":{"@id":"https:\/\/pingler.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#primaryimage"},"image":{"@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#primaryimage"},"thumbnailUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg","datePublished":"2012-02-15T09:00:02+00:00","dateModified":"2014-08-19T21:29:56+00:00","breadcrumb":{"@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#primaryimage","url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg","contentUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg","width":450,"height":360},{"@type":"BreadcrumbList","@id":"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/pingler.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Everything You Need to Know about Robots.txt"}]},{"@type":"WebSite","@id":"https:\/\/pingler.com\/blog\/#website","url":"https:\/\/pingler.com\/blog\/","name":"Pingler.com","description":"Ping","publisher":{"@id":"https:\/\/pingler.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/pingler.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/pingler.com\/blog\/#organization","name":"Pingler.com","url":"https:\/\/pingler.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg","contentUrl":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2024\/09\/pingler-logo.jpg","width":300,"height":160,"caption":"Pingler.com"},"image":{"@id":"https:\/\/pingler.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["http:\/\/facebook.com\/pinglerdotcom"]},{"@type":"Person","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/4995c81e4dee73f9670af7fd1e1529b9","name":"Pingler","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pingler.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/db71def0321a21d7d90da93e8a25172df87a2673a1d70d32d64cc075558a9597?s=96&d=mm&r=g","caption":"Pingler"},"sameAs":["http:\/\/www.pingler.com"],"url":"https:\/\/pingler.com\/blog\/author\/pingler\/"}]}},"_links":{"self":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/596","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/comments?post=596"}],"version-history":[{"count":4,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/596\/revisions"}],"predecessor-version":[{"id":5840,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/posts\/596\/revisions\/5840"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/media\/5838"}],"wp:attachment":[{"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/media?parent=596"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/categories?post=596"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pingler.com\/blog\/wp-json\/wp\/v2\/tags?post=596"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}