{"version":"1.0","provider_name":"Pingler Blog","provider_url":"https:\/\/pingler.com\/blog","title":"Everything You Need to Know about Robots.txt - Pingler Blog","type":"rich","width":600,"height":338,"html":"<blockquote class=\"wp-embedded-content\" data-secret=\"PImyVFwwbs\"><a href=\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/\">Everything You Need to Know about Robots.txt<\/a><\/blockquote><iframe sandbox=\"allow-scripts\" security=\"restricted\" src=\"https:\/\/pingler.com\/blog\/everything-you-need-to-know-about-robots-txt\/embed\/#?secret=PImyVFwwbs\" width=\"600\" height=\"338\" title=\"&#8220;Everything You Need to Know about Robots.txt&#8221; &#8212; Pingler Blog\" data-secret=\"PImyVFwwbs\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\" class=\"wp-embedded-content\"><\/iframe><script type=\"text\/javascript\">\n\/* <![CDATA[ *\/\n\/*! This file is auto-generated *\/\n!function(d,l){\"use strict\";l.querySelector&&d.addEventListener&&\"undefined\"!=typeof URL&&(d.wp=d.wp||{},d.wp.receiveEmbedMessage||(d.wp.receiveEmbedMessage=function(e){var t=e.data;if((t||t.secret||t.message||t.value)&&!\/[^a-zA-Z0-9]\/.test(t.secret)){for(var s,r,n,a=l.querySelectorAll('iframe[data-secret=\"'+t.secret+'\"]'),o=l.querySelectorAll('blockquote[data-secret=\"'+t.secret+'\"]'),c=new RegExp(\"^https?:$\",\"i\"),i=0;i<o.length;i++)o[i].style.display=\"none\";for(i=0;i<a.length;i++)s=a[i],e.source===s.contentWindow&&(s.removeAttribute(\"style\"),\"height\"===t.message?(1e3<(r=parseInt(t.value,10))?r=1e3:~~r<200&&(r=200),s.height=r):\"link\"===t.message&&(r=new URL(s.getAttribute(\"src\")),n=new URL(t.value),c.test(n.protocol))&&n.host===r.host&&l.activeElement===s&&(d.top.location.href=t.value))}},d.addEventListener(\"message\",d.wp.receiveEmbedMessage,!1),l.addEventListener(\"DOMContentLoaded\",function(){for(var e,t,s=l.querySelectorAll(\"iframe.wp-embedded-content\"),r=0;r<s.length;r++)(t=(e=s[r]).getAttribute(\"data-secret\"))||(t=Math.random().toString(36).substring(2,12),e.src+=\"#?secret=\"+t,e.setAttribute(\"data-secret\",t)),e.contentWindow.postMessage({message:\"ready\",secret:t},\"*\")},!1)))}(window,document);\n\/\/# sourceURL=https:\/\/pingler.com\/blog\/wp-includes\/js\/wp-embed.min.js\n\/* ]]> *\/\n<\/script>\n","thumbnail_url":"https:\/\/pingler.com\/blog\/wp-content\/uploads\/2012\/02\/6811443_s.jpg","thumbnail_width":450,"thumbnail_height":360,"description":"Invented in 1994 by Martijn Koster, the robots.txt protocol, also known as the Robots Exclusion Protocol or the Robot Exclusion Standard is used to prevent the web crawlers from search engines as well as other types of web robots from being able to access and index parts of or the entirety of a website that [&hellip;]"}