{"id":6143,"date":"2023-05-09T13:17:11","date_gmt":"2023-05-09T13:17:11","guid":{"rendered":"https:\/\/www.silkstream.net\/blog\/?p=6143"},"modified":"2023-05-09T13:17:12","modified_gmt":"2023-05-09T13:17:12","slug":"google-has-released-a-new-crawler-heres-what-we-know","status":"publish","type":"post","link":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html","title":{"rendered":"Google Has Released A New Crawler – Here\u2019s What We Know"},"content":{"rendered":"\n

At the end of April, Google quietly updated their web crawler list to include a new addition – GoogleOther. Google is constantly updating, with daily changes to the algorithms, improvements to security and search systems and more, so it\u2019s no surprise that this update has come without much in the way of announcement beyond a LinkedIn post<\/a> by Analyst Gary Illyes, stating: <\/p>\n\n\n\n

\u201cWe added a new crawler, GoogleOther to our list of crawlers that ultimately take some strain off of Googlebot. This is a no-op change for you, but it\u2019s interesting nonetheless I reckon. As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot\u2019s crawl jobs are only used internally for building the index that\u2019s used by Search. For this we added a new crawler, GoogleOther, that will replace some of Googlebot\u2019s other jobs like R&D crawls to free up some crawl capacity for Googlebot.\u201d<\/p>\n\n\n\n

GoogleOther was added to the list of crawlers and is designed to relieve the strain the other crawling bots, particularly Googlebot, are under. As one of the most well-known search engines, Google processes over 99,000 searches every second, or over 8.5 billion searches a day and with that level of traffic, the strain on its crawlers to effectively and efficiently index millions of websites to cater to these searches is growing each day. With new AI-based algorithms and machine learning, Googlebot is under more strain and pressure than ever to provide accurate and thorough crawling of websites. <\/p>\n\n\n\n

GoogleOther has been released as a way to reduce the strain, offering a generic crawl bot that works along the same directives and protocols as the main Googlebot crawler but frees up resources for the main crawlers. The listing on the Developers’ site lists Googlebot as: <\/p>\n\n\n\n

\u201cGeneric crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\u201d<\/p>\n\n\n\n

It has the same limitations and features, including robots.txt, host load limitations, the same HTTP protocol version and fetch sizes, effectively making this crawler the same as Googlebot, but under a different name. For webmasters, this means that there is very little that will change or needs to be done. You\u2019ll simply see a new bot appear from time to time. <\/p>\n\n\n\n

For more information about how Google views your website or for help setting up your website to be optimised for not just Google, but other search engines too, we are on hand to help. Simply get in touch to find out more.<\/p>\n","protected":false},"excerpt":{"rendered":"

At the end of April, Google quietly updated their web crawler list to include a new addition – GoogleOther. Here’s what we know.<\/p>\n","protected":false},"author":12,"featured_media":6144,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[133],"class_list":["post-6143","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-online-marketing","tag-featured"],"yoast_head":"\nGoogle Has Released A New Crawler - Here\u2019s What We Know<\/title>\n<meta name=\"description\" content=\"At the end of April, Google quietly updated their web crawler list to include a new addition - GoogleOther. Here's what we know.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Google Has Released A New Crawler - Here\u2019s What We Know\" \/>\n<meta property=\"og:description\" content=\"At the end of April, Google quietly updated their web crawler list to include a new addition - GoogleOther. Here's what we know.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html\" \/>\n<meta property=\"og:site_name\" content=\"Silkstream\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/silkstream\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-09T13:17:11+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-05-09T13:17:12+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.silkstream.net\/blog\/wp-content\/uploads\/2023\/05\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1466\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Robyn\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@silkstreamnet\" \/>\n<meta name=\"twitter:site\" content=\"@silkstreamnet\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Robyn\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html\"},\"author\":{\"name\":\"Robyn\",\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/#\\\/schema\\\/person\\\/879b0ba81e7cf03016a03fe849280713\"},\"headline\":\"Google Has Released A New Crawler – Here\u2019s What We Know\",\"datePublished\":\"2023-05-09T13:17:11+00:00\",\"dateModified\":\"2023-05-09T13:17:12+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html\"},\"wordCount\":452,\"image\":{\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/05\\\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg\",\"keywords\":[\"featured\"],\"articleSection\":[\"Online Marketing\"],\"inLanguage\":\"en-GB\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html\",\"url\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html\",\"name\":\"Google Has Released A New Crawler - Here\u2019s What We Know\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/05\\\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg\",\"datePublished\":\"2023-05-09T13:17:11+00:00\",\"dateModified\":\"2023-05-09T13:17:12+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/#\\\/schema\\\/person\\\/879b0ba81e7cf03016a03fe849280713\"},\"description\":\"At the end of April, Google quietly updated their web crawler list to include a new addition - GoogleOther. Here's what we know.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html#primaryimage\",\"url\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/05\\\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg\",\"contentUrl\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/05\\\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg\",\"width\":2560,\"height\":1466},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/2023\\\/05\\\/google-has-released-a-new-crawler-heres-what-we-know.html#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.silkstream.net\\\/blog\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Google Has Released A New Crawler – Here\u2019s What We Know\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/\",\"name\":\"Silkstream\",\"description\":\"Here you'll find the latest blogs, white papers and case studies from Silkstream experts.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.silkstream.net\\\/blog\\\/#\\\/schema\\\/person\\\/879b0ba81e7cf03016a03fe849280713\",\"name\":\"Robyn\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/baf47c691b9928e2c60c2f00337882b782eb9fc650aa3772dfef2f4d33baef5b?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/baf47c691b9928e2c60c2f00337882b782eb9fc650aa3772dfef2f4d33baef5b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/baf47c691b9928e2c60c2f00337882b782eb9fc650aa3772dfef2f4d33baef5b?s=96&d=mm&r=g\",\"caption\":\"Robyn\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Google Has Released A New Crawler - Here\u2019s What We Know","description":"At the end of April, Google quietly updated their web crawler list to include a new addition - GoogleOther. Here's what we know.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html","og_locale":"en_GB","og_type":"article","og_title":"Google Has Released A New Crawler - Here\u2019s What We Know","og_description":"At the end of April, Google quietly updated their web crawler list to include a new addition - GoogleOther. Here's what we know.","og_url":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html","og_site_name":"Silkstream","article_publisher":"https:\/\/www.facebook.com\/silkstream","article_published_time":"2023-05-09T13:17:11+00:00","article_modified_time":"2023-05-09T13:17:12+00:00","og_image":[{"width":2560,"height":1466,"url":"https:\/\/www.silkstream.net\/blog\/wp-content\/uploads\/2023\/05\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg","type":"image\/jpeg"}],"author":"Robyn","twitter_card":"summary_large_image","twitter_creator":"@silkstreamnet","twitter_site":"@silkstreamnet","twitter_misc":{"Written by":"Robyn","Estimated reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html#article","isPartOf":{"@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html"},"author":{"name":"Robyn","@id":"https:\/\/www.silkstream.net\/blog\/#\/schema\/person\/879b0ba81e7cf03016a03fe849280713"},"headline":"Google Has Released A New Crawler – Here\u2019s What We Know","datePublished":"2023-05-09T13:17:11+00:00","dateModified":"2023-05-09T13:17:12+00:00","mainEntityOfPage":{"@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html"},"wordCount":452,"image":{"@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html#primaryimage"},"thumbnailUrl":"https:\/\/www.silkstream.net\/blog\/wp-content\/uploads\/2023\/05\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg","keywords":["featured"],"articleSection":["Online Marketing"],"inLanguage":"en-GB"},{"@type":"WebPage","@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html","url":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html","name":"Google Has Released A New Crawler - Here\u2019s What We Know","isPartOf":{"@id":"https:\/\/www.silkstream.net\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html#primaryimage"},"image":{"@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html#primaryimage"},"thumbnailUrl":"https:\/\/www.silkstream.net\/blog\/wp-content\/uploads\/2023\/05\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg","datePublished":"2023-05-09T13:17:11+00:00","dateModified":"2023-05-09T13:17:12+00:00","author":{"@id":"https:\/\/www.silkstream.net\/blog\/#\/schema\/person\/879b0ba81e7cf03016a03fe849280713"},"description":"At the end of April, Google quietly updated their web crawler list to include a new addition - GoogleOther. Here's what we know.","breadcrumb":{"@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html#primaryimage","url":"https:\/\/www.silkstream.net\/blog\/wp-content\/uploads\/2023\/05\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg","contentUrl":"https:\/\/www.silkstream.net\/blog\/wp-content\/uploads\/2023\/05\/nathana-reboucas-O5v8heKY4cI-unsplash-scaled.jpg","width":2560,"height":1466},{"@type":"BreadcrumbList","@id":"https:\/\/www.silkstream.net\/blog\/2023\/05\/google-has-released-a-new-crawler-heres-what-we-know.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.silkstream.net\/blog"},{"@type":"ListItem","position":2,"name":"Google Has Released A New Crawler – Here\u2019s What We Know"}]},{"@type":"WebSite","@id":"https:\/\/www.silkstream.net\/blog\/#website","url":"https:\/\/www.silkstream.net\/blog\/","name":"Silkstream","description":"Here you'll find the latest blogs, white papers and case studies from Silkstream experts.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.silkstream.net\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/www.silkstream.net\/blog\/#\/schema\/person\/879b0ba81e7cf03016a03fe849280713","name":"Robyn","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/secure.gravatar.com\/avatar\/baf47c691b9928e2c60c2f00337882b782eb9fc650aa3772dfef2f4d33baef5b?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/baf47c691b9928e2c60c2f00337882b782eb9fc650aa3772dfef2f4d33baef5b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/baf47c691b9928e2c60c2f00337882b782eb9fc650aa3772dfef2f4d33baef5b?s=96&d=mm&r=g","caption":"Robyn"}}]}},"_links":{"self":[{"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/posts\/6143","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/comments?post=6143"}],"version-history":[{"count":1,"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/posts\/6143\/revisions"}],"predecessor-version":[{"id":6145,"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/posts\/6143\/revisions\/6145"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/media\/6144"}],"wp:attachment":[{"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/media?parent=6143"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/categories?post=6143"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silkstream.net\/blog\/wp-json\/wp\/v2\/tags?post=6143"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}