A question that we've been asked countless times when assisting businesses with their websites, is how long will it take Google to notice the changes we are doing. Now, unfortunately, there isn't a cut and dry answer to this and it's very much scenario based. However, Google does offer us some ideas and you can watch the video below to see. Whilst we're on this topic, once we've got an idea of how long it takes, the next question is what can we do to speed it up? Below are three examples of ways to speed up the process of having changes crawled more quickly:
- Submitting a new sitemap to Search Console if new pages have been added.
- Requesting a re-crawl from Search Console.
- Linking to the changes from a high-level page on the website.
See, Google has limitations, believe it or not, and each time it visits your website it does its best to crawl as much of the website as possible but this doesn't mean everything will be crawled. If you're making changes to pages that are fairly hidden, chances are that it will take Google longer to notice them. So, looking at the items above, it might be a good idea to create an area on your homepage where you link to these changes. For example, if you're publishing a new blog post and want the blog post crawled quickly, you may want to have a "latest posts" area on your homepage that leads Google to the new post. If you've added a new page to the website and this page is fairly hidden in the depths of your website, you may want to re-compile your sitemap and submit it to Search Console for review.
John Mueller is a Webmaster Trends Analyst at Google who has taken over the role of publishing videos on Google's Webmaster YouTube channel (He took over from Matt Cutts, for those who are interested). Yesterday he published a video on this exact topic and you may watch it below:
A lot of SEO's talk about a topic called "crawl budget" and refer to it as the amount of time Google has to give to your site. In a way, this is a very good way of looking at it. A frequently updated website that presents great content is more than likely a website that gets crawled frequently and commands more crawl budget. On the contrary, static websites which only get updated with minor changes from year to year are more likely to get less crawl budget and in turn, wait longer for changes to reflect in Google. Does having your website crawled quickly and frequently make a difference? It really depends on the purpose of your website or what you're aiming to achieve. If you're working on your SEO, implementing changes and wanting to see the result, then having the website crawled quickly is crucial or your SEO experiments will take a long time. If you're publishing news on your website that is linked to time then having it crawled quickly is important. If you aren't experimenting with SEO or trying to publish news that's time sensitive then it's not too important to have the high crawl rate. The last thing you want to do is try to encourage Google to come frequently and then not have something new to show it, in our experience this often results in Google then staying away. Last year in September we were contacted by a company that received roughly 5000 visitors per day. On the 25th of September they noticed that something wrong had happened and by the 28th of September their traffic was down about 400%. We received the panicked call and were put to work immediately on resolving the problem. We performed a number of tasks, starting with isolating what the problem was, how it happened and then went about getting the indexation correct again. You can see from below that the problem started on the 25th of September and we started working on September 28th. The traffic was back to its usual levels by roughly October 2nd. In this instance, we are dealing with a website that has hundreds of thousands of pages on it and it took 4 days for Google to completely re-crawl the website and put things back to normal with our assistance.
This gives you a good idea of what's possible when you follow simple processes and have a website that Google is familiar with. The one caveat here is that we did originally work on the website's SEO for a number of months the previous year and knew that we had set the website up in such a way that had Google crawling frequently without a hindrance from the website. A website that doesn't have an optimised foundation will probably not recover nearly as quickly as this.
