How To Get Google To Crawl And Index Your Website Fully
How To Get Google To Crawl and Index Your Website Fully Indoscie Technologies is Web Design and Development, SEO, PPC, Local Listing,
Adwords, E-mail Marketing, Digital Marketing service and solutions providers at a very affordable cost.
A part of SEO is to make sure Google can crawl your website and index all your primary pages. Google is sometimes picky about what pages on a site it will index.
If you have a website indexation problem and want to get
more of your pages indexed on Google, read on.
href="https://www.hobo-web.co.uk/how-to-get-google-to-crawl-and-index-your-website-fully/#low-quality-content-on-part-of-a-web-site-can-affect-rankings-for-the-same-website-on-more-important-keyword-rankings">Low-Quality Content On Part of A Web Site Can Affect Rankings For The Same Website On More Important Keyword Rankings
href="https://www.hobo-web.co.uk/how-to-get-google-to-crawl-and-index-your-website-fully/#is-your-content-useful-for-google-users-">Is Your Content ‘useful for Google users‘?
href="https://www.hobo-web.co.uk/how-to-get-google-to-crawl-and-index-your-website-fully/#help-google-help-you-index-more-pages">Help Google Help You Index More Pages
href="https://www.hobo-web.co.uk/how-to-get-google-to-crawl-and-index-your-website-fully/#have-your-site-produce-proper-404-pages">Have your site produce proper 404 pages
href="https://www.hobo-web.co.uk/how-to-get-google-to-crawl-and-index-your-website-fully/#most-links-in-your-xml-sitemap-should-be-canonical-and-not-redirects">Most links in your XML Sitemap should be Canonical and not redirects
href="https://www.hobo-web.co.uk/how-to-get-google-to-crawl-and-index-your-website-fully/#why-is-the-number-of-indexed-urls-in-search-console-dropping-">Why Is The Number Of Indexed URLs in Search Console Dropping?
Check out the new Indexation Report In Google Search Console
This is sure to be an invaluable addition to Google Search Console for some larger sites.
If you submit an XML sitemap file in Search Console, Google will help you better understand why certain pages are not
indexed.
As you can see, Google goes to great lengths to help you to identify indexation problems on your website, including, in this example:
Status
Reason
Validation
Trend
Pages
Error
Submitted URL marked ‘noindex’
Started
5
Error
Server errors (5xx)
N/A
0
Error
Submitted URL blocked by robots.txt
N/A
0
Error
Submitted URL seems to be a Soft
404
N/A
0
Excluded
Excluded by ‘noindex’ tag
N/A
81,984
Excluded
Page with redirect
N/A
5,982
Excluded
Duplicate page without canonical tag
N/A
4,908
Excluded
Crawled – currently not indexed
N/A
2,082
Excluded
Discovered – currently not indexed
N/A
1,520
Excluded
Blocked by
robots.txt
N/A
647
Excluded
Alternate page with proper canonical tag
N/A
201
Excluded
Soft 404
N/A
147
Excluded
Submitted URL not selected as canonical
N/A
34
Valid
Indexed, not submitted in sitemap
N/A
221,004
Valid
Submitted and indexed
N/A
2,144
How To Use The Indexation Report In Google Search
Console
QUOTE: “Find out which of your pages have been crawled and indexed by Google. In this episode of Search Console Training…. how to use the Index Coverage report to assess its status and identify issues.†GoogleWMC, 2020
Will Google Index Every Page on Your Website?
No.
QUOTE: “We never index all known URLs, that’s pretty normal. I’d focus on making the site awesome and inspiring, then
things usually work out better“. John Mueller, 2018
Some URLs are not that important to Google, some are duplicates, some have conflicting indexation instructions and some pages are low-quality or even spammy.
The Importance of Unique Content For Your Website
QUOTE: “Duplicated content is often not manipulative and is commonplace on many websites and often free from malicious intent. Copied
content can often be penalised algorithmically or manually. Duplicate content is not penalised, but this is often not an optimal set-up for pages, either. Be VERY careful ‘spinning’ ‘copied’ text to make it unique!†Hobo, 2018
From a quality page point of view, duplicate content (or rather, copied content) can a low-quality indicator.
Boilerplate (especially spun) text can be another low-quality indicator.
If your
website is tarnished with these practices – it is going to be classed ‘low-quality’ by some part of the Google algorithm:
If all you have on your page are indicators of low-quality – you have a low-quality page – full stop.
If your entire website is made up of pages like that, you have a low-quality website.
If you have manipulative backlinks, then that’s a recipe for disaster.
Low-Quality Content On Part of A Web Site Can Affect Rankings For The Same Website On More Important Keyword Rankings
Moz has a good video on the Google organic quality score theory. You should watch it. It goes into a lot of stuff I (and others) have been blogging for the last few year.
One thing that could have been explained better in the video was that Moz has topical authority world wide for ‘Google SEO’ terms, hence why they can rank so easily for ‘organic quality score’.
But the explanation of the quality score is a good introduction for beginners.
I am in the camp this organic quality score has been in place for a long time, and more and more are feeling the results from it.
This is
also quite relevant to a question answered last week in the Google Webmaster Hangout which was:
“QUESTION – Is it possible that if the algorithm doesn’t particularly like our blog articles as much that it could affect our ranking and quality score on the core Content?â€
resulting in an answer:
“ANSWER: JOHN MUELLER (GOOGLE): Theoretically, that’s possible. I mean it’s kind of like we look at your web site overall. And if there’s this big chunk of content here or this big chunk kind of important wise of
your content, there that looks really iffy, then that kind of reflects across the overall picture of your website. But I don’t know in your case, if it’s really a situation that your blog is really terrible.â€
Google has introduced (at least) a ‘percieved’ risk to publishing lots of lower-quality pages on your site to in an effort to curb production of old-style SEO friendly content based on manipulating early search engine algorithms.
We are dealing with algorithms designed to target old style SEO – that focus on the truism that DOMAIN ‘REPUTATION’ plus LOTS of PAGES equals LOTS of Keywords equals LOTS of
Google traffic.
A big site can’t just get away with publishing LOTS of lower quality content in the cavalier way they used to – not without the ‘fear’ of primary content being impacted and organic search traffic throttled negatively to important pages on the site.
Google is very probably using user metrics in some way to determine the ‘quality’ of your site.
QUESTION – “I mean, would you recommend going back through articles that we posted and if there’s ones that we don’t necessarily think are great articles, that we just take them away and delete
them?â€
The reply was:
JOHN MUELLER: I think that’s always an option.Yeah. That’s something that–I’ve seen sites do that across the board,not specifically for blogs, but for content in general, where they would regularly go through all of their content and see, well, this content doesn’t get any clicks, or everyone who goes there kind of runs off screaming.
Deleting content is not always the optimal way to handle MANY types of low-quality content – far from it, in fact. Nuking it is the last option unless the pages really are
‘dead‘ content.
Any clean-up should go hand in hand with giving Google something it is going to value on your site e.g. NEW high-quality content:
The final piece of advice is interesting, too.
It gives us an insight into how Google might actually deal with your site:
JOHN MUELLER: “Then maybe that’s something where you can collect some metrics and say, well, everything that’s below this threshold, we’ll make a decision whether or not to
significantly improve it or just get rid of it.â€
E.g. (paraphrasing!)
You can probably rely on Google to ‘collect some metrics and say, well, everything that’s below this threshold, we’ll “…(insert punishment spread out over time).
Google probably has a quality score of some sort, and your site probably has a rating whatever that is relevant to (and if you get any real traffic from Google, often a manual rating).
If you have a big site, certain parts of your site will be rated more useful than others to Google.
Improving the quality of your content certainly works to improve traffic, as
does intelligently managing your content across the site. Positive results from this process are NOT going to happen overnight. I’ve blogged about this sort of thing for many years, now.
Google are going together better at rating sites that meet their guidelines for ‘quality’ and ‘user satisfaction’ here – I am putting such things in quotes here to highlight the slightly Orwellian doublespeak we have to work with.
What Are The High-Quality Characteristics of a Web Page?
What Characteristics Do The Highest Quality Pages Exhibit?
What Are The Low-Quality Signals Google Looks For?
Pages Can Be Rated
‘Medium Quality’
If you are creating content for blogging purposes, consider the following:
Google says:
QUOTE: “Creating compelling and useful content will likely influence your website more than any of the other factors.†Google, 2017
There is no one particular way to create web pages that successfully rank in Google but you must ensure:
QUOTE: “that your content kind of stands on its own†John Mueller, Google
href="https://www.youtube.com/watch?v=9h1t5fs5VcI&feature=youtu.be">2015
If you have an optimised platform on which to publish it, high-quality content is the number 1 user experience area to focus on across websites to earn traffic from Google.
If you have been impacted by Google’s content quality algorithms, your focus should be on ‘improving content’ on your site rather than deleting content:
If you need help with optimising your website content, I can offer this as a service. If you want to learn how to achieve it yourself, read on.
When You Add Pages To A Website,
“you spread the value across the whole set, and all of them get a little bit less.â€
This statement from Google is incredibly important:
QUOTE: “Usually you can’t just arbitrarily expand a site and expect the old pages to rank the same, and the new pages to rank as well. All else being the same, if you add pages, you spread the value across the whole set, and all of them get a little bit less. Of course, if you’re building up a business, then more of your pages can attract more attention, and you can sell more things, so usually “all elseâ€
doesn’t remain the same. At any rate, I’d build this more around your business needs — if you think you need pages to sell your products/services better, make them.†John Mueller, Google 2018
When you expand the content on a website, you apparently dilute the ‘value’ you already have, “all else being the same“. This is a good statement to have. I have long thought this was the direction we were heading in.
This certainly seems to be an ‘answer‘ to ‘domain authority‘ abuse
and ‘doorway page‘ abuse. It is also going to make webmasters think twice about the type of “SEO friendly content†they publish.
If your pages are low-quality, and you add more pages to your site of a similar quality, then your overall low-quality value Google assigns to your website is lowered still. That value is probably based on content relevance and E.A.T. (high-quality backlinks and expertise) and that is generally a measure of how well you deliver for Google visitors.
We already know that low-quality content on one part of a website can impact rankings for other keywords on other (even high-quality) pages on the same website. I go into this
later.
When making ‘SEO-friendly’ content you need to ask yourself:
Is this new ‘SEO-friendly’ content going to pick up links? Do it.
Is this new content going to be useful for your current website visitors? Do it, carefully.
Is this new content autogenerated across many pages? Think twice. Expect value to be diluted in some way.
Does the site already have lots of low-quality content? Deal with that, first, by improving it. Get rid of stale and irrelevant content.
Do I have the keywords I want to rank for on the site? Get those on pages you already have without keyword stuffing and prepare to make a high-quality site
PLUS high-quality content (as these are 2 different factors) to target specific keyword phrases.
I’ve advised clients for a long time it is incredibly worth investing in some higher-quality long-form in-depth content for a few pages on a site. It is a great way to get new links AND to add keywords to your site in a way that is useful to Google algorithms and human visitors and without the risk of adding low-quality pages to your site.
This concept is a bit like a leaky or reversed version of Pagerank applied ON-SITE. In the original patent, I believe Pages did not ‘lose’ PR (in a general sense), they ‘donated’ PR to other pages in the
‘set’. More pages used to result in more PR. If we think ‘value’ is the new PR score, then the more pages you add to your website (all else remaining the same) the primary way Google measures your site ‘score’ means that that quality score is coming down.
You also need to ask yourself WHY Google would rank 10,000 or 100,000 of your pages for free, just because you tweaked a few keywords? Herein lies the answer to many ranking and indexing problems.
TAKEAWAY: Don’t add lots of pages to your site where the single purpose is to meet known Google keyword relevance signals. You may get de-valued for it. Do add content to
your site that has a clear purpose for your users.
Is Your Content ‘useful for Google users‘?
QUOTE: “Our Webmaster Guidelines advise you to create websites with original content that adds value for users.†Google, 2018
Content quality is one area to focus on if you are to avoid demotion in Google.
QUOTE: “high quality content is something I’d focus on. I see lots and lots of SEO blogs
talking about user experience, which I think is a great thing to focus on as well. Because that essentially kind of focuses on what we are trying to look at as well. We want to rank content that is useful for (Googleusers) and if your content is really useful for them, then we want to rank it.†John Mueller, Google 2016
This article aims to cover the most significant challenges of writing ‘SEO-friendly’ text and web page copy for Google. High-quality content is one aspect of a high-quality page on a high-quality site.
SEO
is NO LONGER about adding keywords to pages with 300 words of text. In fact, that practice can be toxic to a site.
Your content needs to be useful to Google users.
If you run an affiliate website or have content that appears on other sites, this is even more important.
QUOTE: “This is particularly important for sites that participate in affiliate programs. Typically, affiliate websites feature product descriptions that appear on sites across that affiliate network. As a result, sites featuring mostly content from affiliate networks can suffer in Google’s search rankings, because they do not have enough added value content that
differentiates them from other sites on the web.†Google, 2018
and
QUOTE: “Google believes that pure, or “thin,†affiliate websites do not provide additional value for web users, especially (but not only) if they are part of a program that distributes its content across a network of affiliates. These sites often appear to be cookie-cutter sites or templates the same or similar content replicated within the same site, or across multiple domains or languages. Because a search results page could return several of these sites, all with
the same content, thin affiliates create a frustrating user experience.†Google, 2018
An example of a ‘thin-affiliate‘ site is a site where “product descriptions and reviews are copied directly from the original merchant without any original content or added value†and “where the majority of the site is made for affiliation and contains a limited amount of original content or added value for usersâ€.
Google says that “Good affiliates add value, for example by offering original product reviews, ratings,
navigation of products or categories, and product comparisons“.
Google offers us the following advice when dealing with sites with low-value content:
QUOTE: “Affiliate program content should form only a minor part of the content of your site if the content adds no additional features.†Google, 2018
QUOTE: “Ask yourself why a user would want to visit your site first rather than visiting the original merchant directly. Make sure your site adds substantial value beyond simply republishing content
available from the original merchant.†Google, 2018
QUOTE: “The more targeted the affiliate program is to your site’s content, the more value it will add and the more likely you will be to rank better in Google search results.â€Google, 2018
QUOTE: “Keep your content updated and relevant. Fresh, on-topic information increases the likelihood that your content will be crawled by Googlebot and clicked on by
users.â€Google, 2018
What Are The Low-Quality Signals Google Looks For?
QUOTE: “Low quality pages are unsatisfying or lacking in some element that prevents them from achieving their purpose well. These pages lack expertise or are not very trustworthy/authoritative for the purpose of the page.†Google Quality Evaluator Guidelines, 2017
These include but are
not limited to:
Lots of spammy comments
Low-quality content that lacks EAT signal (Expertise + Authority + Trustâ€)
NO Added Value for users
Poor page design
Malicious harmful or deceptive practices detected
Negative reputation
Auto-generated content
No website contact information
Fakery or INACCURATE information
Untrustworthy
Website not maintained
Pages just created to link to others
Pages lack purpose
Keyword stuffing
Inadequate customer service pages
Sites that use practices Google doesn’t want you to use
Pages can get a neutral
rating too.
Pages that have “Nothing wrong, but nothing special†about them don’t “display characteristics associated with a High rating†and puts you in the middle ground – probably not a sensible place to be a year or so down the line.
What Are The High-Quality Characteristics of a Web Page?
QUOTE: “High quality pages are satisfying and achieve their purpose well.†Google Quality Evaluator
Guidelines, 2017
The following are examples of what Google calls ‘high-quality characteristics’ of a page and should be remembered:
“A satisfying or comprehensive amount of very high-quality†main content (MC)
Copyright notifications up to date
Functional page design
Page author has Topical Authority
High-Quality Main Content
Positive Reputation or expertise of website or author (Google yourself)
Very helpful
SUPPLEMENTARY content “which improves the user experience.“
Trustworthy
Google wants to reward ‘expertise’ and ‘everyday expertise’ or experience so make this clear (Author Box?)
Accurate information
Ads can be at the top of your page as long as it does not distract from the main content on the page
Highly satisfying website contact information
Customised and very helpful 404 error pages
Awards
Evidence of expertise
Attention to detail
If Google can detect investment in time and labour on your site – there are indications that they will reward
you for this (or at least – you won’t be affected when others are, meaning you rise in Google SERPs when others fall).
Google Demotion Algorithms Target Low-Quality Content
Optimising (without improving) low-quality content springs traps set by ever-improving core quality algorithms.
What this means is that ‘optimising’ low-quality pages is very much swimming upstream.
Optimising low-quality pages without value-add is self-defeating, now that the algorithms – and manual quality rating efforts – have got that stuff nailed down.
If you optimise low-quality pages using old school
SEO techniques, you will be hit with a low-quality algorithm (like the Quality Update or Google Panda).
You must avoid boilerplate text, spun text or duplicate content when creating pages – or you are Panda Bamboo – as Google hinted at in the 2015 Quality Rater’s Guide.
QUOTE: “6.1 Low Quality Main Content One of the most important considerations in PQ rating is the quality of the MC. The quality of the MC is determined by how much time, effort, expertise, and talent/skill have gone into the creation of the page. Consider this example: Most students have to write papers for high school or college. Many students
take shortcuts to save time and effort by doing one or more of the following:
Buying papers online or getting someone else to write for them
Making things up.
Writing quickly with no drafts or editing.
Filling the report with large pictures or other distracting content.
Copying the entire report from an encyclopedia, or paraphrasing content by changing words or sentence structure here and there.
Using commonly known facts, for example, “Argentina is a country. People live in Argentina. Argentina has borders. Some people like Argentina.â€
Using a lot of words to communicate
only basic ideas or facts, for example, “Pandas eat bamboo. Pandas eat a lot of bamboo. It’s the best food for a Panda bear.â€
Unfortunately, the content of some webpages is similarly created. We will consider content to be Low quality if it is created without adequate time, effort, expertise, or talent/skill. Pages with low quality MC do not achieve their purpose well. Important: Low quality MC is a sufficient reason to give a page a Low quality rating.â€
Google rewards uniqueness or punishes the lack of it.
The number 1 way to do ‘SEO copywriting‘ will be to edit the
actual page copy to continually add unique content and improve its accuracy, uniqueness, relevance, succinctness, and use.
Low-Quality Content Is Not Meant To Rank High in Google.
A Google spokesman said not that long ago that Google Panda was about preventing types of sites that shouldn’t rank for particular keywords from ranking for them.
QUOTE: “(Google Panda) measures the quality of a site pretty much by looking at the vast majority of the pages at least. But essentially allows us to take quality of the whole site into account when ranking pages from that particular site and adjust the ranking accordingly
for the pages. So essentially, if you want a blunt answer, it will not devalue, it will actually demote. Basically, we figured that site is trying to game our systems, and unfortunately, successfully. So we will adjust the rank. We will push the site back just to make sure that it’s not working anymore.†Gary Illyes – Search Engine Land, 2017
When Google demotes your page for duplicate content practices, and there’s nothing left in the way of unique content to continue ranking you for – your web pages will mostly be
ignored by Google.
The way I look at it – once Google strips away all the stuff it ignores (duplicate text) – what’s left? In effect, that’s what you can expect Google to reward you for. If what is left is boilerplate synonymised text content – that’s now being classed as web spam – or ‘spinning’.
NOTE – The ratio of duplicate content on any page is going to hurt you if you have more duplicate text than unique content. A simple check of the pages, page to page, on the site is all that’s needed to ensure each page is DIFFERENT (regarding text) page-to-page.
If you have large sections of duplicate
text page-to-page – that is a problem that should be targeted and removed.
It is important to note:
The main text content on the page must be unique to avoid Google’s page quality algorithms.
Verbose text must NOT be created or spun automatically
Text should NOT be optimised to a template as this just creates a footprint across many pages that can be interpreted as redundant or manipulative boilerplate text.
Text should be HIGHLY descriptive, unique and concise
If you have a lot of pages to address, the main priority is to create a UNIQUE couple of paragraphs of text, at least, for the MC (Main Content). Pages do not need
thousands of words to rank. They just need to MEET A SPECIFIC USER INTENT and not TRIP ‘LOW_QUALTY’ FILTERS. A page with just a few sentences of unique text still meets this requirement (150-300 words) – for now.
When it comes to out-competing competitor pages, you are going to have to look at what the top competing page is doing when it comes to main content text. Chances are – they have some unique text on the page. If they rank with duplicate text, either their SUPPLEMENTARY CONTENT is better, or the competitor domain has more RANKING ABILITY because of either GOOD BACKLINKS or BETTER USER EXPERIENCE.
Updating content on a site should
be a priority as Google rewards fresher content for certain searches.
Google Rates ‘Copied’ Main Content ‘Lowest’
This is where you are swimming upstream. Copied content is not going to be a long-term strategy when creating a unique page better than your competitions’ pages.
In the latest Google Search Quality Evaluator Guidelines that were published on 14 March 2017, Google states:
7.4.5 Copied Main
Content
Every page needs MC. One way to create MC with no time, effort, or expertise is to copy it from another source. Important: We do not consider legitimately licensed or syndicated content to be “copied†(see here for more on web syndication). Examples of syndicated content in the U.S. include news articles by AP or Reuters.
The word “copied†refers to the practice of “scraping†content, or copying content from other nonÂaffiliated websites without adding any original content or value to users (see here for more information on copied or scraped content).
If all or
most of the MC on the page is copied, think about the purpose of the page. Why does the page exist? What value does the page have for users? Why should users look at the page with copied content instead of the original source? Important: The Lowest rating is appropriate if all or almost all of the MC on the page is copied with little or no time, effort, expertise, manual curation, or added value for users. Such pages should be rated Lowest, even if the page assigns credit for the content to another source.
and
7.4.6 More About Copied Content
All of the following are considered copied
content:
â— Content copied exactly from an identifiable source. Sometimes an entire page is copied, and sometimes just parts of the page are copied. Sometimes multiple pages are copied and then pasted together into a single page. Text that has been copied exactly is usually the easiest type of copied content to identify.
◠Content which is copied, but changed slightly from the original. This type of copying makes it difficult to find the exact matching original source. Sometimes just a few words are changed, or whole sentences are changed, or a “find and replace†modification is made, where one word is replaced with another throughout the
text. These types of changes are deliberately done to make it difficult to find the original source of the content. We call this kind of content “copied with minimal alteration.â€
◠Content copied from a changing source, such as a search results page or news feed. You often will not be able to find an exact matching original source if it is a copy of “dynamic†content (content which changes frequently). However, we will still consider this to be copied content. Important: The Lowest rating is appropriate if all or almost all of the MC on the page is copied with little or no time, effort, expertise, manual curation, or added value for users. Such
pages should be rated Lowest, even if the page assigns credit for the content to another source.
Help Google Help You Index More Pages
Minimise the production of doorway-type pages you produce on your site
You will need another content strategy. If you are forced to employ these type of pages, you need to do it in a better way.
QUOTE: “if you have a handful of locations and you have unique valuable content to provide for those individual locations I think providing that on your website is perfectly fine if you have hundreds of locations then putting out separate landing
pages for every city or every region is almost more like creating a bunch of doorway pages so that’s something I really discourage†John Mueller Google 2017
Minimise the production of thin pages you produce on your site
You will need to check how sloppy your CMS is. Make sure it does not inadvertently produce pages with little to no unique content on them (especially if you have ads on
them).
QUOTE: “John says to avoid lots of “just automatically generated†pages and “if these are pages that are not automatically generated, then I wouldn’t see that as web spam.†Conversely then “automatically generated†content = web spam? It is evident Googlebot expects to see a well formed 404 if no page exists at a url.†Shaun Anderson, Hobo
This will prevent the automatic creation of thin pages and could help prevent against negative SEO attacks, too.
QUOTE: “Tell visitors clearly that the page they’re looking for can’t be found. Use language that is friendly and inviting. Make sure your 404 page uses the same look and feel (including navigation) as the rest of your site. Consider adding links to your most popular articles or posts, as well as a link to your site’s home page. Think about providing a way for users to report a broken link. No matter how beautiful and useful your custom 404 page, you probably don’t want it to appear
in Google search results. In order to prevent 404 pages from being indexed by Google and other search engines, make sure that your webserver returns an actual 404 HTTP status code when a missing page is requested.†Google, 2018
QUOTE: “Use the robots.txt file on your web server to manage your crawling budget by preventing
crawling of infinite spaces such as search result pages. Keep your robots.txt file up to date.†Google, 2017
This will prevent the automatic creation of thin pages and could help prevent against negative SEO attacks, too. Read a beginner’s guide to Robot.txt files.
Use canonicals properly
QUOTE: “If your site contains multiple pages with largely identical content, there are a number of ways you can indicate your preferred
URL to Google. (This is called “canonicalizationâ€.)†Google, 2007
Use proper pagination control on paginated sets of pages
This will help with duplicate content issues.
QUOTE: “Use rel="next" and rel="prev" links to indicate the relationship between component
URLs. This markup provides a strong hint to Google that you would like us to treat these pages as a logical sequence, thus consolidating their linking properties and usually sending searchers to the first page. Google
How To Deal With Search Console Indexation Report Errors
How to deal with “Submitted URL marked ‘noindex’†and “Excluded by ‘noindex’ tag†notifications in Search console
Why are you creating pages and asking Google to noindex them? There is always a better way than to noindex pages. Review the pages you are making and check they comply with Google guidelines e.g. are not doorway pages. Check if technically there is a better way to handle noindexed pages.
How to handle “Page with redirect†notifications in Search console
Why do you have URLs in your sitemap that are redirects? This is not ideal. Review and
remove the redirects from the sitemap.
What does “ Indexed, not submitted in sitemap†mean in Search Console? It means Google has crawled your site and found more pages than you have in your sitemap. Depending on the number of pages indicated, this could be a non-issue or a critical issue. Make sure you know the type of pages you are attempting to get indexed, the page types your CMS produces.
How to deal with “Duplicate page without canonical tagâ€, “Alternate page with proper canonical tag†and “Submitted URL not selected as
canonical†notifications in Search console
Review how you use canonical link elements throughout the site.
How To Deal with “Crawl anomaly†notifications in search console:
QUOTE: “An unspecified anomaly occurred when fetching this URL. This could mean a 4xx- or 5xx-level response code; try fetching the page using Fetch as Google to see if it encounters any fetch issues. The page was not indexed.†Google, 2018
How To
Deal With Crawled – currently not indexed:
QUOTE: “The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.†Google, 2018
These could be problematic. You should check to see if pages you want indexed are included in this list of URLs. If they are, this could be indicative of a page quality issue.
Does Google crawl an XML sitemap and does it crawl the entire sitemap once it starts?
A question was asked in a recent Google Hangout by someone with a website indexation problem:
QUOTE: “How often does Google crawl an XML sitemap and does it crawl the entire sitemap once it starts?â€
An XML sitemap is inclusive, not exclusive.
QUOTE: “sitemap files do help us to better understand a website and to better figure out which parts of website need to be recrawled so specifically if you have information in like
the last modification date that really helps us to figure out which of these pages are new or have changed that need to be recrawled.†John Mueller Google, 2017
There will be URLs on your site that are not in the XML sitemap that Google will crawl and index. There are URLs in your XML sitemap that Google will probably crawl and not index.
QUOTE: “if you’re looking at the sitemap files in search console you have information on how many URLs are indexed from those sitemap files the important part there is that we look
at exactly the URL that you list in the sitemap file so if we index the URL with a different parameter or with different upper or lower case or a slash at the end or not then all of that matters for for that segment file so that that might be an issue to kind of look out there†John Mueller 2017
and
QUOTE: “in the sitemap file we primarily focus on the last modification date so that’s that’s what we’re looking for there that’s where we see that we’ve crawled this page two days ago and today it has changed
therefore we should recrawl it today we don’t use priority and we don’t use change frequency in the sitemap file at least at the moment with regards to crawling so I wouldn’t focus too much on priority and change frequency but really on the more factual last modification date information an RSS feed is also a good idea with RSS you can use pubsubhubbub which is a way of getting your updates even faster to Google so using pubsubhubbub is probably the fastest way to to get content where you’re regularly changing things on your site and you want to get that into Google as quickly as possible an RSS feed with
pubsubhubbub is is a really fantastic way to get that done.†John Mueller Google 2017
and
QUOTE: “so a [XML] sitemap file helps us to understand which URLs on your website are new or have recently changed so in the second file you can specify a last modification date and with that we can kind of judge as we need to crawl next to make sure that we’re not like behind in keeping up with your website’s indexing so if you have an existing website and you submit a sitemap file and the sitemap file has realistic change dates on
there then in an ideal case we would look at that and say oh we know about most of these URLs and here are a handful of URLs that we don’t know about so we’ll go off and crawl those URLs it’s not the case that submitting a sitemap file will replace our normal crawling it essentially just adds to the existing crawling that we do“. John Mueller 2017
Can I put my sitemap file into separate smaller files? Yes.
QUOTE: “Another thing that sometimes helps is to split the sitemap files
up into separate chunks of logical chunks for your website so that you understand more where pages are not being indexed and then you can see are the products not being indexed or the categories not being indexed and then you can drill down more and more and figure out where where there might be problems that said we don’t guarantee indexing so just because a sitemap file has a bunch of URLs and it doesn’t mean that we will index all of them that’s still kind of something to keep in mind but obviously you can try to narrow things down a little bit and see where where you could kind of improve that situation.†John Mueller,
href="https://www.youtube.com/watch?v=bjbaDP-P7PA&feature=youtu.be&t=41m4s">2017
The URL is naturally important in an XML sitemap. The only other XML sitemap you should really be concerned about is the DATE LAST MODIFIED. You can ignore the FREQUENCY attribute:
QUOTE – “we don’t use that at all ….no we only use the date in the [XML] sitemap file “ John Mueller, Google 2017
How many times a week is the index status data in search console updated?
It is
href="https://www.youtube.com/embed/byDfpbOTSMI?start=2436&end=2505&autoplay=0">updated 2-3 times a week.
Should you use sitemaps with last modified for expired content?
Why Doesn’t Google Crawl and Index My Website XML Sitemap Fully?
QUOTE: “So we don’t guarantee indexing. So just because something is in a sitemap file isn’t a guarantee that we will actually index it. It might be
completely normal that we don’t actually index all of those pages… that even if you do everything technically correct there’s no guarantee that we will actually index everything.†John Mueller, 2018
I have looked at a lot of sites with such indexation problems. In my experience the most common reasons for poor indexation levels of a sitemap on a site with thousands or millions of pages are:
doorway pages
thin pages
Pages that are almost guaranteed to get into Google’s index have one common feature: They have unique content on them.
In short,
if you are building doorway type pages without unique content on them, Google won’t index them all properly. If you are sloppy, and also produce thin pages on the site, Google won’t exactly reward that behaviour either.
QUOTE: “with regards to product pages not being indexed in Google again that’s something where maybe that’s essentially just working as intended where we just don’t index everything from them from any website. I think for most websites if you go into the sitemap section or the indexing section you’ll see that we index just a fraction of all of the content on the
website. I think for any non-trivial sized website indexing all of the content would be a very big exception and I would be very surprised to see that happen.†John Mueller, Google 2018
Google rewards a smaller site with fat, in-depth pages a lot more than a larger site with millions of thinner pages.
Perhaps Google can work out how much unique text a particular site has on it and weighs that score with the number of pages the site produces. Who knows.
The important takeaway is ‘Why would Google let millions of your auto-generated pages
rank, anyway?â€
QUOTE: “really create something useful for users in individual locations maybe you do have some something unique that you can add there that makes it more than just a doorway page“. John Mueller, Google 2017
Google Not Indexing URLs In Your Sitemap? Creating New Sitemaps Won’t Help
It is unlikely that modifying your XML sitemaps alone will result in more pages on your site being indexed if the reason the URLs are not indexed in the first place is quality-related:
QUESTION: “I’ve 100 URLs in a xml sitemap. 20 indexed and 80 non-indexed. Then I uploaded another xml sitemap having non-indexed 80 URLs. So same URL’s in multiple sitemap. . . Is it a good practice? Can it be harmful or useful for my site?â€
and from Google:
QUOTE: “That wouldn’t change anything. If we’re not indexing 80 URLs from a normal 100 URL site, that sounds like you need to work on the site instead of on sitemap submissions. Make it awesome! Make every page important!†John Mueller,
href="https://www.seroundtable.com/google-not-indexing-urls-sitemap-25300.html">2018
Most links in your XML Sitemap should be Canonical and not redirects
Google wants final destination URLs and not links that redirect to some other location.
QUOTE: “in general especially, for landing pages…. we do recommend to use the final destination URL in the sitemap file a part of the reason for that is so that we can report explicitly on those URLs in search console …. and you can look at the indexing information just for that sitemap file and that’s based on the exact URLs that you have there. The other reason we recommend doing that is that we use a sitemaps URLs as a part of trying to understand which URL should be the canonical for a piece of content so that is the URL
that we should show in the search results and if the sitemap file says one URL and it redirects to a different URL then you’re giving us kind of conflicting information.†John Mueller, Google, 2018
and
QUOTE: “actually the date the last modification date of some of these URLs because with that date we can figure out do we need to recall these URLS to figure out what is new or what is different on these URLs or are these old URLs that basically we might already know about we decided not to index so what I would recommend doing there is creating an XML sitemap file with the dates with the last modification dates
just to make sure that Google has all of the information that it can get.†John Mueller, Google, 2018
Read my article on managing redirects properly on a site.
Sometime non-canonical versions of your URLs are indexed instead
QUOTE: “I would recommend doing there is double-checking the URLs themselves and double-checking how they’re actually indexed in Google so it might be that we don’t actually index the URL as you listed in the sitemap file but rather a slightly different version that is perhaps linked in within your website so like I mentioned before the trailing slash is very
common or ducked up the non WWW(version) – all of those are technically different URLs and we wouldn’t count that for the sitemap as being indexed if we index it with a slightly different URL.†John Mueller, Google 2018
It is ‘normal’ for Google NOT to index all the pages on your site.
What Is the maximum size limit of an XML Sitemap?
QUOTE: “We support 50 megabytes for a sitemap file, but not everyone else supports 50 megabytes. Therefore, we currently just recommend sticking to the 10 megabyte limit,“ John Mueller, Google 2014
Google
wants to know when primary page content is updated, not when supplementary page content is modified – “if the content significantly changes, that’s relevant. If the content, the primary content, doesn’t change,then I wouldn’t update it.“
Why Is The Number Of Indexed URLs in Search Console Dropping?
Google has probably worked out you are creating doorway-type pages with no-added-value.
QUOTE: “The Panda algorithm may continue to show such a site for more specific and highly-relevant queries, but its visibility will be reduced for queries where the site owner’s
benefit is disproportionate to the user’s benefit. Google
Page Quality & Site Quality
Google measures quality on a per page basis and also look at the site overall (with site-wide quality being affected by the quality of individual pages.
Do no indexed pages have an impact on site quality?
Only indexable pages have an impact on site quality. You can use a noindex on low-quality pages if page quality cannot be
improved.
QUOTE: “If you if you have a website and you realize you have low-quality content on this website somewhere then primarily of course we’d recommend increasing the quality of the content if you really can’t do that if there’s just so much content there that you can’t really adjust yourself if it’s user-generated content all of these things then there there might be reasons where you’d say okay I’ll use a no index for the moment to make sure that this doesn’t affect the bigger picture of my website.†John Mueller, Google 2017
You should only be applying
noindex to pages as a temporary measure at best.
Google wants you to improve the content that is indexed to improve your quality scores:
