Month: March 2023

An evaluation of major competitors’ link profiles that examines their link-building strategies to gain insight is known as a competitor link analysis.

If you use some of the same tactics in your own link building, you might see better business results and higher search engine rankings.

You can find websites that frequently link to websites that are similar to yours by carrying out this kind of analysis.

If your competitors are doing well and have a lot of the same links, those links may be what help them rank.

Let’s look at how to conduct a fundamental audit. I will demonstrate later how to scale this procedure to obtain even more insights from larger data sets.

How to conduct a competitor link analysis in three steps

You can use that list later in the process if you already know who your competitors are.

If you need assistance, the Organic competitors report in Ahrefs’ Site Explorer will provide you with a list of websites that rank for the same keywords as you do.

if you do not yet have a website, you can possibly identify your rivals by conducting a search for a few key terms.

One more approach to doing this is to just add a couple of terms to Ahrefs’ Catchphrases Wayfarer and go to the Traffic share by spaces report, which will show you which locales are getting the most traffic for those terms.

2. Take your list of rivals and enter them into the Link Intersect report in Site Explorer to find common links.

This will provide you with a list of all your competitors’ referring domains and demonstrate how many competitors have links from those domains.

You can see the kind of links each domain has from each website by clicking the numbers below it.

3. Utilizing this data in a variety of ways includes identifying strategies employed by competitors.

You can work your way through the common links your rivals have that you don’t have, or you can look for patterns in the data to see the kinds of sites where rivals get links.

Niche websites, local directories, and other local websites may appear.

Congratulations, you’ve completed a basic link analysis of your competitors!

Bonus tip:

check your rivals’ generally connected content

Assuming you take a gander at the Best by joins report for a portion of your rivals in Site Pilgrim, you might see a fascinating substance driving connects to contenders that you in any case may not find.

Tools, studies, and even quotes frequently generate a lot of links to individual competitors.

When you look at the overlap of links from competitors, you might miss out on a lot of these links because they may only belong to one competitor.

You might be able to develop content and tools that are comparable to those used by a rival to increase the number of links to your own website.

You can use meta keywords in meta tags to provide search engines with additional information about a page’s content. They are hidden from visitors and can be found in the HTML source code of a webpage.

A meta keywords tag might look something like this:The majority of website builders and content management systems make it simple to add meta keywords, but is it worth the effort?

Do meta keywords still need to be used?

For at least a decade, Google hasn’t used the meta keywords tag to help rank websites. We are aware of this because Matt Cutts, the former Head of Google’s Webspam Team, stated in a 2009 video: […] In our search ranking, we don’t use the keywords meta tag.

Fast forward to 2020, and according to a recent Twitter poll conducted by Bill Slawski, the majority of SEO professionals no longer use the keywords meta tag.

However, you should keep in mind that SEO is about more than just Google, so there are a few good reasons to use the keywords meta tag.

1. It is possible that Yandex still makes use of it.

Yandex is the most popular Russian search engine on the Internet, and its market share in Russia is roughly equivalent to that of Google. Meta keywords “can be used when determining the page’s relevance to search queries,” according to their official documentation.

Does this imply that meta keywords have been shown to influence Yandex search rankings? No. However, the information is probably still correct given that that page has only been online since late-2018.

In a blog post from 2014, Bing stated that the meta keywords tag has “dead in terms of SEO value.” In 2020, Bing’s Head of Evangelism tweeted the same thing, effectively confirming this.

However, in 2011, Bing gave the impression that it might be utilized as a spam signal to assist in the identification of low-quality pages.

Whether that is still the case is unknown, but it is unquestionably not a ranking factor.”

Meta keywords have long been in the garbage heap of history, and we will ignore them directly,”

Given that, we can assume that it is either unimportant or not used at all.

Other uses for the keywords meta tag You probably already know that most people don’t need to use it, but there are a few things you can do with it

Numerous CMS’ and web designers make it simple to add meta catchphrases to pages, as do many free WordPress Search engine optimization modules.

The tag can easily be “repurposed” for use in an internal keyword tagging system as a result of this. Just set the tag as your objective watchword on each page.

For instance, if the target keyword for our page was “SEO tips,” it would look like this:If you do this on each page, it will be simple to determine whether you have already targeted a keyword in the future.

<Meta name=”keywords”

Simply slither your site with an examining device like Ahrefs’ Site Review, then, at that point, use Page Voyager to search for pages with your catchphrase in the meta watchwords tag.

There are three reasons why doing this may be beneficial: Contribute to stopping keyword cannibalism.

Keyword cannibalization occurs when multiple pages on a website target the same keyword. It has the potential to result in a wide range of issues, including undesirable pages outranking desirable pages.

Work should not overlap. There are probably a lot of people and teams working on similar projects if you work in enterprise SEO.

Having an internal system for tagging keywords helps keep different teams from going after the same keyword.

Primary versus secondary keywords

Going ahead, we’ll separate between two sorts of keywords

The primary subject of a page is the primary keyword, also known as the target keyword. Additionally, it is the only keyword for which a page should be optimized.

Any keywords that are closely related to the primary keyword that your page is aiming for are considered secondary keywords.

Secondary keywords include synonyms, subtopics, and variants of long-tail keywords. However, the best way to use secondary keywords for SEO is as relevant subtopics;

in this article, I will concentrate on this.So to summarize the distinction, in the event that the primary keywords is the subject of the book, you can utilize optional catchphrases as subtopics.

Why one essential catchphrase is sufficientThere are somewhere around three justifications for why.l?

Reason 1

Although it may seem obvious, a satisfactory explanation of this concept can quickly become complicated.

Every page needs one clearly defined topic. It’s likely best in the event that we take a gander at this from a client experience point of view.

People search for specific things online, so forcing them to look for those things on pages about multiple things—or worse—everything—is not a good idea.

Therefore, a single page addressing multiple subjects will not be very helpful.And because Google exists to assist individuals in finding specific things, it is more likely to display a page that focuses on a specific topic, or the most relevant one, than a page that tries to rank for multiple topics simultaneously.

Reason2 : Google is good at picking up close spelling variations and misspellings.

Have you ever noticed what happens when you misspell something in Google?Due to the fact that you probably had something else in mind when you typed that search term, Google will correct you like a grammar instructor would.

But what about synonyms and close variations?Same here. Without you intending to target each variation, Google will rank your page for keywords with the same meaning and intent.

It is aware that individuals search for the same thing in various ways.Let’s compare “website submission to search engines” and “submit website to search engines” for illustration.

In the Ahrefs Keywords Explorer SERP comparison, we can see that these two keywords yield nearly identical outcomes.

Reason 3 : If you only optimize for one keyword, you can rank for hundreds of other relevant keywords.

According to our research, the typical first-place page will also rank in the top 10 for nearly 1,000 other relevant keywords.This isn’t just true for keywords with a lot of repetition.

Even for keywords that are less popular, the pattern remains the same.Therefore, while concentrating on a single primary keyword is the most effective strategy, using multiple secondary keywords will produce the best results.

Why secondary keywords are necessary In a nutshell, you need pertinent subtopics to fully cover a topic. Secondary keywords are also a great way to find relevant subtopics.

That’s how Google operates, to use an oversimplified analogy. The system anticipates the searcher’s requirements and tries to provide the most useful result while ignoring other options.

An HTML attribute called the meta description is used to explain what a page is about. To provide additional context, it can be included in the title tag of the search results as a snippet.

How to write a meta description:

To quickly and easily create your meta descriptions, follow these five steps:

Expand on your title tag, match the intent of the search, use active voice, and be brief. Use the meta description.

1. Extend your title tag’s scope

A chance to elaborate further on that is provided by the meta description.Do not be concerned about inserting keywords here. It’s more important to make something that people will want to read and click on.

Where appropriate, make use of your unique selling points..

2. Match search intent

The “why” of a query is called search intent. To put it another way, when searching for your primary keyword, what are the majority of people looking for?

Do they intend to make a purchase?

Are they trying to find a particular website? Are they looking for a quick response to a question?

That length is currently 920px (160 characters) on a desktop computer and 680px (120 characters) on a mobile device.

Utilizing an integrated WordPress SEO plugin like Yoast is your best bet for keeping them to a reasonable length.

Depending on your audience, you should optimize for desktop or mobile users. If you usually write descriptions that are longer for desktop users, make sure to get to the point within a quarter of the description so that even mobile users can see it.

What can we learn from this? Be brief for clients, not web indexes. The limit on words is a good indication that there might be words that aren’t needed.

Step 1. Choose a PRIMARY keyword to focus on.

At Ahrefs, we usually recommend focusing on topics rather than keywords.This is due to the fact that the majority of pages rank not only for their “head” keyword, but also for a large number of related terms and variations (long-tail keywords).

In point of fact, we discovered that the typical page in the top ten will also rank well for approximately 1,000 other relevant keywords.

Example:

There are 400 searches per month for the keyword “best whey protein powder for women.”However, each of the top ten pages receives thousands of monthly organic visits.

This is due to the fact that each of these pages ranks for numerous additional long-tail keywords.

Wow, how cool is that?

However, when it comes to title tags, we advise optimizing primarily for a single primary “head” keyword. Additionally, possibly one or two long-tail keywords.

Step 2 will discuss that more.

Already have an idea of a keyword? Great.If not, do the following:Access the Keywords Explorer.Include a brief description of your post or page.

Step 2. Look for LONG-TAIL variations of your primary keyword.

You should always focus on a single primary keyword.However, it’s also common sense to focus on a few long-tail keywords.

Why? because ranking for your primary keyword may take some time. However, you can quickly begin receiving traffic from long-tail variations.

Additionally, these long-tail variations can frequently be incorporated without unnatural shoehorning into your title tag.

Long-tail keywords, then, are they? David McSweeney will explain:

The search queries known as long tail keywords have a very low individual search volume but a huge total demand for searches as a whole.

The “long tail” of the “search demand curve,” which is a graph that plots all keywords by their search volumes, is where the name comes from.

David also says that these keywords are typically longer and more specific than the keywords that are frequently searched for. (But not always.)

Example:Key phrase: ” SEO tips” (3,000 monthly searches);

First long-tail variant: SEO for novices” (100 monthly searches);Second long-tail variant: small business seo tips” (70 monthly searches) You get the point.

However, how can you locate long-tail keywords?Enter your primary keyword (such as “SEO Tips”) into the Keywords Explorer window.

After that, click “Phrase match” in the sidebar on the left.

Step 3. Draft your Fundamental title tag

Your subsequent stage is to draft an essential title tag.

Rules are as follows:

Concentrate on descriptivity:

It ought to precisely portray what the page/post is about and set peruser assumptions;

Keep it straightforward:

The final length of your title tag should not exceed fifty to sixty characters. You’ll add and repeating upon this draft as you deal with this aide — so pass on a space in excess.

Add your keywords here: In the title tag, be sure to include your primary keyword. Furthermore, if conceivable, sprinkle your long-tail variation(s) in there, as well.

That’s all there is to it.Remember that descriptiveness is your primary focus. You are not attempting to cram a lot of keywords into that.

Simply describe your page in a clear and concise manner. Furthermore, it should contain both your primary and long-tail keywords.

Step 4. Take a people at what’s extraordinary about your substance (and inform individuals)

Everybody that plays out a Google search is searching for something.However, that something varies from time to time.

The specific “qualities” that people look for in the search results will depend on the nature of the query.

However, there may already be overlap between the “qualities” that your content embodies and the qualities that people want to see if it was created around a specific query or topic.In your title tag, you only need to make these clear to potential visitors.

The following are five “qualities” that people tend to value, and how to make them clear in your title tag:

Depth/thoroughness:

Individuals love exhaustive, inside and out assets. It’s likely for this reason that content length and rankings clearly correlate.

Do not conceal a resource that is more comprehensive than other search results.

Add these words or phrases to your title tag to get people to click: final, “complete,” “definitive,” “study,” “step-by-step,” and so forth.

Quantity and Lists: List posts are very popular. For posts with a list format, we recommend that you always include a number in the title tag.

Speed/Quickness:

Overload of information is a real issue. This is your USP if your content is brief and to the point.

In your title tag, use phrases like these to “sell” your content: quick, simple, “…in X minutes,” “today,” “now,” and so forth.

Additionally, for product pages, try the following: free shipping” or “next-day delivery” (if, of course, you provide either option).

Freshness: Some queries, such as “SEO tips 2018,” require new results. For such queries, nobody is going to click on a 2012 result. Include words or phrases like these in your title tag to convey freshness: “… in 20XX,” “last updated January 18,” and so on.Brand:

A brand that people are familiar with and trust is more likely to get them to click on a search result. Are there any figures on that? No, but it seems pretty clear.)

Therefore, include your brand name in your title tag if you even have a semblance of industry recognition.

SEOs or writers who do a little research and rehash a lot of what other articles talk about create a lot of the content you find online. Some content may even adequately address a subject and address the primary points.

However, the articles all read the same, and the information is typically basic. Although this content might bring you some traffic from searches, it won’t impress anyone.

Content that shows real expertise is useful, in-depth, and memorable.Content that shows real expertise is useful, in-depth, and memorable.

You face a difficult task when you have brand-new content. It’s likely that many of the pages that are currently ranking have been there for years.

In fact, the average page in the top 10 ranking is more than two years old.They have gained a lot of links over time, and some of the pages may have recently updated content.

If you want to compete with these sites, you will need to put in a lot of effort and go above and beyond to produce high-quality content.

We’ll take a look at the following in this article:Let’s get started with creating great content and measuring its success.

How to Make Great Content

Producing content is a laborious process. However, even more effort, resources, and expertise are required to truly create something exceptional.

A great procedure and the right team are essential.Keyword research It’s like having a superpower to be able to see everything people search for.

You are hearing everything that people want to say. They’re providing you with a diagram of the substance they see as helpful, and that is precisely exact thing you ought to remember for your substance

Make a framework or content brief

Each article and video we make at Ahrefs begins with a framework. The main content we want to include is determined by utilizing the information gathered during the keyword research procedure.

Then, we add any important points we want to make, data we have, and any additional knowledge or insights that readers might find useful.

You might need to peruse a portion of the pages that generally rank for extra thoughts. Examine the language used and the topics that are discussed in connection with it.

Traditional keyword research may not yield results for many of these terms.In addition, there is a report in Keywords Explorer that can assist you in identifying related words and topics.

It’s known as the Likewise discuss report and shows you the sort of language that is utilized on these positioning pages.

You can get a sense of the language used in the industry, even for something you may not be familiar with.

When you write about your topic, you might want to use terms like this.When writing content for featured snippets, I suggest that you conduct additional research and either include the featured snippets that are currently eligible or even write a featured snippet that you believe has a chance of becoming the top snippet.

This is especially helpful for informational content.For informational queries, this is probably the section to rank first. You can see what is now qualified for a scrap and the sort of things that these pieces notice, alongside why one might be preferable over another.

Now all you have to do is create something better.Interior connections

The other thing that I like to incorporate is recommended inside connects that will go to the new piece of content.Have you ever had content that ranked well at first but dropped off quickly?

New content may initially rank well and inherit some signals. However, it will eventually need to rank using its own signals.Link to your content wherever it makes sense in your other content to give it the best chance of ranking.

With the Link opportunities report in Ahrefs’ Site Audit, we make this simple if your content already exists. For instance, there are nine opportunities to link to with appropriate anchor text in the recently updated content audit post that we wrote.

When starting a new blog, you might want to search for relevant terms in the text of other pages using Page Explorer in Site Audit. In this instance, you can see that there are 33 pages discussing content audits.

These might include a link to the post on the blog about content audits.In most niches, you can only get so far with content alone, and to rank higher, you probably need links. Later, you can target external links, but internal links are simpler to target.

For instance, when I withdrew links to some of our blog posts, all of the posts lost a lot of traffic and several positions for important terms.Let’s take a look at how to create the content with your outline in hand.

I would argue that one person is unlikely to be able to produce great content. Usually, a group of people with different skills is needed.An SEO professional, a specialist, a writer, and an editor who is involved in the process make up the team I typically want.

There are tools you can use for the various tasks (with the exception of expert insights), and it is possible for a single person to perform multiple of these duties.

However, the content will never be as good as what all of these people have created.For instance, I am acting as an SEO expert and writer for this article.

In this instance, I can easily assume the expert and SEO roles. In fact, a lot of the article is about how I work.Despite this, I have never been a good writer.

It is likely that someone who enjoys writing and has a gift for words will produce content much more quickly than I can. Additionally, it is likely that their content will be simpler to comprehend and more enjoyable to read.

Research SEOs are equipped with the knowledge and expertise necessary to conduct research, identify relevant topics, and produce content that is likely to achieve high search engine rankings.

However, the majority of SEOs won’t likely be writing about SEO, so they won’t be an expert.Even though I’ve known a lot of SEOs who know a lot about random niches their clients are in, they aren’t true experts because they aren’t directly involved in that niche.

At best, they can imitate what others have said, which might not even be true, to appear knowledgeable. People frequently copy and paste information from SEO articles without realizing that the information is incorrect.

This makes awful data spread to much more SEOs.An SEO may be able to identify some of the language or abbreviations used in a niche or related topics, and they may even be able to repeat some of the points they read;

however, they lack real expertise and unique insights.To create content for search, at least some fundamental research and processes are required. Many pieces of content are created but never seen.

As per our hunt traffic study, just ~4% of pages get in excess of 10 visits per month from natural pursuit.Experts have years of experience and knowledge in their areas of expertise.

More often than not, individuals making articles for search won’t have this inside and out information and experience.Experts rarely produce content because they are too busy being experts.

Your articles should really stand out, demonstrate your expertise, and provide additional insights.This requires more effort. In any case, in the event that you can talk with the master or have them survey the substance for precision and add their extra experiences, this can truly separate you from the group.

Support an expert who wants to write their own content by helping them with the necessary research and editing if you find a unicorn.Writing A good writer produces amazing work.

Reading their articles is a pleasure because they are concise, clear, and enjoyable. Writing well is both an art and a skill that can be learned by anyone.

When you write, you want to cover a subject thoroughly. Although a page may focus on just one or two primary terms or subjects, the most popular pages typically receive traffic from hundreds of different keywords.

Editing A competent editor is meticulous. They offer writers advice and feedback on how to improve their articles and communicate more effectively.

They accomplish significantly more than Grammarly or Hemingway can.How to determine the success of a piece of content There are numerous indicators you can use to determine success.

Measure what matters to you because there is no one right answer here.There are various use cases for various indicators. Rankings, traffic, or links might be of interest to SEOs.

However, in the end, businesses ought to measure things like revenue, qualified leads, and conversions.The nearer to cash you can get while detailing, the more effective that report is for partners.

On the off chance that the substance isn’t meeting your objectives or assumptions, you might need to return to and review the substance made for search intermittently to survey its exhibition and sort out whether or not/how you can further develop it.

Sitelinks are links that appear under some Google search results to other pages or sections of pages.

They make it easier for website visitors to quickly find relevant information. Sitelinks first surfaced in 2005 and were made public for the first time in 2006.

One of the most prevalent search options is sitelinks. In the US database of Ahrefs, they are present in 1.8% of SERPs. Sitelinks are present on nearly every brand name, and they also appear on numerous other queries, such as informational queries.

Sitelinks make up nearly 67% of all organic keywords for Wikipedia.org.Let’s examine a few common inquiries in greater depth:

The various types of sitelinks The various types of sitelinks Sitelinks have developed over time and display in many distinct ways.

The reasons why you want sitelinks Google frequently alters aspects like the number of sitelinks displayed and their appearance in the search results.

You might see some sitelinks with frames, with pictures, in a merry go round configuration, or ones that grow.

Here are various kinds of sitelinks that Google shows today.

Paid sitelinks This article will primarily focus on organic sitelinks, but I wanted to point out that sitelink extensions can appear on advertisements.

The ease with which you can control the text and URLs that are displayed with your ads is the primary distinction between paid sitelinks and other types of sitelinks.

The content and links of other kinds of sitelinks are determined by algorithms and are generated automatically.

Organic sitelinks are up to six sitelinks to other pages on your website and appear for mostly branded terms.

Organic one-line sitelinks Only appear on the first page of results Organic one-line sitelinks can appear on a variety of queries.

They typically display up to four sitelinks in a carousel, but there is a variant that displays more. Using fragment (#) links, these sitelinks can take users to other pages on a website or directly to content within those pages.

Organic sitelinks search box Using a sitelinks search box, users can search for and go directly to a website or app’s search results.

This just shows up for marked terms and is added naturally by Google. Adding structured data for a sitelinks search box to your homepage can help them understand your website better, but it is not required and does not increase the likelihood that a sitelinks search box will appear in search results.

Many Ahrefs tools and reports include filters for sitelinks as a SERP feature. In Ahrefs’ Site Explorer, for instance, you can find keywords for which you rank that also have sitelinks.One of the most prevalent search options is sitelinks.

In the US database of Ahrefs, they are present in 1.8% of SERPs. Sitelinks are present on nearly every brand name, and they also appear on numerous other queries, such as informational queries.

Sitelinks make up nearly 67% of all organic keywords for Wikipedia.org.Let’s examine a few common inquiries in greater depth:

The various types of sitelinks The various types of sitelinks Sitelinks have developed over time and display in many distinct ways.

The reasons why you want sitelinks Google frequently alters aspects like the number of sitelinks displayed and their appearance in the search results.

Some sitelinks might have outlines, images, carousels, or expandable versions. The various kinds of sitelinks that Google currently displays are as follows:

Paid sitelinks This article will primarily focus on organic sitelinks, but I wanted to point out that sitelink extensions can appear on advertisements.

The ease with which you can control the text and URLs that are displayed with your ads is the primary distinction between paid sitelinks and other types of sitelinks.

The content and links of other kinds of sitelinks are determined by algorithms and are generated automatically.

Organic sitelinks are up to six sitelinks to other pages on your website and appear for mostly branded terms.

They are only shown on the first page of results.

Why you need sitelinks

Sitelinks give you greater perceivability in the query items. In addition to making your website stand out from other search results, the additional links take up more space.

You can also help users get to content they might want faster by using sitelinks. This could be content on one page or another.

For instance, Google Search Console data indicates that the sitelinks, rather than the homepage, receive 12.9% of clicks on our branded search “ahrefs.”

A forward slash (“/”) that appears at the end of a URL, such as domain.com/ or domain.com/page/, is known as a trailing slash.

It is typically used to distinguish a file that does not contain a trailing slash from a directory that contains one.

On the other hand, these are guidelines, not requirements.Previously, a file did not have a trailing slash while a folder did.

A folder would indicate that there were additional files, and an index file (index.html, index.php, etc.) would typically be present. where the page’s content would come from when it loads.

Therefore, although domain.com/page/index.html would be displayed to users, the content would originate from there.

You would have the file name without the trailing slash when dealing with individual files.Nowadays, most systems’ URLs do not point to files.

A record in a database is the URL. Your server’s files are not even hosted by serverless systems.It’s possible to treat different URL structures differently.

Whether or not to use a trailing slash is largely a matter of personal preference. Let’s examine some typical scenarios.It doesn’t matter if the domain name ends with a trailing slash: domain.com = domain.com/

These URLs are treated the same regardless of which version you use.In addition to the trailing slash immediately following the root domain, all other trailing slashes will be treated as distinct URLs, including domain.com/page and domain.com/page/.

Adding a trailing slash to a file, such as.html,.php,.js,.css,.pdf,.jpg, etc., usually results in the following: The file won’t load.

This is due to the fact that the majority of systems will assume the file is a folder, and since there is nothing after this path, a 404 page will typically be returned.

Let’s now examine the effect on SEO.

SEO and trailing slashes You might want to make different choices based on how your systems operate.

A few typical scenarios you might encounter are listed below.As previously stated, if your content can be seen on both the trailing slash version and the non-trailing slash version of pages, the pages can be treated as distinct URLs.

The same content is displayed on URLs with and without trailing slashes. The usual concern in this situation is that different versions’ content will result in duplicate content.

This shouldn’t be a problem in most cases because a canonical tag probably will specify a preferred version.

Even in the absence of that, Google will typically select a preferred version for you in which the signals will be consolidated.

You can compel the URLs to your favored adaptation in the event that you need.

Regardless of whether you use a trailing slash, you should check for redirects, sitemaps, internal links, canonical tags, and other canonicalization signals.

indicate the version that you want indexed.When two systems share the same folder structure or when using certain A/B testing software, you may end up with a situation in which the version of a URL with and without the trailing slash shows completely different content.

This can happen when the trailing slash is present. In these situations, the ideal strategy would be to select one version to index and display to users, then to redirect the other version to it.

Hreflang More complicated setups involving hreflang may present challenges. The indexed version of the pages should be the destination of hreflang links.

When Google indexes a page, a canonical tag may point to a version of the page with a trailing slash, but hreflang tags may not be respected if they point to a version of the page without a trailing slash.

When deciding whether or not to use a trailing slash, the impact on reporting should be taken into consideration. You can, for instance, set up a domain or URL prefix property in Google Search Console.

When setting up a URL prefix property (like domain/folder), Google will still add a trailing slash if you don’t include one. Because domain.com/folder/ (with the trailing slash) is a level higher, all visits to domain.com/folder (without the trailing trash) will not be reported.

If your main pages do not contain the slash, Google Analytics (GA) encounters the same issue when attempting to perform a content drilldown by folder. Both the slash- and non-slash-encoded versions of your URLs that work may be reported to GA.

You can add a channel as displayed beneath to drive following cuts on the URLs in your examination reports if you have any desire to combine the information.

“Indexed, though blocked by robots.txt” indicates that your website’s robots.txt file prevented Google from crawling certain URLs.

Most of the time, this will be a simple problem caused by your robots.txt file blocking crawling.

However, there are a few additional conditions that could cause the issue; therefore, in order to efficiently diagnose and resolve the issue, let’s go through the troubleshooting procedure that follows:

Add a noindex meta robots tag to the URL to prevent it from being indexed and check to allow crawling—assuming it is canonical.

Google may still index a page that you block from being crawled because crawling and indexing are distinct processes.

Google won’t see the noindex meta tag on a page unless they can crawl it, but they might still index it because it has links.

A noindex meta robots tag should not be included if the URL canonicalizes to another page. Allow crawling and the proper canonicalization signals to pass and consolidate, including a canonical tag on the canonical page.

You must determine why Google cannot crawl the URL and remove the block if you want the URL to be indexed.The most probable reason is a slither block in robots.txt.

However, there are a few additional situations in which you might receive messages stating that you are blocked. Let’s go over these in the likely order you should look for them.

The robots.txt tester in GSC, which will flag the blocking rule, is the easiest way to identify the issue. Check for crawl blocks in robots.txt. Check for intermittent blocks.

Check for user-agent and IP blocks. Check for crawl blocks in robots.txt.You can locate the file at domain.com/robots.txt if you know what you want or do not have access to GSC.

Our robots.txt article has more details, but you probably want a disallow statement like:Abstain: /It could block everyone or mention a specific user agent.

If your website is brand-new or just launched, you should look for:

Agent-user: *Abstain: /Can’t find a problem?

It’s conceivable that somebody previously fixed the robots.txt block and settled the issue before you’re investigating the issue.

That is the ideal situation. However, you may have an intermittent block if the issue appears to be resolved but returns shortly thereafter.

How to fix It is necessary to get rid of the disallow statement that is causing the block.

The technology you use determines how you go about doing this.WordPress If the problem affects your entire website, you probably checked a setting in WordPress that prevents indexing.

This error is common on new websites and websites that have been moved.

Follow these moves toward check for it:

Web index Perceivability is uncontrolled.

WordPress with Yoast To remove the blocking statement, you can directly edit the robots.txt file with the Yoast SEO plugin.Click Tools, Yoast SEO, and File Editor, WordPress with Rank Math. Like Yoast, Rank Math lets you directly edit the robots.txt file.

Edit robots.txt FTP or hosting FTP or hosting If you have FTP access to the site, you can directly edit the robots.txt file to remove the disallow statement that is causing the problem.

Click Rank Math. Click General Settings. Click Edit robots.txt You might also get access to a File Manager from your hosting provider that lets you directly access the robots.txt file.

Check for intermittent blocks Because the conditions that are causing the block may not always be present, intermittent issues can be more difficult to resolve.

What I’d suggest is really taking a look at the historical backdrop of your robots.txt document. For instance, the GSC robots.txt tester has a dropdown menu that lets you select previous versions of the file and view their contents.

The robots.txt files for the websites they crawl can also be viewed in the history of the Wayback Machine on archive.org. You can see what the file contained on any given day by clicking on any of the dates for which they have data.

Alternately, you can make use of the Changes report’s beta version, which lets you quickly and easily compare two versions’ content.How to fix It depends on what is causing the problem how to fix intermittent blocks.

A shared cache between a test environment and a live environment, for instance, is one possibility. The robots.txt file may contain a blocking directive when the cache from the test environment is active.

Additionally, the site might be crawlable when the live environment’s cache is active. In this scenario, you should either split the cache or remove.txt files from the test environment’s cache.

Check for user-agent blocks When a site blocks a particular user agent, such as Googlebot or AhrefsBot, it is called a “user-agent block.” To put it another way, the website is blocking the user agent that is associated with a particular bot.

If changing your user-agent allows you to view a page normally but blocks you from doing so, this indicates that the user-agent you entered is blocked.

How to fix it

Unfortunately, you won’t know how to fix this one unless you know where the block is. .htaccess, server configuration, firewalls, CDN, and even something your hosting provider controls that you may not be able to see can all block bots.

Contacting your CDN or hosting provider to inquire about the source of the block and the best course of action might be your best option.

Check for IP blocks If you have verified that you are not blocked by robots.txt and have ruled out user-agent blocks, then an IP block is probably the problem.

Finding solutions to IP block issues is difficult. Similar to user-agent blocks, your best bet may be to contact your CDN or hosting provider and inquire about the source of the block and its resolution.

Google uses a web crawler called Googlebot to gather the necessary data and construct a web index that can be searched.

Googlebot has news, images, and video-specific crawlers in addition to desktop and mobile crawlers.Each crawler is identified by a distinct string of text known as a “user agent,” and Google uses multiple crawlers for various tasks.

Googlebot sees websites in the same way that users would in the most recent Chrome browser because it is evergreen.Numerous machines host Googlebot.

On websites, they decide how quickly and what to crawl. However, in order to avoid overwhelming websites, they will slow down their crawling.

Let’s take a look at how they created an internet index.How Googlebot crawls and indexes the web It goes over this once more and looks for new links or changes to the page. In Google’s index, the content of the rendered pages is stored and searchable.

It returns any newly discovered links to the URL bucket for crawling.How to control Googlebot Google provides a few options for controlling what is crawled and indexed.

How to control what is crawled Robots.txt:

You can control what is crawled with this file on your website.A link attribute or meta robots tag called “nofollow” indicates that a link should not be followed. It may be ignored because it is only a hint.

Change your crawl rate: You can slow down Google’s crawling with this tool in Google Search Console.Control over indexing: Delete your content: If you delete a page, there is nothing for the index to look at.

The drawback of this is that nobody else can get to it either.

Restrict access to the content. Since Google does not sign in to websites, it will be unable to view the content if it is protected with a password or authenticated in some other way.

Noindex: In the meta robots tag, type “noindex” to tell search engines not to index your page.

URL removal tool: The name of this Google tool is a little misleading because it hides the content for a while. This content will still be seen and crawled by Google, but the pages will not be listed in search results.

Robots.txt (only images): If you prevent Googlebot from crawling your images, they won’t be indexed.Is it the genuine Googlebot?

Some malicious bots and SEO tools will pretend to be Googlebot. They might be able to access blocked websites thanks to this.In the past, Googlebot verification required performing a DNS lookup.

However, Google recently made it even simpler by providing a list of public IPs that you can use to confirm that the requests are coming from Google. This can be compared to the information in the server logs.

A “Crawl stats” report is also available in Google Search Console. The report gives you a lot of information about how Google is crawling your website by going to Settings > Crawl Stats. You are able to see which Googlebot is accessing which files and when.