Don’t Let Your Blogger Blog Disappear from Google: The Quick SEO Fix You Need

Google Search Console robots.txt Photo Gemini

Have you ever checked your Google Search Console and seen the dreaded “Blocked by robots.txt” error? 😨 It’s a common problem that can tank your blog’s visibility, but the fix is simple, even for Blogger users.

This guide will teach you the difference between two key SEO tools—robots.txt and noindex—and show you how to set up your Blogger blog for success in just a few clicks.

The Bouncer vs. The Librarian: robots.txt vs. noindex

The biggest mistake new bloggers make is confusing these two powerful SEO tools. Here’s a simple way to remember them:

• robots.txt is the Bouncer: It’s a file that tells search engine crawlers which parts of your site they are allowed to visit. Think of it as a set of rules for managing traffic. You use it to tell bots to stay away from low-value pages like your internal search results or admin sections. But here’s the crucial part: if a page is blocked by robots.txt, Google might still index the URL if it finds links to it from other sites. This leads to a generic, ugly listing in search results with no title or description.

• noindex is the Librarian: This is a meta tag you place on a specific page that tells a crawler, “You can come in and read this page, but don’t add it to your library (the search index).” This is the correct way to hide a page from Google search. A search engine must be able to crawl the page to see the noindex tag.

This is why you should never use robots.txt to try and hide a page from search results. Instead, use the noindex tag, and make sure robots.txt is not blocking that page.

The Best Blogger Settings for SEO Success

Blogger makes this easy by giving you a simple dashboard to manage both. Here are the optimal settings you should be using for your blog.

1. Custom robots.txt

On Blogger, this is where you manage the “bouncer” rules. The best practice is to block internal search pages to avoid cluttering search results with duplicate content.

How to set it:

1. From your Blogger dashboard, go to Settings.

2. Under Crawlers and indexing, enable Custom robots.txt.

3. Paste this code into the text box:

   User-agent: *
   Disallow: /search/
   Sitemap: https://www.blowthetruth.com/sitemap.xml
   Sitemap: https://www.blowthetruth.com/sitemap-pages.xml
   

   (Replace the sitemap URLs with your own and see the change in performance in a few weeks, this is much better if you blog post is full of words, Category and tags are important for Blogger formats.)

This file tells all crawlers (User-agent: *) to stay out of your internal search pages (Disallow: /search/), but it allows them to crawl everything else by default. It also provides your sitemaps, which helps Google find all your posts and pages.

2. Custom robot header tags

This is where you set the “librarian” rules. Blogger lets you apply these rules to different sections of your blog.

How to set them:

1. From your Blogger dashboard, go to Settings.

2. Under Crawlers and indexing, enable Custom robots header tags.

3. Click on each of the three options and apply the recommended settings below.

• Home page: Select all and noodp. This tells search engines to index your home page and follow its links.

• Archive and search page tags: Select noindex and noodp. This is the key to preventing duplicate content issues. It tells Google not to index your label, category, and archive pages, so your blog posts are the only content showing up in search.

• Posts and pages tags: Select all and noodp. This ensures that all of your individual blog posts and static pages (your most important content) are fully indexed and discoverable.

Final Thoughts

By taking just a few minutes to properly configure these settings, you can ensure that your blog is not only visible to search engines but is also being crawled and indexed in the most efficient way possible. It’s a simple but powerful step toward better SEO for your Blogger blog.