SEO Eblog by SEO Advantage, Inc.

Tips, tricks & musings from inside the SEO trenches

Categories

Home / Site Architecture / Reasons Why Your Site May Be Invisible to Google: Cause #2

Reasons Why Your Site May Be Invisible to Google: Cause #2

February 7, 2017 By Gaby Zapien 3 Comments

site hidden from google

If you’re reading this blog post, then chances are you are having a problem with your site not showing up on Google or another search engine – or you’re at least interested in preparing for this scenario.

So in continuation of our ongoing blog series about the five potential reasons why a website may not appear in Google Search, we’ll discuss the second culprit behind this frustrating problem:

  1. The site is too new and Google hasn’t had the chance to crawl it yet
  2. The site content is hidden from search engines
  3. The site content isn’t optimized for search engines
  4. The target keyword market is too competitive
  5. The site has been penalized by Google

Is Your Site Content is Hidden from Search Engines?

It’s common practice for developers to keep “staging sites” hidden from search engines. Websites and new pages need to be made available to a client for preview and testing before making it live to the rest of the world. At this point, functionality and content is still being tweaked and worked on, and you don’t want to risk the URL being indexed by Google (yet).

Developers are human too though, which unfortunately means we sometimes forget to remove these directives once the website goes live, and it can take weeks (or even months!) before anyone notices – at which point the developer faces an onslaught of angry emails and calls…

So whether you’re the poor developer groundhog in the image above, or one of the tigers hunting for a reason why your website is invisible to Google, we recommend reviewing these two common ways developers keep sites hidden from search engines so that you can easily spot the error and remove it immediately:

1. Meta “no index” tag

“Noindex” or “nofollow” meta tags are inserted on each page that developers wish to conceal from search engines, and are relatively easy to locate by viewing the page’s source code:

The noindex tag is typically found at the top of the page within the <head> tags and looks like this:

<meta name=”robots” content=”noindex”>

OR

<meta name=”robots” content=”noindex, nofollow”>

A noindex tag tells a search engine that white it can crawl the page, it cannot add the page into its search index. So any page with the noindex directive on it will not go into the search engine’s search index, and therefore will not be shown in the search engine results pages.

A nofollow tag also disallows search engines from crawling the links on that page, which means that any ranking authority the page has on SERPs will not be passed onto pages it links to. Simply put, any page with a nofollow directive on it will have all its links ignored by Google and other search engines.

If you don’t feel comfortable checking a page’s source code for noindex tags, don’t fret! There are plenty of Noindex Tag Checker tools out there that allow you to copy/paste your URL into a search box.

2. “robots.txt” file

The second way to check if your website or page content is being hidden from search engines is to review the robots.txt file on your site. The robots.txt file is where you can specify whether you’d like to block bots from a single page, a whole directory, or even just a single image or file. This file allows developers to work on an entire website or directory without having to add nofollow tags to every single page.

To view your robots.txt file, go to the home page on the site you wish to test and add “/robots.txt” at the end of the URL. It should look something like this:

https://www.your-website.com/robots.txt

If your robot.txt file is making your site or page content invisible to Google, then you might see something like this:

User-agent: * (means “all” search bots)

Disallow: / (means disallow “all” directories)

Or you might see some kind of variation, like these below. (Robots.txt can get pretty long!)

User-agent: Googlebot

User-agent: Yahoo! Slurp

User-agent: bingbot

Disallow: /subdir1/

Disallow: /subdir2/

Disallow: /subdir3/

Disallow: /page1.html

Disallow: /page2.html

Disallow: /page3.html

How to Make Your Site Visible to Google

If you find that your site is using either of these tactics listed above, it’s important you fix it immediately.

In most cases, all you’ll need is File Transfer Protocol (FTP) access to your site and basic HTML knowledge, but check with your server administrator to make sure you follow the right procedures – especially if you’re running a content management system (CMS).

If you’re running a WordPress site, the noindex tags can easily be removed site-wide by going to Settings > Reading > Search Engine Visibility and making sure the “Discourage search engines from indexing this site” box is NOT checked. Don’t forget to save your changes!

Keep in mind that you may not see results instantaneously. Your changes probably won’t kick in until the next time a search engine crawls your site. Depending how often you publish new pages to your website, this could take a few weeks.

Stay tuned for our next blog in this series for more possible reasons why your site is nowhere to be found on Google or another search engine. Thanks for reading!

Filed Under: Site Architecture Tagged With: google, organic search, search engine optimization, Site Architecture

About Gaby Zapien

Gaby drives creative direction, design and usability for our website development projects. She brings together ideas from all those around her in a continual search for the best ways to serve end users, keeping her team at the forefront of emerging technologies and web standards. When her busy family life, travel and pets allow, you might catch her experimenting with different creative media. A true artist at heart, her portraits have been featured in gallery exhibitions. She is known to maintain her Zen with meditation. Connect with Gaby on LinkedIn.

Get our blog by email

Contact us

CALL 800-366-1639

SEO Advantage, Inc.
3690 W Gandy Boulevard, Ste 444
Tampa, Florida 33611
United States
Site summary
SEO News

SEO Topics
Connect with us

About us

  • Search Engine Optimization Firm
  • About SEO Advantage®
  • Meet Your SEO Specialists
  • Company Press Releases
  • Search Engine Marketing Services
  • Guaranteed SEO Services
  • Contact SEO Advantage®
  • Internet Optimization and Marketing Company

SEO Programs

  • Basic SEO Programs
  • Monthly SEO Program
  • Total Advantage Program
  • Basic Advantage Program
  • Shopping Cart SEO
  • PPC Management Services
  • Search Engine Ranking & Reporting Service
  • SEO Trend Advantage

SEO Copywriting

  • SEO Copywriting Solutions
  • Specialized Search Engine Copywriters
  • Fresh Content Creation Service
  • e-Commerce Search Engine Copywriting
  • B2B Search Engine Copywriting

Search Friendly Web Design

  • Web Design Solutions
  • SEO Web Design & Development
  • Design Advantage Program
  • SEO Content Management System
  • Logos and Corporate Identity Solutions
  • Design Portfolio

Media Marketing

  • Buzz Marketing
  • Press Release Optimization
  • Social Media Marketing
  • Optimized Blog Copywriting
  • Directory Submission Service
  • Online Reputation Management

Resources

  • Helpful SEO Resources
  • Client Case Studies
  • SEO Knowledge Center
  • SEO Newsletter
  • SEO-E

Industry Specific Marketing

  • Cosmetic Surgery Web Site Marketing
  • Attorney Internet Marketing

Copyright © 2023 SEO Advantage® Privacy Policy