If you’re reading this blog post, then chances are you are having a problem with your site not showing up on Google or another search engine – or you’re at least interested in preparing for this scenario.
So in continuation of our ongoing blog series about the five potential reasons why a website may not appear in Google Search, we’ll discuss the second culprit behind this frustrating problem:
- The site is too new and Google hasn’t had the chance to crawl it yet
- The site content is hidden from search engines
- The site content isn’t optimized for search engines
- The target keyword market is too competitive
- The site has been penalized by Google
Is Your Site Content is Hidden from Search Engines?
It’s common practice for developers to keep “staging sites” hidden from search engines. Websites and new pages need to be made available to a client for preview and testing before making it live to the rest of the world. At this point, functionality and content is still being tweaked and worked on, and you don’t want to risk the URL being indexed by Google (yet).
Developers are human too though, which unfortunately means we sometimes forget to remove these directives once the website goes live, and it can take weeks (or even months!) before anyone notices – at which point the developer faces an onslaught of angry emails and calls…
So whether you’re the poor developer groundhog in the image above, or one of the tigers hunting for a reason why your website is invisible to Google, we recommend reviewing these two common ways developers keep sites hidden from search engines so that you can easily spot the error and remove it immediately:
1. Meta “no index” tag
“Noindex” or “nofollow” meta tags are inserted on each page that developers wish to conceal from search engines, and are relatively easy to locate by viewing the page’s source code:
The noindex tag is typically found at the top of the page within the <head> tags and looks like this:
<meta name=”robots” content=”noindex”>
OR
<meta name=”robots” content=”noindex, nofollow”>
A noindex tag tells a search engine that white it can crawl the page, it cannot add the page into its search index. So any page with the noindex directive on it will not go into the search engine’s search index, and therefore will not be shown in the search engine results pages.
A nofollow tag also disallows search engines from crawling the links on that page, which means that any ranking authority the page has on SERPs will not be passed onto pages it links to. Simply put, any page with a nofollow directive on it will have all its links ignored by Google and other search engines.
If you don’t feel comfortable checking a page’s source code for noindex tags, don’t fret! There are plenty of Noindex Tag Checker tools out there that allow you to copy/paste your URL into a search box.
2. “robots.txt” file
The second way to check if your website or page content is being hidden from search engines is to review the robots.txt file on your site. The robots.txt file is where you can specify whether you’d like to block bots from a single page, a whole directory, or even just a single image or file. This file allows developers to work on an entire website or directory without having to add nofollow tags to every single page.
To view your robots.txt file, go to the home page on the site you wish to test and add “/robots.txt” at the end of the URL. It should look something like this:
https://www.your-website.com/robots.txt
If your robot.txt file is making your site or page content invisible to Google, then you might see something like this:
User-agent: * (means “all” search bots)
Disallow: / (means disallow “all” directories)
Or you might see some kind of variation, like these below. (Robots.txt can get pretty long!)
User-agent: Googlebot
User-agent: Yahoo! Slurp
User-agent: bingbot
Disallow: /subdir1/
Disallow: /subdir2/
Disallow: /subdir3/
Disallow: /page1.html
Disallow: /page2.html
Disallow: /page3.html
How to Make Your Site Visible to Google
If you find that your site is using either of these tactics listed above, it’s important you fix it immediately.
In most cases, all you’ll need is File Transfer Protocol (FTP) access to your site and basic HTML knowledge, but check with your server administrator to make sure you follow the right procedures – especially if you’re running a content management system (CMS).
If you’re running a WordPress site, the noindex tags can easily be removed site-wide by going to Settings > Reading > Search Engine Visibility and making sure the “Discourage search engines from indexing this site” box is NOT checked. Don’t forget to save your changes!
Keep in mind that you may not see results instantaneously. Your changes probably won’t kick in until the next time a search engine crawls your site. Depending how often you publish new pages to your website, this could take a few weeks.
Stay tuned for our next blog in this series for more possible reasons why your site is nowhere to be found on Google or another search engine. Thanks for reading!