Last Updated on February 22, 2025 by Rakesh Gupta
Big Idea or Enduring Question:
-
What is a Meta Robots tag? What are Robots Meta Tags used for? How can you hide an Experience Cloud public page from search engines to prevent indexing?
Objectives:
After reading this blog, you’ll be able to:
- Understand Meta Robots Tags and how they affect search engine indexing.
- Use Meta Robots Tags to hide pages from search engines.
- Improve SEO for your Experience Cloud portal.
- And much more!
👉 Previously, I’ve shared several posts on effectively implementing key branding and SEO features for Experience Cloud sites. Why not check them out while you’re here?
- Step-by-Step Guide to Generate a Sitemap in Salesforce Experience Cloud
- Step-by-Step Guide to Configuring Custom Domains for Experience Cloud Sites
- Step-by-Step Guide to Configure Google Analytics™ for Experience Cloud Sites
- Step-by-Step Guide to Write SEO-Friendly Titles & Descriptions for Experience Cloud Pages
Business Use case
Olivia Bennett, a Junior Developer at Gurukul on Cloud (GoC), is part of a team working on building an Experience Cloud site for the company’s help portal. She branded the portal URL https://help.gurukuloncloud.com/ as described in this post and configured Google Analytics™ for Experience Cloud Sites. Now, Olivia has a better understanding of Meta Titles and Meta Descriptions and is able to add and verify them for Experience Cloud pages.
She met again with Jake Thompson, the team’s Webmaster, who asked whether the necessary adjustments have been made to hide the Thank You public page from search engine indexing and should not be discoverable in search engines. Olivia needs our help to figure this out, as she is still learning about SEO.
What Is a Meta Robots Tag?
A meta robots tag tells search engines how to crawl and index your webpage. It should be placed in the <head> section of your HTML. You can add your instructions for search engines inside the content attribute. Common values include:
- Follow: Allows search engines to crawl links on the page.
- Nofollow: Tells a crawler not to follow any links on a page or pass along any link equity.
- Index:Tells a search engine to index a page. Note that you don’t need to add this meta tag; it’s the default.
- Noindex: Tells a search engine not to index a page.
Meta Robots Tag SyntaxÂ
The meta robots tag helps control how search engines crawl and index your pages. Proper use can improve SEO and prevent unwanted pages from appearing in search results. The meta robots tag has two key attributes:
- name: Specifies which crawler should follow the instructions (e.g., robots for all crawlers or specific names like googlebot).
- content: Defines directives like index, noindex, follow, and nofollow.
For example, the following instructs all search engines not to index the page or follow its links:Â
Which Search Engines Support Meta Robots Tags?
Different search engines interpret meta robots tags in varying ways. While most major search engines support common directives like noindex and nofollow, some may not fully adhere to all instructions. Below is a comparison of search engine support for meta robots tags.
| Search Engine | Supports index/noindex |
Supports follow/nofollow |
Notes |
|---|---|---|---|
| Yes | Yes | Fully supports meta robots tags | |
| Bing | Yes | Yes | Adheres to standard directives |
| Yahoo | Yes | Yes | Uses Bing’s index, so it follows Bing’s rules |
| DuckDuckGo | Yes | Yes | Follows general web standards |
| Baidu | Yes | Partial | May not fully respect nofollow |
| Yandex | Yes | Partial | nofollow may not be strictly followed |
Different Ways to Add Meta Tags to Your Experience Cloud Site Pages
In Experience Cloud, you can add meta tags to optimize SEO and control search engine visibility. These tags help improve search rankings, enhance social media sharing, and ensure the right pages are indexed or hide pages from search engines. Here are two ways to do it:
Site-Wide Meta Tags in Experience Cloud
Site-wide meta tags help control how search engines index and display your entire Experience Cloud site. By adding these tags in Edit Head Markup, you can optimize SEO, improve social sharing, and ensure a mobile-friendly experience across all pages. This is useful for setting global rules, such as allowing search engines to index your site or defining a default description for search results.
Use Case: Site-Wide Meta Tags
For example, A company wants its Experience Cloud site to be fully indexed by search engines, ensuring that all pages are discoverable and that search engines follow internal links.
They add the following meta tag globally using Edit Head Markup in Experience Builder:
How to Add Site-Wide Meta Tags
To apply meta tags across your entire site, perform the steps below:
- Open Experience Builder.
- To add Use Edit Head Markup under Settings → Advanced in Experience Builder to apply meta tags across your entire site.
- Publish your changes.
Use Case: Page Sepecific Meta Tags
A company has a Thank You page that users see after submitting a form. This page should not appear in search results, as it is only relevant after form submission.
They add the following meta tag to the Thank You page using Edit Head Tags in Experience Builder:
Steps to Add Meta Tags to a Specific Page on Your Experience Cloud Site
Before we begin, it’s important to note that SEO is only supported on production orgs for Experience Builder sites that are either fully public or have some public pages. SEO is not supported on Developer Edition, sandbox, or trial orgs. Once your site is made public, search engines can index its public pages, making them discoverable in search results.
Now, let’s help Olivia add a meta tag to the Thank You page to instruct all search engines not to index it. She should follow the steps below:
- In Experience Builder, navigate to Pages → Page Settings.
- Select the Thank You page where you want to add the meta tag.
- Click on Edit Head Tags.
- Add the required meta tag in the <head> section as shown in the screenshot:
- Click Save to apply your changes.
- Once done, be sure to Publish the Experience Cloud site.
Things To Remember
- Previously Indexed Pages May Still Appear:Â If a page has already been crawled and indexed by search engines, adding
<meta name="robots" content="noindex, nofollow">does not guarantee immediate removal. You may need to request removal via Google Search Console. - Noindex Doesn’t Delete Pages: The
noindextag prevents search engines from displaying the page in results but does not delete it from your site or restrict direct access. - Nofollow Only Affects Crawling, Not Clicks:Â The
nofollowdirective stops search engines from following links on the page but does not prevent users from clicking and navigating to those links. - Changes Take Time to Reflect: Search engines don’t process meta tag changes instantly; updates depend on when the site is recrawled.
- Use Robots.txt for Additional Control:Â If you want to prevent search engines from crawling an entire section of your site, consider using the
robots.txtfile in addition to meta tags. I will cover this topic in my next blog post. - Meta Tags Apply Per Page: The
noindexandnofollowattributes only affect the page where they are added. For global rules, you must apply them site-wide using Edit Head Markup in Experience Builder.
Formative Assessment:
I want to hear from you!
What is one thing you learned from this post? How do you envision applying this new knowledge in the real world? Feel free to share in the comments below.
