More in this section
Forums / Deployment / robots.txt with azure deployments

robots.txt with azure deployments

The forums are in read-only mode. In case that you want to directly contact the Progress Sitefinity team use the support center. In our Google Plus group you can find more than one thousand Sitefinity developers discussing different topics. For the Stack Overflow threads don’t forget to use the “Sitefinity” tag.
2 posts, 0 answered
  1. Bil
    Bil avatar
    51 posts
    27 Aug 2012
    20 May 2015
    Link to this post

    Does anyone have an idea on how to add a robots.txt file to an azure deployment of Sitefinity for different environments. For DEV and TEST I don't want the search engines to be crawling my sites so I want a robots.txt file like this:

     User-agent: *
    Disallow: /

    However for production I would want them to crawl it.

    I'm not sure, other than adding the file to the web project, how to get it up to Azure. Adding it to the package is fine but how do you add a file based on the deployment?


  2. Atanas Valchev
    Atanas Valchev avatar
    414 posts
    04 Jan 2016
    25 May 2015
    Link to this post

    For the Sitefinity backend pages, you have the option to disable crawlers from Title and Properties - Allow external search engines to index this page and include in Sitemap.

    This option adds the following in the head:
    <meta name="robots" content="noindex" />

    To easily turn this on and off, you can create a widget and do this programatically. For example:
    var pageMan = PageManager.GetManager();
               var pages = pageMan.GetPageNodes().Where(p => p.RootNodeId == SiteInitializer.CurrentFrontendRootNodeId);
               foreach (var page in pages)
                   page.Crawlable = false;

    That way you can easily disable this on test before/after deploying. After you propagate to production, you can enable it with a click of a button.

    Atanas Valchev
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items
2 posts, 0 answered