+1-888-365-2779
Try Now
More in this section

Forums / Developing with Sitefinity / Updating Search Index Manually/Search Index Fields

Updating Search Index Manually/Search Index Fields

22 posts, 0 answered
  1. Michael
    Michael avatar
    3 posts
    Registered:
    28 Sep 2011
    28 Sep 2011
    Link to this post
    Hi,

    We are working through custom content types and publishing pipes with 4.2. I have two questions:

    1) How do I go about triggering an incremental update to a search index? I have an inbound pipe hooked up to the "SearchItemTemplate" pipe template and it works fine when reindexing through the UI. I can't figure out how to hook into the pipe, however, to add or delete an item individually. I've tried grabbing a handle to the pipe using PublishingSystemFactory.GetPipe(s) when executing the ContentManagerBase.Publish() method of the custom content but everything in the pipe seems to be null — pipesettings, PublishingPoint, etc.

    2) In Sitefinity 3.7 I knew how to add custom fields to the Lucene index so that I could do a targeted search. In 4.2 I'm not sure where to do that or if it's even possible. Somehow the inbound pipe is hooked up to Lucene using the SearchItemTemplate but I can't find a way to manipulate the fields that Lucene generates in the index.

    Thank you.


    Michael
  2. Milena
    Milena avatar
    75 posts
    Registered:
    15 Jul 2016
    04 Oct 2011
    Link to this post
    Hi Michael,

    1) There is no way to trigger an incremental update to a search index at the moment.

    In order to trigger a pipe you need to call method

    public virtual void Initialize(PipeSettings pipeSettings), pipeSettings can be retrieved from publishing point or via Publishing manager like this:

    pipeSettings = PublishingManager.GetManager(provider.Name).GetPipeSettings<SearchIndexPipeSettings>().Where(ps => ps.PublishingPoint.Id == pointGuid).FirstOrDefault();

    After the pipe is initialized you can call method

     public virtual void PushData(IList<PublishingSystemEventInfo> items) 

    I need to make proof of concept for extending search fields at Lucene and I need more time. I will provide you with a sample code when I am ready.

    Let me know if you still have issues with first point.

    Greetings,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  3. Milena
    Milena avatar
    75 posts
    Registered:
    15 Jul 2016
    04 Oct 2011
    Link to this post
    Hello Michael,

     About your second question:

    1) I created a custom field for news module via user interface with name "TestSearch".

    2) Inherited  ContentInboundPipe and overrided property PipeSettings.

    public class ContentInboundPipeNew : ContentInboundPipe
       {
           public override Sitefinity.Publishing.Model.PipeSettings PipeSettings
           {
               get
               {
                   var settings = base.PipeSettings;
                              var mappings = settings.Mappings.Mappings;
                   var customField = "TestSearch";
                   var customFieldMapping = PublishingSystemFactory.CreateMapping(customField, ConcatenationTranslator.TranslatorName, true, customField);
                   if(!mappings.Contains(customFieldMapping))
                     mappings.Add(customFieldMapping);
                  
                   return settings;
               }
               set
               {
                   base.PipeSettings = value;
               }
           }
       }

    3) Inherited SearchIndexOutboundPipe and overrided property PipeSettings.

    public class SearchIndexOutboundPipeNew : SearchIndexOutboundPipe
    {
        public override Sitefinity.Publishing.Model.PipeSettings PipeSettings
        {
            get
            {
                var settings = base.PipeSettings;
                                                   var mappings = settings.Mappings.Mappings;
                var customField = "TestSearch";
                var customFieldMapping = PublishingSystemFactory.CreateMapping(customField, ConcatenationTranslator.TranslatorName, true, customField);
                if (!mappings.Contains(customFieldMapping))
                    mappings.Add(customFieldMapping);
     
                return settings;
            }
            set
            {
                base.PipeSettings = value;
            }
        }
    }

    4) Replaced SearchIndexOutboundPipe and ContentInboundPipe with newly created pipes. I did this at Global.asax like this:

    protected void Application_Start(object sender, EventArgs e)
        {
            Bootstrapper.Initialized -= Bootstrapper_Initialized;
            Bootstrapper.Initialized += Bootstrapper_Initialized;
        }
     
        protected void Bootstrapper_Initialized(object sender, Telerik.Sitefinity.Data.ExecutedEventArgs e)
        {
            if (e.CommandName == "Bootstrapped")
            {
                this.RegisterCustomSearchFieldMapping();
            }
        }
     
        public void RegisterCustomSearchFieldMapping()
        {
           PublishingSystemFactory.UnregisterPipe(SearchIndexOutboundPipe.PipeName);
            PublishingSystemFactory.RegisterPipe(SearchIndexOutboundPipe.PipeName, typeof(SearchIndexOutboundPipeNew));
     
            PublishingSystemFactory.UnregisterPipe(ContentInboundPipe.PipeName);
            PublishingSystemFactory.RegisterPipe(ContentInboundPipe.PipeName, typeof(ContentInboundPipeNew));
        }

    These steps will ensure that added custom field is indexed at Lucene.

    In order to be able to search by field "TestSearch"  you need to make some further steps:

    1) You need to inherit class LuceneResultSet and override method ExecuteQuery:

    protected virtual void ExecuteQuery(bool tryToFix = true)
           {
               try
               {
                   var path = this.provider.GetCataloguePath(catalogueName);
                   if (Directory.Exists(path))
                   {
                       var defautlOperator = QueryParser.AND_OPERATOR;
                       var analyzer = this.GetAnalyzer();
                        
                       var parser = new MultiFieldQueryParser(new string[] { "Title", "Content", "TestSearch"}, analyzer);
                       parser.SetDefaultOperator(defautlOperator);
     
                       //query = QueryParser.Escape(query);
     
                       if (searcher != null)
                           searcher.Close();
     
                       searcher = new IndexSearcher(path);
                       Query queryObj = searcher.Rewrite(parser.Parse(query));
                       Query = queryObj;
     
                       //Search only for current language
                       if (AppSettings.CurrentSettings.Multilingual == true)
                       {
                           TermQuery languageQuery = new TermQuery(new Term("Language", CultureInfo.CurrentUICulture.Name));
                           TermQuery languageNullQuery = new TermQuery(new Term("Language", SearchProvider.FieldNullValue));
     
                           BooleanQuery filterQuery = new BooleanQuery();
                           filterQuery.Add(languageQuery, BooleanClause.Occur.SHOULD);
                           filterQuery.Add(languageNullQuery, BooleanClause.Occur.SHOULD);
     
                           QueryFilter languageFilter = new QueryFilter(filterQuery);
                                                    
                           this.hits = searcher.Search(queryObj, languageFilter);
                       }
                       else {
                           this.hits = searcher.Search(queryObj);
                       }
     
                       for (var iter = 0; iter < this.HitCount; iter++)
                       {
                           var doc = this.hits.Doc(iter);
                       }
                   }
               }
               catch (Exception ex)
               {
                   if (tryToFix && ex is ParseException)
                   {
                       this.query = this.TryFixQuery(this.query);
                       this.ExecuteQuery(false);
                   }
                   else
                   {
                       this.error = ex;
                       this.hits = null;
                   }
               }
           }

    2) You need to inherit LuceneSearchProvider class and override methods Find:
    public class LuceneSearchProviderNew : LuceneSearchProvider
        {
            public override IResultSet Find(string catalogueName, string query, int skip, int take, params string[] orderBy)
            {
                return new LuceneResultSetNew(this, catalogueName, query, skip, take, orderBy);
            }
     
            public override IResultSet Find(string catalogueName, string query, params string[] orderBy)
            {
                return new LuceneResultSetNew(this, catalogueName, query, orderBy);
            }
        }

    3) You need to replace LuceneSearchProvider with newly created provider via configuration.

    Let me know if you have any issues with achieving required scenario!


    Greetings,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  4. Casey
    Casey avatar
    25 posts
    Registered:
    22 Oct 2010
    06 Mar 2012
    Link to this post
    Milena,
    I'm trying to search by a custom field, "Brand", but no results are returned when i try it out. I copied the code sample that you posted (exchanging "TestSearch" with "Brand"). One part I wasn't sure of is configuring the new custom search provider. I'm using my code below (actually from another post). Should that configuration code work? Does configuring the search provider need to come before or after registering the new inbound and outbound pipes?

    <BR>
    void configureSearchProvider() {
          var section = Config.Get<SearchConfig>();
          foreach (var provider in section.Providers.Values) {
             if (provider.Name == "LuceneSearchProvider") {
                if (provider.ProviderType == typeof(LuceneSearchProvider)) {
                   provider.ProviderType = typeof(LuceneSearchProviderNew);
                   ConfigManager.GetManager().SaveSection(section);
                   break;
                }
             }
          }
       }

  5. Milena
    Milena avatar
    75 posts
    Registered:
    15 Jul 2016
    06 Mar 2012
    Link to this post
    Hello Casey ,

    you can replace the old search provider and verify that it is replaced via Sitefinity configuration interface.

    Under Administration > Settings > Advanced > Search > Providers> replace old LuceneSearchProvider.
    Screenshot attached. 

    Let me know if you still have any issues with customizing search.
    Thanks!

    Greetings,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  6. Casey
    Casey avatar
    25 posts
    Registered:
    22 Oct 2010
    06 Mar 2012
    Link to this post
    Thanks Milena,

    I registered the LuceneSearchProviderNew. But the search still isn't including the custom field, "Brand".
    I put a break point in the LuceneSearchProviderNew.Find method and can see that the LuceneResultSetNew is returned. However the breakpoint in LuceneResultSetNew.ExecuteQuery never gets reached.
    Do you have any ideas what might be wrong?

    Any help would be appreciated,
    Casey
  7. Milena
    Milena avatar
    75 posts
    Registered:
    15 Jul 2016
    07 Mar 2012
    Link to this post
    Hi Casey,

    I think that problem is at LuceneResultSet,  ExecuteQuery method declaration should be: 

    protected override void ExecuteQuery(bool tryToFix = true), in order to replace the behaviour,  otherwise old code is executed.

    Let me know if this helps! Thanks!

    Regards,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  8. Casey
    Casey avatar
    25 posts
    Registered:
    22 Oct 2010
    07 Mar 2012
    Link to this post
    thanks again Milena,
    That was the reason ExecuteQuery wasn't being called; I should have noticed that.

    So now my break point in ExecuteQuery is hit, but searching by my custom field "Brand" still doesn't work. If I search by Title or Content then the search works. It looks like the query is set correctly and i checked that products have the Brand term I'm searching for but still no results.

    I noticed that my breakpoint in my ContentInboundPipeNew.PipeSettings (which I copied from your post) never gets hit even though I register that pipe same as the outbound one. I don't know if that is an issue.

    Do you have any ideas why searching by Brand isn't working? Or know of anything else I should check?

    I posted the ExecuteQuery method below in case there is an issue there. It should be the same as the code you  posted here except we don't need the multilingual part yet and we're using the default "or" operator (but i tried it with "and" too and it still didn't work).

    public class LuceneResultSetNew : LuceneResultSet
    {
       private Exception error;
       private string query;
       private IndexSearcher searcher;
     
       public LuceneResultSetNew(LuceneSearchProvider provider, string catalogueName, string query, params string[] orderBy)
          : this(provider, catalogueName, query, 0, 0, orderBy) {
     
       }
     
       string catalogueName = String.Empty;
       public LuceneResultSetNew(LuceneSearchProvider provider, string catalogueName, string query, int skip, int take, params string[] orderBy)
          : base(provider, catalogueName, query, 0, 0, orderBy) {
          this.catalogueName = catalogueName;
          this.query = query;
       }
     
       protected override void ExecuteQuery(bool tryToFix = true)
       {
          try {
             var provider = SearchManager.GetManager("").Provider as LuceneSearchProvider;
             var path = provider.GetCataloguePath(catalogueName);
     
             if (System.IO.Directory.Exists(path)) {
                var analyzer = this.GetAnalyzer();
                var parser = new MultiFieldQueryParser(new string[] {"Title", "Content", "Brand" }, analyzer);
      
                if (searcher != null)
                   searcher.Close();
      
                searcher = new IndexSearcher(path);
                Query queryObj = searcher.Rewrite(parser.Parse(query));
                System.Diagnostics.Debug.WriteLine(queryObj.ToString());
                Query = queryObj;
                this.hits = searcher.Search(queryObj);
      
                for (var iter = 0; iter < this.HitCount; iter++) {
                   var doc = this.hits.Doc(iter);
                }
             }
          }
          catch (Exception ex) {
             if (tryToFix && ex is ParseException) {
                this.query = this.TryFixQuery(this.query);
                this.ExecuteQuery(false);
             }
             else {
                this.error = ex;
                this.hits = null;
             }
          }
       }
     
       public void Dispose() {
          if (this.searcher != null) {
             this.searcher.Close();
             this.searcher = null;
          }
       }
    }

  9. Milena
    Milena avatar
    75 posts
    Registered:
    15 Jul 2016
    08 Mar 2012
    Link to this post
    Hi,

    I tested with version Sitefinity 4.2 and I managed to index and search by custom field "brand" which is added to News module.
    If you do not break at ContentInboundPipeNew, this means that you didn't replaced the ContentInboundPipe,  therefore the "Brand" field is not indexed.

    I am providing sources of the solution, if this do not work for you let me know!

    Regards,
    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  10. Casey
    Casey avatar
    25 posts
    Registered:
    22 Oct 2010
    08 Mar 2012
    Link to this post
    thanks Milena,

    Will the sample you posted search different product types with a custom field "Brand"?

    Casey
  11. Milena
    Milena avatar
    75 posts
    Registered:
    15 Jul 2016
    08 Mar 2012
    Link to this post
    Hi,

    the provided sample will work with Sitefinity content types - news, events, blogs, etc.

    For product types you need to replace the product pipe with new pipe(inherit the ProductInboundPipe) and at the new pipe add additional mappings to product pipe for "brand" field, the same way it is done at ContentInboundPipeNew.

    Let me know if you have any issues.

    Regards,

    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  12. Casey
    Casey avatar
    25 posts
    Registered:
    22 Oct 2010
    10 Mar 2012
    Link to this post
    Thanks for your help Milena. I got searching products by custom field "brand" working.

    I registered a new product inbound pipe like you said. I also had to move the new product inbound pipe registration from the bootstrapper_initialized event handler to a later event because the default product inbound pipe wasn't registered yet when i was handling the bootstrapper intialized event.
  13. Casey
    Casey avatar
    25 posts
    Registered:
    22 Oct 2010
    10 Mar 2012
    Link to this post
    The only field that I am not able to search by that I need to is "Sku". Is there anything different I need to do in order to search by "Sku"?

    I noticed that product.DoesFieldExist("Sku") is true, but product.FieldValue<object>("Sku") throws and argument exception.
    No such persistent field known.
    Parameter name: nameOfPersistentField
    Actual value was Sku.

    Sorry for all the questions but I can't find information on these specifics anywhere else.
  14. Milena
    Milena avatar
    75 posts
    Registered:
    15 Jul 2016
    12 Mar 2012
    Link to this post
    Hi Casey,

    In order to be able to index and search by "Sku" product field the steps are the same:

    1) add "Sku" field to mappings at class that override ProductInboundPipe (ProductInboundPipeNew) and mappings of  SearchOutboundPipeNew

    2) LuceneResultSetNew add "Sku" field in order to be able to  search by new field:
    var parser = new MultiFieldQueryParser(new string[] {"Title", "Content", "Brand", "Sku" }, analyzer);

    I tested step1 and the product "Sku" field was indexed successfully.

    Regards,

    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  15. Casey
    Casey avatar
    25 posts
    Registered:
    22 Oct 2010
    12 Mar 2012
    Link to this post
    Thanks Milena for your help. Searching by Sku is working now too. I didn't change anything, but I did another Reindex; I guess when I Reindexed the time before something went wrong. Anyways appears to be working now. Thanks again.
  16. Beth
    Beth avatar
    25 posts
    Registered:
    20 Aug 2012
    16 May 2012
    Link to this post
    I've tried your code for 5.2500 but it doesn't seem to be working too well. I'm just trying to get the "Author" show in my search results.
  17. Milena
    Milena avatar
    75 posts
    Registered:
    15 Jul 2016
    17 May 2012
    Link to this post
    Hi,

    Can you give more information about the problem you have?

    Does the custom product pipe stopped to work at all after upgrade or just Author field is not indexed ?
    Is the indexing and search working for Sku field?

    Kind regards,

    Milena
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  18. Beth
    Beth avatar
    25 posts
    Registered:
    20 Aug 2012
    17 May 2012
    Link to this post
    Well the Author isn't getting indexed for content like News Items or Downloads, so I was your code might help me add it to the IResult Set, or is that the wrong strategy? 

    I don't need anything from Products, just the Author field for all content types to be indexed and seen in the search results.
  19. Beth
    Beth avatar
    25 posts
    Registered:
    20 Aug 2012
    17 May 2012
    Link to this post
    It seemed that this image didn't really help, you should change the ProviderType to Telerik.Sitefinity.Services.Search.Data.LuceneSearchProviderNew, not the name.

    After doing that and resaving the web/config the search seemed to work appropriately.
  20. Beth
    Beth avatar
    25 posts
    Registered:
    20 Aug 2012
    17 May 2012
    Link to this post
    However, How do you get the HighLighterResult to grab the Custom Results now?
  21. steve
    steve avatar
    10 posts
    Registered:
    18 Aug 2011
    18 May 2012
    Link to this post
    Hi Milena,
    I see you briefly mentioned how to initialise a pipe at the top of this thread:
    >>In order to trigger a pipe you need to call method
    >>public virtual void Initialize(PipeSettings pipeSettings), pipeSettings can be retrieved from >>publishing point or via Publishing manager like this:
     
    Would you please provide a more detailed code snippet?
    In my case, we have implemented a module for creating custom indexes.
    public class CoursesModule : ModuleBase...
    public class CoursesInboundPipe : PipeBase, IPushPipe, IInboundPipe
    But we need to reindex programatically on a nightly basis.
    My question is, how we sucessfully instantiate the custom pipe and trigger the reindexing process from the code-behind of custom control?

    I have this at the moment, but the publishing point is always null.
    var pipeSettings=PublishingSystemFactory.GetPipeSettings("CoursesInboundPipe");
    (Most of the pipeSettings look fine, but PublishingPoint is null.)
    var
    pipe = PublishingSystemFactory.GetPipe("CoursesInboundPipe");

    pipe.Initialize(pipeSettings);

    pipe.Initialise should set the PublishingPoint:
    public virtual void Initialize(Telerik.Sitefinity.Publishing.Model.PipeSettings pipeSettings)
            {
                this.PipeSettings = pipeSettings;
                this.PublishingPoint = PublishingSystemFactory.
                             GetPublishingPoint(this.PipeSettings.PublishingPoint);
            }

    When reindexing through Admin, the PipeBase Initialise method is passed a fully populated PipeSettings.PublishingPoint  and reindexing works fine.
    If I can properly instantiate the PublishingPoint, then I'm guessing that I could then use code similar to the following:

      PublishingManager.InvokeInboundPushPipes(point.Id, null);

    Thanks in advance....

    Steve



  22. Bonny
    Bonny avatar
    58 posts
    Registered:
    11 Nov 2016
    18 May 2012
    Link to this post
    Hello,

    steve:

    I answered your question in the new thread that you oppened, if you have any other questions, you can continue the discussion there, so we can track the conversation better.

    Beth:

    As I understand, you have custom control, that you want to use for visualizing highlighted results. You can register this control in the toolbox and use it on your pages instead of the build one. The search query can be retrieved from the query string and then you can search in your custom index and display results in the way you want. If I misunderstood you, please write us back.

    Regards,
    Bonny
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
22 posts, 0 answered