+1-888-365-2779
Try Now
More in this section
Categories
Bloggers
Blogs RSS feed

Extending Sitefinity Search and searching by category

by Svetla Yankova

Overview

Sitefinity Search, true to the Sitefinity mantra, provides not only powerful set of features and a user friendly UI to manage the functionality, but also an interesting extension point that gives developers freedom to make all necessary  tweaks to meet any requirements.

In this blog post we are going to discuss the different extension points that the search feature provides and illustrate them in a sample that uses the different hooks in the Search Utility - index, search widget and search results widget.

The example

To illustrate the ideas discussed in this blog post we are going to cover a code sample. In this sample we are going to achieve searching by category. This consists of:

  • Including categories in our search index document and our search results
  • Extending the search box to include a dropdown for our existing categories
  • Extending the result widget to pass a query to Lucene that filters by the category 



But first lets get down to some of the core concepts of Sitefinity search and how it can be extended.

Extending Search

In terms of usability, Sitefinity search gives administrators and business users the ability set up very sophisticated search functionality. But it also goes beyond this. It gives developers the ground to be creative and extend the search functionality based on ideas and requirements that may not necessarily fit inside a box and we are going to discuss exactly those extension points in this post. To get in a bit more details on this let us first talk about the engine that Sitefinity Search is based on.

Sitefinity and Lucene .NET

Sitefinity provides its own implementation of the increasingly popular Lucene .NET search engine. A few benefits of this could be mentioned:

  • The search utility provides support for the immense list of great features that Lucene provides including scalability, performance, efficient search algorithms, fielded searching, simultaneous updates and searching, powerful query types etc.
  • The search utility is entirely in the hands of the Sitefinity development team and there is no need to rely on a third party search appliance that may or may not provide extension points. This from one perspective allows us to create great features like content aware search, search in documents, search in custom fields, search in custom modules and so on. 
  • On the other side it gives you the capability to utilize our extension points and meet any requirements at hand. With Lucene Sitefinity can expose hooks to the search functionality through different types of extension points allowing developers to hook into the engine and customize the search process to very powerful extent.
  • Sitefinity Search does not require additional licensing or rely on expensive search services and appliances

Core components of search and indexing

Each search engine essentially consists of a crawler(spider),an index and a search box. Crawling is really not the right term for built-in search engines as the CMS lets the index know about the changes and no crawling is needed.

Indexing is the process of extracting plain text from the CMS items (pages, blog posts, news) and saving it in an optimized structure for fast search in these contents.

Sitefinity creates and writes to index files that are stored in the file system – in the App_Data folder of the Sitefinity site.

What actually gets indexed?

question markThe fields that are getting indexed are by default Title, Content, Summary, ContentType, OriginalItemId, Link, DateCreated, PipeId, Language, IdentityField. This is enough to resolve each content item, but there is a reason we are actually indexing stuff using Lucene instead of querying the database – performance. Adding additional fields is supported out of the box


How the search works

Now that we have grounding on what the engine around the search indexes is, let’s look at the search process itself. Each time a user creates content (e.g news, events, lists item, custom content, page) the publishing system kicks in.

When you publish you practically invoke a content pipe that persists data to the publishing point. In more simple terms Sitefinity not only writes data to the database but also exposes this data and ask other systems – hey do you want to do your thing with that data, without bothering me with the details.  The point of asking this question is in technical terms the publishing point where thereafter the search outbound pipe gets all the data it wants and writes into the Lucene indices. Much in the same way the RSS pipe takes this data to expose an RSS feed or the twitter pipe...well...tweets.

From this point on things are in the court of the Search Pipe to take Sitefinity content and translate it to something Lucene can understand. The pipe sends all the needed data to the Search Provider which then writes to the Lucene files. The search index technically is a set of segmented binary files and you can think of it as the physical representation of the content item, with all the relevant fields persisted. The files themselves consist of a bunch of binary data that would probably not tell you much if you open it up in notepad, but you can think of it as key:value pairs for field:indexed content.

So as we keep publishing and editing content in Sitefinity those search indexes are automatically updated for us. Or, when we click reindex this is done in batch. 

the process of writing to the search index
How Sitefinity writes to the search index

What happens when we search

The search box widget is probably the simplest widget you can think of, it simply points the URL to a search results page.

There the search results widget is really the one talking to the Sitefinity APIs – the API here is the Search Service that passes the query to Lucene. So if we were to look at the anatomy of search those are the 3 extensible components that we have as well as the different extension points exposed by them



What parts can be extended?

Now that we have a better understanding of the entire Search infrastructure, a logical question that might arise is where can this be extended.

1)      Presentation – the search box and the search result widgets are based on a widget template that can be modified to display practically anything: fields, images, promoted searches. Additionally code behind can be added to this results widget. Since Lucene gives us back enough information about the content items that are currently indexed (outside of the indexed fields there is also metadata like the item GUID, URL, default page and type that in essence give you everything you need to extend this infinitely.

2)      Fields that are indexed. Sitefinity gives you the power to define custom fields and custom modules. These are not only part of the search infrastructure but you can also easily define where you want Sitefinity to search. For more information, check this blog post

3)      Search Queries. The queries you type in have the full power of the Lucene framework, which means:

  • Wildcards (Party in New * returns Party in New York and Party in New Mexico) ,phrased queries (“new york”)
  • Proximity queries ("foo bar"~4 means foo and bar are 4 words away)
  • Keyword matching (title:"foo bar" AND body:"quick fox")
  • Boolean conditions(title:Sitefinity AND content:Party)
  • Boosts (title:foo OR title:bar)^1.5 (body:foo OR body:bar) means the title will carry heavier weight)
  • etc.

Type in any of those to experiment how they relate to the search results. In the sample we discuss an elegant way that achieves keyword matching

4) The search input. You can manipulate the query that is passed to the results page and ultimately to Lucene, and by doing this you can easily add any type of functionality and logic behind search. In the code sample we are going to explore this option

5) Using the Search Service independently. The search service exposes public APIs which you can use to search for items in your own custom development. A great example can be seen in this Autocomplete blog post

Code Sample Walkthrough

In this code sample we are setting up search by category. Here are the steps that are involved in the initial preparation

  • Set up search to index the categories field. Reindex afterwards to have it index all content.
  • Add the project attached to this blog post to your solution and add reference SitefinityWebApp to it. Fix the assembly references if you are running a version different than Sitefinity 5.2
  • Register both the new search box and search results widget in the toolbox using Sitefinity Thunder
  • Add a virtual path for our search box widget
  • Just for kicks make the search results widget also highlight the category field using the Advanced Options by having Title,Content,Categories as search fields
  • Now kindly ask your users type in AND Categories:Community every time they search.
Kidding. Although this would actually work, because of the way you can query Lucene. But we will quickly build a great looking UI for our customers to choose from categories and submit the search. Here is what happens in the code sample:

In the project we have added the following tweaks to the search box:
  • A ComboBox displayed on the template(might as well be any other control)
  • This ComboBox is data bound to the categories on InitializeControls.
  • The Javascript takes the category selected and sends an additional query parameter to the search results page.

We have also added logic for modifying the search results.

This is actually the fun part – we hook directly into the Lucene and specifically in the place where Lucene does it’s query building. By inheriting the SearchResults widget you can override the BuildSearchQuery method. Here the code sample provides a nice extension method that generates the search query based on terms, AND/OR conditions etc. and it is configured explicitly to support categorized search as well. The categorized sample is only one of many examples you can explore with this mechanism to add any additional logic to your search query.

     var queryBuilder = service.CreateQueryBuilder();
            if (category != null)
            {
                queryBuilder.CreateGroup(QueryOperator.And);
            }
 
...
 
            if (category != null)
            {
                queryBuilder.AddTerm("Categories", HttpUtility.UrlDecode(category));
                queryBuilder.CloseGroup(); // close the category group
            }
 
            var compiledQuery = queryBuilder.GetQuery();


You can download the code sample below:

Search By Category code sample

Happy Searching!

Sitefinity-CMS-trial-blogs-banner1

22 comments

Leave a comment
  1. Darian Dec 07, 2012
    Hi Svetla,

    I'm trying to change the combobox for a treeview, to show the hierarchy of categories and the user can select multiples of them. The problem I'm facing it's in the javascript, I have no clue of where to change the combobox client object for the treeview. Could you please give me hint?

    Darian
  2. Tim Dec 10, 2012
    Hi Svetla

    We've followed your post, and discovered it only works for the basic built-in modules. Sitefinity search seems unable to index any choice/multichoice/categories field on Dynamic modules. This is a huge showstopper for us.

    For example, we have a Supplier entity built with the Module Builder and search is not able to index our Categories taxonomy nor State dropdown fields. 

    Don't know whether to laugh or cry :)
  3. Svetla Dec 10, 2012
    @Darian Here you would just change the control in your template, the type in your cs file and within AddScriptDescrptors() you can change the descriptor added for the comboBox to this:

     descriptor.AddComponentProperty("treeView", this.Treeview.ClientID);

    From here on out you can use the treeView's client side APIs. One think that is always good to make sure of whenever you change code related to script descriptors is that all your names match so that for instance all the getters and setters in the javascript are correctly renamed to get_treeView and set_treeview, the variable is called treeview etc.

  4. Svetla Dec 10, 2012
    Hi Tim

    Thank you very much for letting us know about this and I apologize for the inconvenience caused!  We are investigating the alternatives and will definitely update you know as soon as possible with a potential solution that would help you finish the task at hand.

    Thank you for your patience!

    Svetla
  5. Darian Dec 20, 2012
    Hi, Svetla

    I change the combobox for a treeview (in the template ascx, the CustomSearchBox class, and in the js file replace every combobox word for treeView), but in there is a client side error:

    "TypeError: this._treeView.get_selectedNodes is not a function"

    My getLocation function is as follow:

    getLocation: function () {
            var query = this._searchTextBox.value;
            var category = [];
            var selectedNodes = this._treeView.get_selectedNodes();
            for (var i = 0; i < selectedNodes.length; i++)
                category.push(selectedNodes[i].get_text());

            var separator = (this._resultsUrl.indexOf("?") == -1) ? "?" : "&";
            var indexCatalogue = String.format("{0}indexCatalogue={1}", separator, Url.encode(this._indexCatalogue));
            var searchQuery = String.format("&searchQuery={0}", Url.encode(query));

            for (i = 0; i < category.length; i++)
                var categoryQuery = String.format("&category={0}", Url.encode(category[i]));

            var wordsMode = String.format("&wordsMode={0}", this._wordsMode);
            var url = this._resultsUrl + indexCatalogue + searchQuery + categoryQuery + wordsMode;

            return url;
        }

    I can not think where it's wrong my code.
  6. Abhilash Mar 04, 2013

    The 1st screenshot shown in Code Sample Walkthrough, I don't see that in the search index screen. Where exactly do I need set this? I am using sitefinity 5.2

    Thanks,

    Abhilash

  7. Svetla Mar 04, 2013

    Hi Abhilash, 

    You will see this once you edit/create an index and toggle the advanced section at the bottom. This was introduced in Sitefinity 5.1 so it should be there.  here is a screenshot: http://screencast.com/t/dfbIHGMx

     

    Let me know if it worked.

    Svetla

  8. Andrew Mar 08, 2013

    Hi Svetla,

    Thanks for this walkthrough, it was very helpful. My question is whether there is a solution to Tim's question about indexing multichoice fields in custom modules. I know multichoice selections are stored differently (choice array), so I am just wondering if it is possible to search against them.

    Thanks,

    Andrew

  9. Robby Desmond Jun 20, 2013
    Presentation – the inquiry box and the output widgets are dependent upon a widget arrangement that could be changed to showcase basically anything: fields, pictures, advertised pursuits. Moreover code behind could be <a href="http://fetelenorocoase.com/">fetele norocoase</a> added to this effects widget. Since Lucene gives us back enough qualified information about the substance things that are as of now recorded (outside of the recorded fields there is additionally metadata like the thing Guid, Url, default page and sort that generally give you everything you have to expand this boundlessly.
  10. Perera Jun 21, 2013
    I can't unzip the code file attached(Search By Category code sample) let me know the reason.
  11. Svetla Jun 25, 2013
    Hi Perera.
    I am able to extract the widget with no problem, but here is another temporary dropbox link you can try on your end for the same sample, on this time in a rar file. Let me know if it works for you:

    https://dl.dropboxusercontent.com/u/20795800/Search%20By%20Cat/Search%20By%20Cat.rar

    Thanks!

    Svetla
  12. Milan Dec 28, 2013
    Hii,
    Grt Solution 
    I am using this solution but getting below error pls help me to solve the below error, if it possible .

    Server Error in '/' Application.
    The current type, 
    Telerik.Sitefinity.Abstractions.VirtualPath.IVirtualFileResolver, is an
    interface and cannot be constructed. Are you missing a type mapping?
  13. Milan Dec 28, 2013
    Hii,

    I am using  sitefinity version 6.2 and i am using this solution(perfect solution for me thanks a lot for that ), i have one query regarding this can i use more than one drop down and checkboxes with search textbox.

    And the second issue is i am geting error while using this sloution in my project please help me for that same

    Server Error in '/' Application.
    The current type, 
    Telerik.Sitefinity.Abstractions.VirtualPath.IVirtualFileResolver, is an 
    interface and cannot be constructed. Are you missing a type mapping? 
  14. milan Dec 31, 2013
    Hey Please relpy, m waiting for your answer.plsss
  15. Svetla Dec 31, 2013
    Hi Milan,
    On your first question- of course - you can use any user interface you wish to in order to serve search results. The trick I am using in the sample is to use the query string in the url that gets send to the search results page, so that the page can use these parameters, but naturally you can add more dropdowns, checkboxes, filters or facets to this search and change the results accordingly. 

    on your other question - are you sure you are setting the virtual path correctly and the resolver is set to EmbeddedResourceResolver. Double check these settings as this error is consistent with something being wrong there.

    Let me know if this works!

    Svetla
  16. Milan Jan 06, 2014
    Hii,

    Yes now it is working fine, But the main thing is i cant get any search Results in Custom Search Result it give only (No Results Found) every time.

    I have created 3 to 4 categories in it. And now i want to search with selected category and search box related results.

    Please give me the code for adding more two drop down in it and related search pls

    its urgent .........
  17. Milan Jan 06, 2014
    Hii,

    Yes now it is working fine, But the main thing is i cant get any search Results in Custom Search Result it give only (No Results Found) every time.

    I have created 3 to 4 categories in it. And now i want to search with selected category and search box related results.

    Please give me the code for adding more two drop down in it and related search pls

    its urgent .........
  18. Milan Jan 06, 2014
    Hii,

    Yes now it is working fine, But the main thing is i cant get any search Results in Custom Search Result it give only (No Results Found) every time.

    I have created 3 to 4 categories in it. And now i want to search with selected category and search box related results.

    Please give me the code for adding more two drop down in it and related search pls

    its urgent .........
  19. Milan Jan 10, 2014
    Hey  Svetla,

    pls relpy
  20. Stony Feb 19, 2014
    Hi Svetla,
    When I add Virtual path then run the site, I get an error server:

    File "/FirstSite/CustomSearch/SearchByCategory.Widgets.Templates.CustomSearchBox.ascx" does not exist.
    Parameter name: virtualPath

    Please give me an advice, many thanks. Stony.


  21. Shawn Jun 27, 2014

    HI Svetla,

    I'm not using thunder, I am modifying the toolboxes.config manually. Which do I register for the searchbox control the CustomSearchBox.ascx or the CustomSearchBox.cs?

    Thanks,

    Shawn

  22. Rohit Nov 08, 2016
    How to fix the assembly references? I am working on SF 9.1. 

    Leave a comment