+1-888-365-2779
Try Now
More in this section

Forums / General Discussions / Include external websites in Sitefinity search indexes

Include external websites in Sitefinity search indexes

22 posts, 0 answered
  1. jkregala
    jkregala avatar
    159 posts
    Registered:
    22 Sep 2009
    03 Jan 2011
    Link to this post
    Is there a way to accomplish this? The aim is to be able to include in a search index for a certain Sitefinity site pages from a website that is not in Sitefinity.

    We're hoping to centralize our search results in the main Corporate site (which would be in Sitefinity 4.0) and that means we must find a way to "crawl" Sitefinity's search function into our other sites under the same brand which are obviously not managed by the CMS. Thanks for the help as always. :)
  2. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    03 Jan 2011
    Link to this post
    Hello jkregala,

    Out of the box you cannot search external pages from Sitefinity. There is no base implementation for this. Possible solution would be creating a custom class that inherits from PagePipe and there you can override ConvertImportedItemForMapping where you can add your custom data as "Content"

    protected override Dictionary<string, object> ConvertImportedItemForMapping(object item)
           {
     
               var result = base.ConvertImportedItemForMapping(item);
     
               InMemoryPageRender render = new InMemoryPageRender();
               result.Add("Content", "your custom data here");
     
     
            }

    You can replace the default PagePipe with your custom one or just add your custom pipe inside Global.asax, Application_Start where you can use Bootstrapper.Initialized event


    Bootstrapper.Initialized += new EventHandler<Telerik.Sitefinity.Data.ExecutedEventArgs>(this.AfterInit);

    add your pipe

    PipeFactory.RegisterPipeType("CustomPagePipe", typeof(CustomPagePipe));

    remove pipe

    PipeFactory.UnregisterPipe("PagePipe");


    All the best,
    Ivan Dimitrov
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  3. Ancuta Me
    Ancuta Me avatar
    43 posts
    Registered:
    26 Apr 2010
    06 Feb 2011
    Link to this post
    Hello,


    I have a Sitefinity 4.0 installation and I'm trying to create a custom search provider.
    I'm not looking to index another site, I just want to add some custom content to be indexed, while keeping the clasic index for all pages.

    Adding custom search provider was pretty straight forward with SF 3.7 but with SF 4.0 and Pipes, even with the indications Ivan provided, I was not able to do it.

    So far, I was able to change the indexed content for an already existing page using a PagePipe starting from the above post.

    What I would like is to index some custom content that is not inside sitefinity pages/news/blogs (it is taken from an external database).
    How can I add custom content to be indexed and link to an external page? I also want to keep the index SF has for its own pages.


  4. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    07 Feb 2011
    Link to this post
    Hi,

    Generally the way that you will index the content from another database is the same. You have to return your content in the result as shown in the code above. I found issues related to creating custom pipes and I logged a bug with ID 385822 for this. The current status of the bug is - not done.

    Best wishes,
    Ivan Dimitrov
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  5. Ancuta Me
    Ancuta Me avatar
    43 posts
    Registered:
    26 Apr 2010
    07 Feb 2011
    Link to this post
    I appreciate the update.
    Is there a location where the bug is posted, is there a way for me to follow the progress?

    Thank you.
  6. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    07 Feb 2011
    Link to this post
    Hi jkregala,

    We do not have a section in PITS for Sitefinity from where you can track the status of the bug. I am adding the ID here for reference in case someone else has the same issue and request the status of the bug.

    Kind regards,
    Ivan Dimitrov
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  7. Joseph Rosado
    Joseph Rosado avatar
    3 posts
    Registered:
    08 Apr 2010
    07 Mar 2011
    Link to this post
    Has this issue been resolved?  I need to do something similar ....

    Also, what do you mean by this, "your custom data here", is this a class, string, etc?  Do you have an example?
  8. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    08 Mar 2011
    Link to this post
    Hi Joseph ,

    The issue with custom pipes has been resolved. We will have a sample how to create a custom pipe with Q1 release. We will include the sample in the SDK, Products sample module.

    All the best,
    Ivan Dimitrov
    the Telerik team
  9. Imran
    Imran avatar
    3 posts
    Registered:
    16 Apr 2009
    11 May 2011
    Link to this post
    Hi Joseph,

    Can you confirm if the samples for creating the custom pipes have been released with Sitefinity 4.1. If so can you please point me to where it is in the documentation as I tried to look for it and could not find it.

    Thanks
  10. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    13 May 2011
    Link to this post
    Hello Imran,

    We did not include the sample in the SDK, because we are going to change the pipes API this Q which might cause some issues with the custom pipes and basically you will have to rewrite them on the next upgrade. We decided to provide more elegant and easiest way to crawler custom data.

    Regards,
    Ivan Dimitrov
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  11. Hahn
    Hahn avatar
    11 posts
    Registered:
    22 Mar 2011
    20 May 2011
    Link to this post
    has anyone had any success creating a custom search index based off of a custom pipe thus far?
  12. Ed Hardy
    Ed Hardy avatar
    17 posts
    Registered:
    28 Aug 2012
    23 May 2011
    Link to this post
    I've made a custom pipe. Will post it on the marketplace shortly for sale.
  13. Ancuta Me
    Ancuta Me avatar
    43 posts
    Registered:
    26 Apr 2010
    15 Jun 2011
    Link to this post
    Hi,

    Is there a sample available in documentation or on the market place?

    Thanks
  14. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    15 Jun 2011
    Link to this post
    Hello Ancuta Me,

    I attached a  working sample that shows how to implement a custom pipe for Products sample module that comes with the SDK. You can extend it to fit your need.

    Best wishes,
    Ivan Dimitrov
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  15. Saeed
    Saeed avatar
    14 posts
    Registered:
    21 Jun 2011
    05 Jul 2011
    Link to this post
    Hi There,

    I added the CustomPipe folder to my ProductCatalogSample project.
    I expected it should add a new scope to the Search Index page! but there weren't any changes there, and I still cannot see any search result if I select other Scopes!

    Did I miss something?

    Thanks,
    Saeed!
  16. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    05 Jul 2011
    Link to this post
    Hello Saeed,

    We are working on refactoring of the publishing system. We are adding options for custom pipes to work properly. With the current implementation  custom PopeSettings could not be used by the default providers, so you cannot have custom pipe settings. This is something that will be fixed for Q2.

    Regards,
    Ivan Dimitrov
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  17. Jeremy
    Jeremy avatar
    14 posts
    Registered:
    29 Jun 2010
    23 Nov 2011
    Link to this post
    Has this become available yet? I am trying to add a scope for items stored in a SQL database and can't figure out how to set this up.
  18. Bonny
    Bonny avatar
    58 posts
    Registered:
    11 Nov 2016
    24 Nov 2011
    Link to this post
    Hello Jeremy,

    I will try to explain you how to do what you want.
    1. You need to iterate all of the objects that you want to include in the index and for each object - create WrapperObject, that holds it's properties (new WrapperObject(your object here)).
    2. Set the language for this wrapper object, if your site is multilingual (wrapperObject.Language = lang)
    3. Create the document that will be included in the search index
    (new Document(wrapperObject, "Id", new List<Mapping>()) - "Id" is the name of the identity field)
    4. Now, when you have the document, you just need to include it in the search index - SearchManager.UpdateCatalogue(catalogueName,documents) , where "documents" represents all documents that you want to be included in the search index.

    We don't have example for such scenario at the moment, but as you see, this is not hard to be implemented.

    Keep in mind that our default behaviour is to index only "Title" and "Content" fields. So if you need to index another properties, you may need to look at this post:
    http://www.sitefinity.com/blogs/teodorgeorgiev/posts/11-11-14/search_index_with_custom_fields_in_sitefinity_4.aspx

    We refactored the publishing system and you can use custom pipes, but I think that this is much more easier way to achieve what you need.

    Best wishes,
    Bonny
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  19. Jeremy
    Jeremy avatar
    14 posts
    Registered:
    29 Jun 2010
    24 Nov 2011
    Link to this post
    What I don't understand is where do I put this code? On what backend or what page? I've already created a search pipe using the above sample but it is not showing in the search scopes. what do I need to do to make it show? I did comment out the.inbound and outbound pipes. Are they needed?
  20. Bonny
    Bonny avatar
    58 posts
    Registered:
    11 Nov 2016
    28 Nov 2011
    Link to this post
    Hello Jeremy,

    You can write this code in new aspx page in your project. For example in Page_Load method. When you open this page, the code will run. After it completes, the normal search index for pages (for example with name "TestPageIndex") will be updated and in the catalogue with name "TestPageIndex" will contains the pages from your external database. Keep in mind that those pages should have Title and Content properties, because by default we index only those properties.
    You dont need any custom pipes to achieve this. Just run this code, to push all data from external database in your normal pages search index.
    Also keep in mind that when you make changes to your external database, you should run this page again to keep your search index updated.

    Regards,
    Bonny
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  21. Jeremy
    Jeremy avatar
    14 posts
    Registered:
    29 Jun 2010
    28 Nov 2011
    Link to this post
    Does the Identity Field need to be a GUID or something? When I add it with the ID field as a Long, it gives an error about a GUID not being in the correct format.

    I added a zRestart.aspx page and added the following Procedure under if (!IsPostback), but when go to the Admin side and look at the Search Indexes, the Courses Scope does not show.

    private void RefreshCourseSearch()
        {
            try
            {
                List<Document> locDocs = new List<Document>();
                ClsSearch_Data locSearch = new ClsSearch_Data();
                ClsSearch_Data.ClsInputParamsSearchCourses locSearchInput = new ClsSearch_Data.ClsInputParamsSearchCourses();
                ClsDataSearchReturnSearch locSearchResult = new ClsDataSearchReturnSearch();
     
                locSearch.zValInput = locSearchInput;
     
                if (locSearch.zProSearch(ref locSearchResult))
                {
                    foreach (DataRow locRow in locSearchResult.zValSearchReturn.Rows)
                    {
                        string locTitle = "";
                        string locContent = "";
                        //string locIDStr = "-1";
                        string locSearchTerms = "";
                        //long locID = -1;
     
                        if (locSearchResult.zValSearchReturn.Columns.Contains("Title") && (locRow["Title"].ToString() != null) && (locRow["Title"].ToString() != ""))
                            locTitle = locRow["Title"].ToString();
     
                        if (locSearchResult.zValSearchReturn.Columns.Contains("Description") && (locRow["Description"].ToString() != null) && (locRow["Description"].ToString() != ""))
                            locContent = locRow["Description"].ToString();
     
                        if (locSearchResult.zValSearchReturn.Columns.Contains("SearchString") && (locRow["SearchString"].ToString() != null) && (locRow["SearchString"].ToString() != ""))
                            locSearchTerms = locRow["SearchString"].ToString();
     
                        //if (locSearchResult.zValSearchReturn.Columns.Contains("WebPage") && (locRow["WebPage"].ToString() != null) && (locRow["WebPage"].ToString() != ""))
                        //{
                        //    locIDStr = locRow["WebPage"].ToString();
     
                        //    if (locIDStr.IndexOf("&page=") > -1)
                        //    {
                        //        locIDStr = locIDStr.Substring(locIDStr.IndexOf("&page=") + 6);
                        //    }
                        //}
     
                        //long.TryParse(locIDStr, out locID);
     
                        ClsSearch locData = new ClsSearch(Guid.NewGuid(), locTitle, locContent, locSearchTerms);
     
                        WrapperObject locWrapper = new WrapperObject(locData);
     
                        Document locDoc = new Document(locWrapper, "ID", new List<Mapping>());
                        locDocs.Add(locDoc);
                    }
                }
     
                SearchManager locManager = new SearchManager();
                locManager.UpdateCatalogue("Courses", locDocs);
     
                lblRefreshedSearchCourse.Visible = true;
            }
            catch (Exception ex)
            {
                SCAppVars.AppLogEntry locErrorHandler = new SCAppVars.AppLogEntry();
                locErrorHandler.zVal_LastErrorCodeException = SCAppVars.AppLogEntry.ZENUM_ErrorCodeException.ZPRO_EXCEPTION;
                //same for every procedure
                StackFrame locSF = new StackFrame();
                MethodBase locMB = locSF.GetMethod();
                locErrorHandler.ZPRO_ERROR_HANDLE_ExceptionAlerts(locMB.Name + ":Produced EXCEPTION", locErrorHandler.ZPRO_ERROR_GetLastExceptionMessage(), ex, locMB.Name);
     
            }
        }
  22. Bonny
    Bonny avatar
    58 posts
    Registered:
    11 Nov 2016
    29 Nov 2011
    Link to this post
    Hi Jeremy,

    About first question - yes, the id should be guid. However, keep in mind that this id should be unique for every item and every item should have exactly one id. What I mean is that if you push one item 2 times with different id-s it will exist 2 times in the search index. I see that you use Guid.NewGuid(), which is really unique, but executing this code 2 times for one search index will add the items 2 times each.
    About the "scopes" that you can't see - let me try to explain what search index is :
    1. you should create new search index (I see that you want to idex both internal Sitefinity pages and external pages from your DB), so you can create normal page index (static html content) and reindex it. This way all pages that you have will be added to it. (name the search index "Courses" like in your code)
    2. If you check the directory "App_Data\Sitefinity\Search\Courses" you should see some files in it - there the search objects are stored.
    3. Now you can run your code - all external items will be added in the catalog, so when you search in the search index named "Cources", that you initially created, you should see both your external pages and sitefinity internal pages.

    Kind regards,
    Bonny
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
22 posts, 0 answered