+1-888-365-2779
Try Now
More in this section

Forums / Developing with Sitefinity / Dynamic Site Search

Dynamic Site Search

15 posts, 0 answered
  1. Lee
    Lee avatar
    13 posts
    Registered:
    15 Aug 2008
    21 Oct 2008
    Link to this post
    Hi All,

    I really hope someone can help.  My site has a lot of dynamic pages for products in a shop, i.e it generates http://www.site.com/product1.aspx and http://www.site.com/product2.aspx .

    I need to add a site search to the site which I can do via the controls etc and index the site fine, however, the index is showing a very low number of pages that have been indexed (81 where I would expect over 500) and therefore some key words that I would expect to pull back results are not doing so.

    I have turned on auto indexing and indexing non public pages and the starting node is left blank to do the whole site.  Does anyone know where I'm going wrong or if this control can search dynamically added pages?

    Many thanks

    Lee
  2. Gabe Sumner
    Gabe Sumner avatar
    440 posts
    Registered:
    09 Sep 2007
    21 Oct 2008
    Link to this post
    Hi Lee,

    I don't have a quick answer, but I do have some quick questions that might help others suggest a fix.

    ---

    Were the dynamically added pages generated via the Sitefinity API? 

    Do the dynamic pages appear in the Sitefinity Admin in the Site Map?  Or are these pages created outside Sitefinity?

    If you click the "Start Indexing" link inside Sitefinity and manually generate the index, does it find more pages?

    Gabe Sumner

  3. Georgi
    Georgi avatar
    3583 posts
    Registered:
    28 Oct 2016
    24 Oct 2008
    Link to this post
    Hi guys,

    Gabe, these are questions that I'm really interested in too :)

    In order to be searched, the pages should be included in the Sitefinity sitemap, and to have valid URLs. The search engine works pretty much as the spiders and browsers do - it requests the page, and processes the content. One more question, which version of Sitefinity are you using?

    Kind regards,
    Georgi
    the Telerik team

    Check out Telerik Trainer, the state of the art learning tool for Telerik products.
  4. Venkat
    Venkat avatar
    105 posts
    Registered:
    20 Nov 2008
    12 Dec 2008
    Link to this post
    Hi Lee I think it is possible to index whole content which is shown dynamically by creating custom modules .

    Create a custom  module  then place your dynamic content in that module as items.

    Then create an index provider to create an index for this module.

    then all the words in the module get indexed will be displayed in search results

    does it makes any sense?
    Venkat
  5. Georgi
    Georgi avatar
    3583 posts
    Registered:
    28 Oct 2016
    16 Dec 2008
    Link to this post
    Hi Venkat,

    Thank you for taking a part in this discussion.
    It sounds as possible. Still we are waiting for a feedback from Lee. Hopefully it would be suitable for his needs :)


    Greetings,
    Georgi
    the Telerik team

    Check out Telerik Trainer, the state of the art learning tool for Telerik products.
  6. Lee
    Lee avatar
    13 posts
    Registered:
    15 Aug 2008
    13 Jan 2009
    Link to this post
    Hi all,

    Thank you for the responses and suggestions and sorry for the delay in response. 

    We have about 230 pages on our site and the indexing is picking up about 60ish, this is due to a lot of our pages having url rewrites on them so they are

    www.site.com/product.aspx?productID=1 rewritten as www.site.com/product1name.aspx

    The problem I'm having is that if I search for product1 in the search then it is not appearing.

    I can't expect sitefinity to try every product ID so is there a way of including a xml file sitemap of pages that I want to be indexed or some similar method?

    I'm currently using version 3.2.

    Many thanks

    Lee

  7. Georgi
    Georgi avatar
    3583 posts
    Registered:
    28 Oct 2016
    14 Jan 2009
    Link to this post
    Hi Lee,

    There is no built-in fuctionality for crawling URLs with dynamic query strings. PageIndex is getting the data for a certain page by it's URL without executing or following the links inside it. URLs of the pages are read from the sitemap.

    In your situation one way to achieve such indexing is writing a custom index provider. You will have to provide the links as a list to indexing engine. You may refer to the following forum thread: Indexing Troubles,where you will find Custom Index Provider implementation and some additional information.

    Regards,
    Georgi
    the Telerik team

    Check out Telerik Trainer, the state of the art learning tool for Telerik products.
  8. Lee
    Lee avatar
    13 posts
    Registered:
    15 Aug 2008
    14 Jan 2009
    Link to this post
    Hi,

    Thanks for your help that worked a treat, unfortunatly I have another question/problem.

    I'm trying to create a index with the custom index I've just created and the pae index, now this seems like it is a feasible thing to do and creates ok but one I start indexing I get an error Could not find file 'C:\Projects\ESPA\Websites\ESPAOnline2008\App_Data\Search\Espa\Index\_a.fnm'.

    I am able to create them as two seperate indexes and run the indexing on these fine with no errors, its just when they are in the same one.

    Many thanks

    Lee
  9. Georgi
    Georgi avatar
    3583 posts
    Registered:
    28 Oct 2016
    16 Jan 2009
    Link to this post
    Hi Lee,

    I am not sure what exactly is causing this. Could you please try delete everything under \App_Data\Search, and then to re index the web site?

    All the best,
    Georgi
    the Telerik team

    Check out Telerik Trainer, the state of the art learning tool for Telerik products.
  10. Lee
    Lee avatar
    13 posts
    Registered:
    15 Aug 2008
    16 Jan 2009
    Link to this post
    Hi Georgi,

    I've already tried this unfortunatly :o(

    Really not sure what it's doing, it's taking its time indexing before throwing an error so I'm guessing its doing half of it but falling over when it starts the other index, odd.

    Thanks

    Lee
  11. Georgi
    Georgi avatar
    3583 posts
    Registered:
    28 Oct 2016
    20 Jan 2009
    Link to this post
    Hello Lee,

    I am not really sure what is happening there. As a last resort, I could suggest you to send me your files. We will inspect them and debug locally. Hopefully we would come up with a suggestion.

    All the best,
    Georgi
    the Telerik team

    Check out Telerik Trainer, the state of the art learning tool for Telerik products.
  12. Rich
    Rich avatar
    27 posts
    Registered:
    20 Feb 2010
    04 Sep 2010
    Link to this post
    I have 7 Sitefinity sites where the searching works for all but 1 website.

    I create a basic index with All Pages, Auto Index and All Blogs with not customization.

    I have Blogs and Pages being indexed.  The Blog index is just fine, but even though finds the 36 pages, it is indexing only 6 terms.  The site (http://www.rwinsphoto.com) has plenty of text on it, so it should index.  Even modified or new pages don't get new terms, but the pages count increments as expected.

    I've deleted the contents of the App_Data/Search folder and re-indexed for the same results.

    I'm using Sitefinity 3.7 with the latest patch.  I'm also using the latest RADControls 519.

    What can I do to get more terms?
  13. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    06 Sep 2010
    Link to this post
    Hello Rich Winslow,

    1. Check App_Data/sitefinity.log for any errors that are thrown by Lucene search engine. During crawling there could be some problems and silent errors are thrown

    2. Most probably you have some control on a page or your master page that stops the crawler. You could try to create a new master page without controls on it and put a GenericContent with a simple text. Then run the search and check whether the word you typed is added to the index. After that start removing controls one by one from your master page until you see that number of the indexed words increase significantly.

    Kind regards,
    Ivan Dimitrov
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
  14. Rich
    Rich avatar
    27 posts
    Registered:
    20 Feb 2010
    07 Sep 2010
    Link to this post
    I had custom page logic that referenced the Session object, which runs fine for normal website usage, but the Search Indexing logic must not have a session variable.  I simple check for the existence of the Session object corrected it.  However, I suspect that other spiders may have had problems too, so this was a great double find!

    Thanks,
    Rich
  15. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    07 Sep 2010
    Link to this post
    Hi Rich Winslow,

    You can check whether the current request is crawler request and then do not execute your session logic. This will not affect the control and its normal behavior

    if (!CmsContext.IsRequestCrawler(this.Context))
     {
     
     
     }


    All the best,
    Ivan Dimitrov
    the Telerik team
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
Register for webinar
15 posts, 0 answered