+1-888-365-2779
Try Now
More in this section

Forums / Ecommerce / Sitefinity 200k products importing very slow

Sitefinity 200k products importing very slow

6 posts, 0 answered
  1. Praneeth
    Praneeth avatar
    26 posts
    Registered:
    22 Jun 2012
    05 Dec 2014
    Link to this post

    Hi Team,

     I have a task of migrating data from a CSV to Sitefinity ecommerce. The tedious part is importing 200k products from CSV. Every single product savechanges call takes time. I tweaked the code as below to run in batches of 10. Still the performance is really sad. To import 1,000 products it takes around 10 minutes. For 1 lakh proudcts it is taking more than 10 hours. Please advise on this as this is really very slow and I am stuck with a migration task to reduce the time to 1 hour.

     int skip = 0;
                    int take = _batchSize == 0 ? 10 : _batchSize;
                    int currentCount = 0;

                    int totalCount = productImportCsvs.Count();
                    while (currentCount < totalCount)
                    {
                        string skusNotImported = string.Empty;
                        try
                        {

                            IEnumerable<ProductImportCSV> chunkedProducts = productImportCsvs.Skip(skip).Take(take);
                            skip = skip + take;
                            currentCount = skip;

                            foreach (ProductImportCSV product in chunkedProducts)
                            {
                                if (string.IsNullOrEmpty(product.SKU))
                                {
                                    continue;
                                }

                           
                                ImportSingleProduct(dynamicModuleManager, scheduledJob, catMgr, liveProducts, masterProducts, ref successfullyImportedProducts,
                                    ref errorImportProducts, product, deleteExistingProducts);
                            }

                               
                            catMgr.SaveChanges();
                        }

  2. Svetoslav Manchev
    Svetoslav Manchev avatar
    735 posts
    Registered:
    29 Nov 2016
    10 Dec 2014
    Link to this post
    Hi Praneeth,

    You can check the Product import widget sample.

    More information and installation instructions is available in that article.

    Regards,
    Svetoslav Manchev
    Telerik
     
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items
     
  3. Praneeth
    Praneeth avatar
    26 posts
    Registered:
    22 Jun 2012
    10 Dec 2014 in reply to Svetoslav Manchev
    Link to this post

    HiSvetoslav ,

     I had already tried it even before writing the post :-(. The performance was not satisfactory and hence writing this post. Actually some part of implementation I had taken from the code itself. As mentioned 200k records even with that code took almost 10 to 15 hours. We need something which can do the work atleast within 2 hours.

     

    Do you have some sort of SQLMapper class. I am planning to write SQLBulkCopy and finish the job, which I am completely against :-(

     Thanks

    Praneeth

  4. Svetoslav Manchev
    Svetoslav Manchev avatar
    735 posts
    Registered:
    29 Nov 2016
    15 Dec 2014
    Link to this post
    Hello Praneeth,

    In order to speedup the product creation, you can try to use manager.Lifecycle.Publish(item). Also, SaveChanges() should be called in batches for example for every 50 or 100 items created.

    You can check the attached code sample. This should speedup the creation.

    Regards,
    Svetoslav Manchev
    Telerik
     
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items
     
  5. Praneeth
    Praneeth avatar
    26 posts
    Registered:
    22 Jun 2012
    15 Dec 2014 in reply to Svetoslav Manchev
    Link to this post

    Tried this code already. The methodImportSingleProduct what I mentioned does the same :-(. Still it is slow. Any other suggestions.

     

    Thanks

    Praneeth

  6. Svetoslav Manchev
    Svetoslav Manchev avatar
    735 posts
    Registered:
    29 Nov 2016
    18 Dec 2014
    Link to this post
    Hello Praneeth,

    What is the time you have experienced for 200k products using the latest example provided? During the tests we have 20k products have been created for about 18 min.

    In addition, would you please upload your entire code sample in order to inspect and test it.

    Regards,
    Svetoslav Manchev
    Telerik
     
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items
     
6 posts, 0 answered