+1-888-365-2779
Try Now
More in this section

Forums / Set-up & Installation / Switching from database to filesysystem for files.

Switching from database to filesysystem for files.

17 posts, 0 answered
  1. Roberto Modica
    Roberto Modica avatar
    146 posts
    Registered:
    08 Feb 2008
    21 May 2010
    Link to this post
    Hi All,

    I have an installation of siteFinity, and the site uses alot of images uploaded to the system, the database has grown from 14MB when released to well over 150MB in about a month which i feel is too big and will only continue to grow (The log file was close to a GB).  What i want to do (if this is even possible) is extract all the files out of the system (images/documents) from the libraries and place them on the file system instead and have sitefinity access them that way and then compact the database.

    Are there any procedures to do this?

    Thanks,

    Rob
  2. Radoslav Georgiev
    Radoslav Georgiev avatar
    3370 posts
    Registered:
    01 Feb 2016
    21 May 2010
    Link to this post
    Hello Roberto Modica,

    Thank you for using our services.

    This blog post explains how to get items from libraries and create actual files from them. You can use the approach there and combine it with the approach discussed in this article to re-add your existing files to the file system instead of DB. You can also use the DeleteContent method of library manager to delete older images.

    All the best,
    Radoslav Georgiev
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  3. Roberto Modica
    Roberto Modica avatar
    146 posts
    Registered:
    08 Feb 2008
    26 May 2010
    Link to this post
    Hi,

    Thanks for the response.  I am a little confused, maybe i am bieng a little dense but i am not sure where i am supposed to implement any of this code, so i have some questions.

    1) The first link, is for images, is the process the same for documents (as i have to extract those aswell), also where does it go, is it just a control, can i create this in a console app so i can reuse?

    2) how does this all work together, and how do libraries know you are using the file system instead of the database, is there a configuration setting i am missing.

    3) The second link, where does that code go?

    4) out of the box how do i make sure i start a project by using the file system instead of the database for files, becuase any website that exposes more then the normal posting for images is just gonna have a stupidly big database.

    Thanks,

    Rob
  4. Radoslav Georgiev
    Radoslav Georgiev avatar
    3370 posts
    Registered:
    01 Feb 2016
    26 May 2010
    Link to this post
    Hi Roberto Modica,

    Let me start by saying that the Images and Documents module works only either with storing files in db or in file system. You have to choose which one you want. Then you have to modify your web.config file if you decide to use the file system provider (by default the db is used). This article explains how to achieve this: How to enable the Files provider for Images and Documents module in Sitefinity 3.7 SP3.

    1) The process is the same for documents.

    2) With the above said the libraries determine whether to use the files or db provider by the setting in web.config. Already uploaded items retain their storage provider.

    3) You can combine the code from the first and second links in an aspx page on your webiste. Just create a page and add a button with click event handler. Add all the logic in this handler and open the web page.

    4) As said above you have to change the setting in web.config to use file system instead of db.


    Best wishes,
    Radoslav Georgiev
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  5. Roberto Modica
    Roberto Modica avatar
    146 posts
    Registered:
    08 Feb 2008
    01 Jun 2010
    Link to this post
    Hi,

    I am getting there with this but need some clarfication on somepoints. I have been able to grab all the files from the database and export them to a directory (a standard one so far, not the app_data one)

    1)  I have exisiting items in the database (images and documents) how can i update them so that they use the files located on the file system instead of the blobs in the database, i understand i can re-upload but wouldnt that mean i have to add all the files again, deleting all the old ones and starting again and pretty much start all the libraries from scratch?

    2) How do i, after i have adjusted the system to work in question 1, remove the blobs of file data from the database and reduce the size of the database down.

    Thanks for you support.

    Rob
  6. Roberto Modica
    Roberto Modica avatar
    146 posts
    Registered:
    08 Feb 2008
    01 Jun 2010
    Link to this post
    Hi,

    Sorry i should have re-read the original response, i think i will have to delete all that is there, then re-populate using the files that have been extracted (using the link in the post), is that correct?

    Rob
  7. Radoslav Georgiev
    Radoslav Georgiev avatar
    3370 posts
    Registered:
    01 Feb 2016
    01 Jun 2010
    Link to this post
    Hi Roberto Modica,

    Yes this is correct.

    Regards,
    Radoslav Georgiev
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  8. Roberto Modica
    Roberto Modica avatar
    146 posts
    Registered:
    08 Feb 2008
    01 Jun 2010
    Link to this post
    Hi,

    I am using this code (below) to add files back into their respecitve libraries the behaviour is not as expected.  What this does is add the file back to the libraries.  This works, but it doesnt use the file system, it just puts the file in the database as before (my system is setup to use the file system, it works when i do it manually) but when i use the code suggested in article regarding using content manager to re-upload it just uses the database again.  Am i doing something wrong here???  I am trying to automate as much of this as possible as i have a few sites to do this on.

    man.UploadFile(fBuffer,
        fileNameNoExt + "(a)",
        file.Extension,
        GetMimeType(fullFilePath),
        file.Length,
        lib);
  9. Radoslav Georgiev
    Radoslav Georgiev avatar
    3370 posts
    Registered:
    01 Feb 2016
    02 Jun 2010
    Link to this post
    Hi Roberto Modica,

    Could you please try out the code bellow:
    LibraryManager manager = new LibraryManager("Libraries");
    ContentManager libraryManager = new ContentManager("Libraries");
    ILibrary imagesLibrary = manager.GetLibrary("Images");
    IList allImages = imagesLibrary.GetItems();
    foreach (IContent image in allImages)
    {
        IContent temp = libraryManager.GetContent(image.ID);
        IContent newImage = libraryManager.CreateContent(temp.MimeType);
        newImage.ParentID = imagesLibrary.ID;
        libraryManager.SaveContent(newImage);
        newImage = (IStreamableContent)libraryManager.GetContent(newImage.ID);
        if (libraryManager.StreamingProvider != null && newImage != null)
        {
            using (var contentStream = libraryManager.StreamingProvider.GetUploadStream((IStreamableContent)newImage))
            {
                StreamHelper.CopyStream(new MemoryStream((byte[])image.Content),contentStream,false,libraryManager.Provider);
            }
            //copy metafields
            foreach (var metaKey in libraryManager.Provider.MetaKeys.Keys)
            {
                if (image.GetMetaData(metaKey.ToString()) != null)
                    newImage.SetMetaData(metaKey.ToString(), image.GetMetaData(metaKey.ToString()));
            }
            if (libraryManager.Provider.AllowComments)
                foreach (IComment comment in image.Comments)
                {
                    IComment newComment = libraryManager.CreateComment(newImage);
                    newComment.Author = comment.Author;
                    newComment.Email = comment.Email;
                    newComment.IpAddress = comment.IpAddress;
                    newComment.Owner = newComment.Owner;
                    newComment.Visible = comment.Visible;
                    newComment.Text = comment.Text;
                    newComment.WebSite = comment.WebSite;
                    libraryManager.SaveComment(newComment);
                }
            //copy item info
            if (image.ItemInfo != null)
                newImage.ItemInfo = image.ItemInfo;
            //copy thumbnails
            if (image.Thumbnails.Count > 0)
                foreach (var thumbnail in image.Thumbnails)
                {
                    try
                    {
                        newImage.Thumbnails.Add(thumbnail);
                    }
                    catch { }
                }
            libraryManager.DeleteContent(image);
            libraryManager.SaveContent(newImage);
        }
    }



    Best wishes,
    Radoslav Georgiev
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  10. Roberto Modica
    Roberto Modica avatar
    146 posts
    Registered:
    08 Feb 2008
    02 Jun 2010
    Link to this post
    Hi,

    Thankyou for the code, well, it is doing something.  The code seems to add the records to the database, creates the files in the file system but when you try and go into the library itself in the admin i recieve a Object reference not set to an instance of an object, looking at the stack trace if seems to be an issue with the thumbnail generation.

    [NullReferenceException: Object reference not set to an instance of an object.]
       Telerik.Libraries.WebControls.Admin.LibraryItemList.RenderThumbList(IContent currentItem, RepeaterItemEventArgs e) +872
       Telerik.Libraries.WebControls.Admin.LibraryItemList.ImagesList_ItemDataBound(Object sender, RepeaterItemEventArgs e) +100
       System.Web.UI.WebControls.Repeater.OnItemDataBound(RepeaterItemEventArgs e) +108
       System.Web.UI.WebControls.Repeater.CreateItem(Int32 itemIndex, ListItemType itemType, Boolean dataBind, Object dataItem) +136
       System.Web.UI.WebControls.Repeater.CreateControlHierarchy(Boolean useDataSource) +443
       System.Web.UI.WebControls.Repeater.OnDataBinding(EventArgs e) +51
       System.Web.UI.WebControls.Repeater.DataBind() +75
       System.Web.UI.WebControls.Repeater.EnsureDataBound() +55
       System.Web.UI.WebControls.Repeater.OnPreRender(EventArgs e) +15
       System.Web.UI.Control.PreRenderRecursiveInternal() +80
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Control.PreRenderRecursiveInternal() +171
       System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +842
  11. Radoslav Georgiev
    Radoslav Georgiev avatar
    3370 posts
    Registered:
    01 Feb 2016
    02 Jun 2010
    Link to this post
    Hello Roberto Modica,

    Thank you for getting back to me.

    It seems that the latest version of the item did not get properly saved. You can reiterate the items in the library with this:
    LibraryManager manager = new LibraryManager("Libraries");
    ILibrary imagesLibrary = manager.GetLibrary("Images");
    foreach (IContent image in imagesLibrary.GetItems())
    {
        StagedContent stg = manager.GetCurrentState(image.ID, true, true);
        foreach (var metaKey in manager.Provider.MetaKeys.Keys)
        {
            if (image.GetMetaData(metaKey.ToString()) != null)
                stg.SetMetaData(metaKey.ToString(), image.GetMetaData(metaKey.ToString()));
        }
        manager.SaveContent(stg, ContentStatus.Published);
        
    }

    This will resolve the issue you are experiencing. Bellow is a modified version of the previous code. It should work without problems now (if you need it for other libraries):
    LibraryManager manager = new LibraryManager("Libraries");
    ContentManager libraryManager = new ContentManager("Libraries");
    ILibrary imagesLibrary = manager.GetLibrary("Images");
    IList allImages = imagesLibrary.GetItems();
    foreach (IContent image in allImages)
    {
        IContent temp = libraryManager.GetContent(image.ID);
        IContent newImage = libraryManager.CreateContent(temp.MimeType);
        newImage.ParentID = imagesLibrary.ID;
        libraryManager.SaveContent(newImage);
        newImage = (IStreamableContent)libraryManager.GetContent(newImage.ID);
        if (libraryManager.StreamingProvider != null && newImage != null)
        {
            using (var contentStream = libraryManager.StreamingProvider.GetUploadStream((IStreamableContent)newImage))
            {
                StreamHelper.CopyStream(new MemoryStream((byte[])image.Content), contentStream, false, libraryManager.Provider);
            }
            manager.SaveContent(newImage, ContentStatus.Published);
            StagedContent stagedImage = manager.GetCurrentState(newImage.ID,true);
            //copy metafields
            foreach (var metaKey in libraryManager.Provider.MetaKeys.Keys)
            {
                if (image.GetMetaData(metaKey.ToString()) != null)
                    stagedImage.SetMetaData(metaKey.ToString(), image.GetMetaData(metaKey.ToString()));
            }
            if (libraryManager.Provider.AllowComments)
                foreach (IComment comment in image.Comments)
                {
                    IComment newComment = libraryManager.CreateComment(stagedImage);
                    newComment.Author = comment.Author;
                    newComment.Email = comment.Email;
                    newComment.IpAddress = comment.IpAddress;
                    newComment.Owner = newComment.Owner;
                    newComment.Visible = comment.Visible;
                    newComment.Text = comment.Text;
                    newComment.WebSite = comment.WebSite;
                    libraryManager.SaveComment(newComment);
                }
            //copy item info
            if (image.ItemInfo != null)
                stagedImage.ItemInfo = image.ItemInfo;
            //copy thumbnails
            if (image.Thumbnails.Count > 0)
                foreach (var thumbnail in image.Thumbnails)
                {
                    try
                    {
                        stagedImage.Thumbnails.Add(thumbnail);
                    }
                    catch { }
                }
            libraryManager.DeleteContent(image);
            libraryManager.SaveContent(stagedImage,ContentStatus.Published);
        }
    }



    Greetings,
    Radoslav Georgiev
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  12. Roberto Modica
    Roberto Modica avatar
    146 posts
    Registered:
    08 Feb 2008
    03 Jun 2010
    Link to this post
    Hi Radoslav,

    its almost there, i have tried to fix it but not really working.  The files get created and everything works, but the thumbnails are not generated? i know that part of the code is being fired, i have stepped through it but the thumbnails do not appear?

    Rob
  13. Radoslav Georgiev
    Radoslav Georgiev avatar
    3370 posts
    Registered:
    01 Feb 2016
    04 Jun 2010
    Link to this post
    Hello Roberto Modica,

    The first error you showed was not because thumbnails were not generated. It was because the staged content item did not have the Name metafield set. The first block of code from my previous reply would fix it. Are you still getting the same error? Bellow is a piece of code which generates and saves thumbnails for images uploaded in the file system:
    LibraryManager manager = new LibraryManager("Libraries");
    ILibrary imagesLibrary = manager.GetLibrary("Images");
    foreach (IContent image in imagesLibrary.GetItems())
    {
        if (manager.StreamingProvider != null)
        {
            StagedContent stg = manager.GetCurrentState(image.ID, true, true);
            MemoryStream imageContent = new MemoryStream();
            using (var contentStream = manager.StreamingProvider.GetDownloadStream((IStreamableContent)image))
            {
                StreamHelper.CopyStream(contentStream, imageContent, false);
            }
            if (imageContent.Length > 0)
            {
                System.Drawing.Image tempThumb = ImagesHelper.GenerateThumbnail(manager.DefaultThumbnailWidth, manager.DefaultThumbnailHeight, System.Drawing.Image.FromStream(imageContent));
                MemoryStream tempThumbStream = new MemoryStream();
                tempThumb.Save(tempThumbStream, ImageFormat.Jpeg);
                IThumbnail imageThumbnail = manager.AddThumbnail(image, tempThumbStream.ToArray(), "image/jpeg", ".jpg", stg.GetMetaData("Name") as string, true);
                stg.Thumbnails.Add(imageThumbnail);
                manager.SaveContent(stg, ContentStatus.Published);
            }
        }
    }


    Sincerely yours,
    Radoslav Georgiev
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  14. Roberto Modica
    Roberto Modica avatar
    146 posts
    Registered:
    08 Feb 2008
    06 Jun 2010
    Link to this post
    Hi Radoslav,

    I am not recieving any errors, simply the thumbnails are not generated, the files are, the thumbnails are not for images.  below is my aspx and cs, please exscuse the amount of things commented out, it has changed quite alot from when i first started this excercise.

    What i have is a web form with two buttons (used to be one) one to generate the files, which works, and the other to generate thumbs which doesnt.  Currently i am working from a test library called Test 123, i upload some images to the library and then run the two operations in the page attached.

    Not sure what i am doing wrong as really i just copied and pasted???

    Thanks,

    Rob

    ASPX
            <asp:Button ID="cmdExtract" runat="server" Text="Extract Files" 
                onclick="cmdExtract_Click" />
            <asp:Button ID="cmdThumbs" runat="server" Text="Generate Thumbs" 
                onclick="cmdThumbs_Click" />

    CS
    protected void DoMove()
       {
           LibraryManager manager = new LibraryManager("Libraries");
           ContentManager libraryManager = new ContentManager("Libraries");
           ILibrary imagesLibrary = manager.GetLibrary("Test 123");
           IList allImages = imagesLibrary.GetItems();
           foreach (IContent image in allImages)
           {
               IContent temp = libraryManager.GetContent(image.ID);
               IContent newImage = libraryManager.CreateContent(temp.MimeType);
               newImage.ParentID = imagesLibrary.ID;
               libraryManager.SaveContent(newImage);
               newImage = (IStreamableContent)libraryManager.GetContent(newImage.ID);
               if (libraryManager.StreamingProvider != null && newImage != null)
               {
                   using (var contentStream = libraryManager.StreamingProvider.GetUploadStream((IStreamableContent)newImage))
                   {
                       StreamHelper.CopyStream(new MemoryStream((byte[])image.Content), contentStream, false, libraryManager.Provider);
                   }
                   manager.SaveContent(newImage, ContentStatus.Published);
                   StagedContent stagedImage = manager.GetCurrentState(newImage.ID, true);
                   //copy metafields
                   foreach (var metaKey in libraryManager.Provider.MetaKeys.Keys)
                   {
                       if (image.GetMetaData(metaKey.ToString()) != null)
                           stagedImage.SetMetaData(metaKey.ToString(), image.GetMetaData(metaKey.ToString()));
                   }
                   //if (libraryManager.Provider.AllowComments)
                   //    foreach (IComment comment in image.Comments)
                   //    {
                   //        IComment newComment = libraryManager.CreateComment(stagedImage);
                   //        newComment.Author = comment.Author;
                   //        newComment.Email = comment.Email;
                   //        newComment.IpAddress = comment.IpAddress;
                   //        newComment.Owner = newComment.Owner;
                   //        newComment.Visible = comment.Visible;
                   //        newComment.Text = comment.Text;
                   //        newComment.WebSite = comment.WebSite;
                   //        libraryManager.SaveComment(newComment);
                   //    }
                   //copy item info
                   if (image.ItemInfo != null)
                       stagedImage.ItemInfo = image.ItemInfo;
                   //copy thumbnails
                   if (image.Thumbnails.Count > 0)
                       foreach (var thumbnail in image.Thumbnails)
                       {
                           try
                           {
                               stagedImage.Thumbnails.Add(thumbnail);
                           }
                           catch (Exception ex) { throw ex; }
                       }
                   libraryManager.DeleteContent(image);
                   libraryManager.SaveContent(stagedImage, ContentStatus.Published);
               }
           }
       }
       private void DoThumbs()
       {
           LibraryManager manager = new LibraryManager("Libraries");
           ILibrary imagesLibrary = manager.GetLibrary("Test 123");
           foreach (IContent image in imagesLibrary.GetItems())
           {
               if (manager.StreamingProvider != null)
               {
                   StagedContent stg = manager.GetCurrentState(image.ID, true, true);
                   MemoryStream imageContent = new MemoryStream();
                   using (var contentStream = manager.StreamingProvider.GetDownloadStream((IStreamableContent)image))
                   {
                       StreamHelper.CopyStream(contentStream, imageContent, false);
                   }
                   if (imageContent.Length > 0)
                   {
                       System.Drawing.Image tempThumb = ImagesHelper.GenerateThumbnail(manager.DefaultThumbnailWidth, manager.DefaultThumbnailHeight, System.Drawing.Image.FromStream(imageContent));
                       MemoryStream tempThumbStream = new MemoryStream();
                       tempThumb.Save(tempThumbStream, ImageFormat.Jpeg);
                       IThumbnail imageThumbnail = manager.AddThumbnail(image, tempThumbStream.ToArray(), "image/jpeg", ".jpg", stg.GetMetaData("Name") as string, true);
                       stg.Thumbnails.Add(imageThumbnail);
                       manager.SaveContent(stg, ContentStatus.Published);
                   }
               }
           }
       }
       protected void cmdExtract_Click(object sender, EventArgs e)
       {
           DoMove();
          // DoThumbs();
           //var man = new LibraryManager();
           //IList libraries = man.GetAllLibraries();
           //foreach (ILibrary l in libraries)
           //{
           //        ILibrary lib = l;
           //        IList allImages = lib.GetItems();
           //        int i = 0;
           //        string dir = l.Name.Replace(":", "---");
           //        Directory.CreateDirectory(storage + dir);
           //        foreach (IContent contentItem in allImages)
           //        {
           //            IContent cnt = man.GetContent(contentItem.ID);
           //            string name = (string)cnt.GetMetaData("Name");
           //            string ext = (string)cnt.GetMetaData("Extension");
           //            byte[] buffer1 = (byte[])cnt.Content;
           //            string fileName = name + ext;
           //            string fullFilePath = storage + dir + "\\" + fileName;
           //            ByteArrayToFile(fullFilePath, buffer1);
           //            i++;
           //        }
           //        Response.Write(i.ToString() + "-" + "items have been uploaded to:" + storage + "<br/>");
           // }
       }
       protected void cmdThumbs_Click(object sender, EventArgs e)
       {
           DoThumbs();
       }

  15. Radoslav Georgiev
    Radoslav Georgiev avatar
    3370 posts
    Registered:
    01 Feb 2016
    07 Jun 2010
    Link to this post
    Hello Roberto Modica,

    I have attached a sample project. Here is what you have to do to try out the code there:

    1) Connect it to a database.
    2) Upload some images in the database (set to use db by default)
    3) Set the images and documents module to use file system (from web.config):
    <libraries defaultGenericProvider="Libraries" streamingProviderName="Files" streamingIsEnabled="True" streamingChunkSize="20971520">

    4) Open the ~/TestPage.aspx and click the button. By default it is looking for a library called Images.

    All the best,
    Radoslav Georgiev
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  16. Cameron
    Cameron avatar
    42 posts
    Registered:
    27 Jul 2009
    12 May 2011
    Link to this post
    I hate bringing back an older thread, but I just came across it. I tried the project Radoslav attached and it seems to work great, with one minor issue. When I ran the Testpage.aspx page and clicked the button, it did in fact copy the documents to the file system and the links to the documents on any page that had a library control on it still worked.

    However, any blog posts that included a link to a document or image in a library no longer works. Any fixes for that problem?
  17. Ivan Dimitrov
    Ivan Dimitrov avatar
    16072 posts
    Registered:
    25 Nov 2016
    13 May 2011
    Link to this post
    Hi Michael,

    Please check the dynamic url of the old links and the new one. If the GUID of the items transferred to the file system is different then the urls will not work, because they cannot be resolved properly. An alternative woudl using the replace tool .
    I suppose that this is the issue, because LibraryManager.Upload creates a new item with a new guild. Also the provider gets changed.

    Regards,
    Ivan Dimitrov
    the Telerik team
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
Register for webinar
17 posts, 0 answered