Sitefinity 4.x requires 500MB+ application pool memory for the backend pages to function properly. If you have allocated less then 500MB, Sitefinity will work, but you might experience some performance issues when browsing the backend pages of the CMS. This is because all of the modules are put in one assembly - Telerik.Sitefinity, and the whole assembly is loaded into the memory.
What I can suggest you is to review the following page, which provides few tips for optimizing performance:
Please note that If caching is turned on then the page output should be many times faster.
You can also have a look at the following thread which holds a similar discussion:
the Telerik team
Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Public Issue Tracking
system and vote to affect the priority of the items