So after trying and trying and after some blog and forum hunting i found out that you can check if the request is an index request.
I added this check to each of my custom widget and layout controls. This solved the issue, but was a lot of work.
For the future:
- Please add more debug info (or atleast give the option to generate more log entries)
I don't think this is supposed to work like this? :-)