So with further digging I answered part 1. to the question above. Adding the following attribute to the ButtonSelector allowed me to browse Document Libraries (but not both which is what I'd prefer).
<sfLib:ButtonSelector ID="ButtonSelector3" WindowNavigateUrl="~/Sitefinity/UserControls/Dialogs/ItemDialog.aspx" .....
Is there a way to get it to display both? Also, the multiple selectors breaking the ImageManager in Rad Editor still exists. To clarify it happens when clicking on the "Select from already uploaded images" link, and it's a JS error "Line 4149, Sys.ArgumentUndefinedException: Value cannot be undefined..." The issue is that the fix to have multiple requires a clientID and associatedControls array, but the client ID is null.
Hope that helps someone and hope it helps someone else help me :)