There is a LibraryImageDialog that returns the path to the image, but inside <img> tag which automatically renders the item. So, you need a custom RadEdtior button
which calls LibraryImageDialog and then you should return the image location to the href of anchor. You could reuse the current dialog -Sitefinity\Admin\ControlTemplates\Libraries\Dialogs\ImageEditorDialog.ascx
and modify its insertLink() function.
the Telerik team
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Public Issue Tracking
system and vote to affect the priority of the items