There are several approaches:
1. Creating a new mode for the ImageGallery control. Please take a look ImageGallery control templates. You should add rel="shadowbox", and of course include the script in the beginning of the new template.
2. Second approach, less elegant one is to tweak the render html code. You should replace all links pointing to jpg files (and all others that you want to show via shadowbox) with such having rel="shadowbox" attribute.
3. Another approach is to use our API and dynamically build links to items in libraries.
I hope this helps.
the Telerik team