For the sake of completeness, here is some more detailed info:
(because I don't want to have to remember this again in 2 months. ;-)
1. Create an Icon Image
If you want to make your custom icons look similar to the built in Sitefinity widget icons, it appears that you will need to begin with a template image (see attached example) that includes the box and shadow image, and paste your image into the box. The shadow box has 3 pixels of transparent space on the top and left sides.
Likewise you can avoid the shadow box look as well, your choice. If possible output to an 8-bit PNG to save space while retaining icon clarity.
2. Create a Class Name in Admin Toolbox Settings
When you create a custom icon class, make sure it isn't likely to conflict with something else. I recommend using your OWN prefix, don't use the 'sf' prefix that the out of the box icons use.
Enter this new class into:
Administration > Settings > Advanced > Toolboxes > Toolboxes > PageControls > Sections > (the section name) > tools > (the widget name) > CssClass
3. Add a CSS rule to your front end theme style sheet
Create that CSS class in your style sheet used for the front end theme. As discussed above, Sitefinity will ignore the rule on the front end site assuming that you don't use those classes in your publicly facing pages, and will apply it to the back end administrative interface. As needed. This CSS rule will look something like this:
CSS note: If you inspect the widget button DOM, you will see that there are MANY applicable CSS rules marked with !important;. Be as it may, you'll need to remember to use !important here too, or else your rule will be ignored for more important applicable rules.
background: url("/yourfilepath/custom-icon.png") no-repeat 0px 0px !important;
Security note: anyone who can see your front end website can ALSO see this rule. Therefore, be discrete in what kinds of comments and names you use here. This is one area where details about your backend administration is visible to the front end population. Not likely to be a problem, but something to keep in mind.
4. Fun With Sprites
If you have multiple custom icons, it is best to use CSS sprites to help load the button images faster. To accomplish that, you will need to start with a template such as the attached sprite example and add your own icon graphics.
Then you will create separate button class names using the steps above until you get to the CSS rules. At that point, you will need to calculate the pixels to the left and top edges of each button and adjust the background X,Y coordinates accordingly. Since this is a background image, much like the quirky new Apple mouse scroll, the numbers will be opposite of what is intuitive. ;-)
5. Refresh the page to view the results.
/* This button is already the top and left most image, so we use 0,0 */
background: url("/yourfilepath/custom-sprites.png") no-repeat 0px 0px !important;
/* move the image up 37 pixels to center on the middle button, will vary depending on your source image */
background: url("/yourfilepath/custom-sprites.png") no-repeat 0px -37px !important;
/* move the background image up 74 pixels for the last button. Again, your results depend on your source image */
background: url("/yourfilepath/custom-sprites.png") no-repeat 0px -74px !important;
/* if you had a second vertical row of sprites, you would need to shift the X coordinate accordingly. it would be -53px using my example image, to skip the 1st column. Your results vary depending on your input image. */
Have fun! Please comment if you notice any bugs or have additional useful tips related to these custom buttons.