Safari, in a attempt to speed up web browsing, caches favicons for each site you visit. Over time, however, this potentially can make web browsing slower, rather than faster. Deleting these saved favicons just might make your web browsing a little faster. Obviously, caching these tiny icons does help, so deleting them too frequently will be counter productive. Delete them with the following command (and restart Safari):
# rm -r ~/Library/Safari/Icons
As always, your mileage may vary.