I know it's not HPC exactly, but a few years ago the company I worked for decided to replace its desktop PCs as they were a couple of years old. Some bright spark had the idea that it would be cool to build a cluster out of some of the old desktops and use it as a server.
We learned a lot from that exercise. Primarily that a cluster of half a dozen 2 year old desktops is pretty much useless as a server and will be massively out-performed by a single new desktop.