How should I benchmark optimization solvers?

What's the best way to compare solvers? What should I measure and how? Should I care about the hardware? Let's find out!