Bit of an old topic, but there's no specific number for this. Instead, try using http://www.webpagetest.org/ . to check the speed of your websites and determine if there's a speed problem and if so, where.
For example, one of the things I do us use a professional DNS system that has DNS servers located all over the place. DNS lookups are then done close to the visitor, speeding things up quite a bit.
I also am colocated in a facility that has peer sharing with most of the main ISP providers in my country. So when visitors access our sites, there's not many hops to get here.
Both of those things can make a visibile difference without upgrading hardware.
More ram is always an easy fix too. 4g seems low, 16 gig would be better. And failing that, a simple bump to an SSD hard drive can work wonders as well.
Most of the other hardware stuff will likely have a small impact compared to all of the above.