The Right Way to do Vendor Comparisons

Good old Chuck Hollis has stirred up the vendor vitriol over the last week with two posts comparing the capacity efficiency of EMC's CX4, Netapp's FAS and HP's EVA products. See "Your Usable Capacity May Vary" and "Updates to Capacity Post".

Unsurprisingly, Chuck's conclusion is that EMC comes out on top (if it hadn't, would he still have posted - I think not). Now, not content with the statements Chuck made, HP have responded in kind. Check out the blog here (HP). There have also been plenty of comments on Chuck's posts, many of them non-too positive.

Whilst the opposing sides continue to score points off each other by highlighting the merits of their own technology, my mind drifts to the subject of exactly how vendor comparisons can be made. In some of my previous roles, I've had to help bring quotes for storage and SAN switches into line to make them as equivalent as possible in terms of their capacity. The trouble is, it isn't that simple.

Think of the difference between the switch vendors. Until recently, some vendors had full line speed blades in their hardware, however others followed the over-subscription model, sharing bandwidth between physical ports. If you're being charged by the port, then there's a clear difference in what you are getting for your money with these two technologies. My view was to work out the bandwidth per port as another comparison and to break down the cost of individual components, creating a more detailed cost model.

The same thing applies to arrays, whether enterprise or modular. Inevitably, most users don't follow the vendor best practices, choosing to use their own design (whether tested in their labs) or using a customised best practice model. There are also those who don't follow any model at all. :-)

Why do users do this? Easy - they all have different requirements and customise their hardware to match this.

Back to testing. We need some real world independent testing. So rather than vendors submitting their hardware to SPC tests in which they set the configuration, the independent testers need to set the hardware specification based on common sense configurations. Now you may say common sense isn't that common and this would allow "interpretations" of configurations but I disagree. I think common sense configurations would more likely get user approval and any vendor who believes their hardware is best would have no reason not to take part.

So, vendors out there - got any hardware you want to loan out?