Who Ya Gonna Call?

Here's a quality piece of reporting from TechCrunch on the state of Facebook and their data problems. I mentioned just last week in this post about their data growth. It's incredible that they're purchasing a new Netapp 3070 filer each week!

I'm surprised that Facebook would be continually purchasing NAS filers to grow their content. There must be a rolling set of pictures, thumbnails and so on that are frequently looked at, but there also must be a significant amount that aren't and could be archived to super-dense nearline type technology akin to the Copan products.

Unfortunately when data growth is so intense, it isn't always easy to see the wood for the trees and from previous and current experience, using Netapp creates the risk of wasted resources.

In my experience, looking at just block-based arrays, I've always seen around 10-15% of orphan or unused resources and sometimes higher. When host-based wastage is taken into consideration, the figure can be much worse, although host reclamation is a much more intense process.

I'm willing to offer to anyone out there who has more than 50TB of storage on storage arrays a free analysis of their environment - for a 50:50 split of any savings that can be made. As budgets tighten, I think there will be more and more focus on this kind of work.

Could Netapp make a virtual NAS appliance?

As well as storage, one area of IT I find really interesting is virtualisation. Over the years I've used VM (e.g. the IBM mainframe platform), MVS (now morphed into z/OS) as well as products such as Iceberg. More recently I've been using VMware since it was first released and finally have managed to deploy a permanent VMware ESX installation in my home/office datacentre. That has given me the opportunity to install and test virtual SAN appliances, such as VSA from LeftHand Networks and Network Storage Server Virtual Appliance from FalconStor. I'll publish more on these in a week or so once I've done some homework, but for now I want to discuss Netapp.

As many of you will know, Netapp have offered a simulator for ONTAP to their customers for some time (BTW, Dave and the crew, although I'm not a customer, I would be grateful of an up-to-date copy). The simulator is great for script testing and learning new commands without totally wrecking your production operations. However I think it is about time Netapp took the plunge and offered ONTAP as a virtual appliance.

It shouldn't be hard to do for two reasons (a) the code is mostly Unix anyway and (b) most if not all the code exists in the simulator. It also seems to me to be an easy win; there are many organisations who wouldn't consider placing a Netapp filer into a branch office due to cost, but would deploy VMware for other services. A virtual filer could provide File & Print, iSCSI, SAN *and* most usefully, replicate that data back to core using standard Netapp protocols such as Snapmirror and Snapvault.

Perhaps Netapp haven't done it as they don't want to cut into their generous hardware margin on disk, but with a virtual offering to complement their physical ones, Netapp could retain their position as NAS vendor of choice.

The Right Way to do Vendor Comparisons

Good old Chuck Hollis has stirred up the vendor vitriol over the last week with two posts comparing the capacity efficiency of EMC's CX4, Netapp's FAS and HP's EVA products. See "Your Usable Capacity May Vary" and "Updates to Capacity Post".

Unsurprisingly, Chuck's conclusion is that EMC comes out on top (if it hadn't, would he still have posted - I think not). Now, not content with the statements Chuck made, HP have responded in kind. Check out the blog here (HP). There have also been plenty of comments on Chuck's posts, many of them non-too positive.

Whilst the opposing sides continue to score points off each other by highlighting the merits of their own technology, my mind drifts to the subject of exactly how vendor comparisons can be made. In some of my previous roles, I've had to help bring quotes for storage and SAN switches into line to make them as equivalent as possible in terms of their capacity. The trouble is, it isn't that simple.

Think of the difference between the switch vendors. Until recently, some vendors had full line speed blades in their hardware, however others followed the over-subscription model, sharing bandwidth between physical ports. If you're being charged by the port, then there's a clear difference in what you are getting for your money with these two technologies. My view was to work out the bandwidth per port as another comparison and to break down the cost of individual components, creating a more detailed cost model.

The same thing applies to arrays, whether enterprise or modular. Inevitably, most users don't follow the vendor best practices, choosing to use their own design (whether tested in their labs) or using a customised best practice model. There are also those who don't follow any model at all. :-)

Why do users do this? Easy - they all have different requirements and customise their hardware to match this.

Back to testing. We need some real world independent testing. So rather than vendors submitting their hardware to SPC tests in which they set the configuration, the independent testers need to set the hardware specification based on common sense configurations. Now you may say common sense isn't that common and this would allow "interpretations" of configurations but I disagree. I think common sense configurations would more likely get user approval and any vendor who believes their hardware is best would have no reason not to take part.

So, vendors out there - got any hardware you want to loan out?

Could IBM be buying Netapp?

Over at Tech Trader Daily, Eric Savitz has picked up on a 6% rise of Netapp shares today. There are no theories as to why, but I have my own. Could IBM be planning to buy Netapp?

If you think about it, the purchase would make sense. IBM is a huge reseller of Netapp as the N-Series. IBM can give Netapp access to a massive sales force, accelerating the plans Netapp has to move their sales channel to a more direct model. At $8.7 billion, it's a snip!

Then there's XIV. Could Netapp add the extra touch required to make XIV an enterprise product?

Remember you heard it here first.

P.S. I don't own shares in either Netapp or IBM

I Stand Corrected

In a previous post earlier this year I mentioned the Onaro purchase by Network Appliance. As I said at the time, I wasn't aware Onaro's SANScreen product even had a NAS module. It seems I was wrong, and thanks for Deni O'Connor for indirectly pointing it out. In fact, SANScreen now has NAS Insight which provides for NAS monitoring support. However this feature was only made general availability on 31 December 2007, so you can hopefully excuse my oversight for not realising it has been released.

(On a side note, why are large Enterprises such as Onaro still not using RSS to announce product releases? I haven't got the time or inclination to trawl their websites each day. RSS is so much easier.)

I had a quick look at the NAS Insight press release and details on their website. Although there's a demo, I couldn't ascertain what NAS products (other than Netapp filers which are in the demonstration) the product supports. That makes me think it supports nothing BUT Netapp (although I again stand to be corrected). If that's true then NAS Insight is a pointless feature for many customers who would want to use the product for cross vendor consolidation and certainly doesn't demonstrate Onaro's NAS credentials. Compared to the last release of DFM I saw, NAS Insight is pretty poor.

To date my SANScreen exposure has been based on one large "global installation" of the product and presentations from the Onaro marketing team. When an instance of SANScreen was enabled in one location of the global deployment, it created thousands of exceptions which then required manual intervention. When I last had a presentation on the product (in October) there was a large number of SAN scenarios SANScreen wasn't reporting on, a lot of these relating to non-EMC and replication support. There's still a long way to go yet.

Up to this point, Onaro may have developed relationships with the vendors which provides for ongoing access to new releases of their management tools in order to extract configuration information. Going forward, will those companies still be as keen to provide that information to Netapp, who may be their direct competitor in the NAS (and non-NAS) marketplace?

Netapp/Cisco

I've been a little quiet on the blog front over the last week, mainly because I've been away on business and I didn't take my laptop ( :-( ). I travelled "lite", which I'm not normally used to doing and that meant taking only the essentials. In fact, as I didn't have any checked baggage, I forgot about a corkscrew in my washbag, which was summarily extracted from me at the security checks at Heathrow.

Anyway enough of that, I've also had another issue to resolve attempting to link two Cisco fabrics via FCIP. It's a frustrating problem which has taken up more of my time than I would like and I still haven't managed to resolve it. Both fabrics already successfully move data via FCIP links, will connect to each other (and the end devices are visible and logged in) but the initiator HBA can't see any targets in the same zone.

These sorts of problems become annoying to resolve as most vendors take you through the level 1 process of problem determination (which translates to "you are an idiot and have configured it wrong") then level 2 ("Oh, perhaps there is a problem, send is 300GB of diagnostics, traces, configurations, date of birth, number of previous girlfriends etc") who get you to "try this command" - usually things you've already tried to no avail, because you actually know what you are talking about.

I'm almost at level 3 ("we've no idea what's causing the problem, we will have to pass to the manufacturer"). Hopefully at this stage I will start to get some results. Does anyone out there have a way to bypass all this first level diagnostics nonsense?

The other thing that caught my eye this week was the comment on Netapp and their targets miss. There is lots of speculation on what has occurred; here's my (two penn'orth/two cents).

Netapp had a great product for the NAS space, there's no doubting that. They made a great play of expanding into the Enterprise space when NAS-based storage became widely accepted. Some features are great - even something as simple as snapshots, replication and flexclones. However I think they now have some fundamental issues.

1. The Netapp base product is not an Enterprise storage array for NAS/FC/iSCSI. It doesn't scale to the levels of DMX-4 and USP. I think it is a mistake to continue to sell the Netapp appliance against high end arrays. Those of us who deploy USP/DMX technology regularly know what I mean.
2. The original Netapp technology is hitting a ceiling in terms of its useful life. The latest features customers demand, such as multi-node clustering can't be achieved with the base technology (hence the Spinnaker acquisition).
3. The product feature set is too complicated. There are dozens of product features which overlap each other and make it very difficult to determine when developing a solution, which is the right to choose (some have fundamental restrictions in the way they work that I found even Netapp weren't clear about).
4. Netapp developed a culture of the old IBM - that is to say expecting their customers to purchase their products and deriding them if they didn't choose them, attempting to resurrect the old addage "No-one Ever Got Fired for Buying IBM" to "No-one should get fired for buying Netapp".

I think I found point 4 most difficult to deal with; Netapp seemed to think they had a right to be No. 1 selection, almost forcing technical people to have to justify why *not* to buy Netapp.

Perhaps a little humility is long overdue.

Storage as a commodity

I just read a comment over at Zerowait regarding Netapp and proprietary hardware. It reminded me of something I was thinking about recently on the commoditisation of storage.

There's nothing worse to my mind than a storage vendor who has no competition. Inevitably in some organisations that situation can exist when a single supplier is chosen to supply (for example) switches, SAN or NAS. The difficulty though is how to avoid that situation. Most vendors would love to lock you into their proprietary tools and relating back to the above article link, Netapp is one I see who try that more than anyone. They have a bewildering array of interlinked product options; once your hooked (especially where you use a feature to retain long term backups via snapshot/vaults) then you're sucked into a dependency on their products which just isn't healthy.

What's the solution? Well, for me I like to commoditise storage functionality. Pick out those features which all vendors support and only use the proprietary features where absolutely necessary. At least then you can maintain multiple vendors all on the hook for your next piece of business.

Of course implementing commoditised storage is more difficult than just picking a few common product features. However as far as your users are concerned, a LUN is a LUN and NAS storage is NAS storage, with a few caveats on things like driver levels for HBAs and so on.

I've previously posted a modular storage comparison sheet. As an example, here are some of the features that almost all support:

• RAID 5 protection
• Consistent LUN size
• dual pathing
• Active/Passive failover
• remote replication
• Fibre Channel presentation
• SNMP alerting
• online code upgrades
• hot swappable components

Before I get lots of comments saying "hold on, not all modular products are the same"; remember I'm not saying that. What I am saying is having a consistent set of requirements allows you to maintain a shortlist of vendors who can all enjoy the healthy competition of bidding for business. So, time to draw up a NAS spreadsheet....

more about iSCSI

I mentioned as a "Storage Resolution" to look more in-depth at iSCSI. Well I've started doing just that today.

The first thing I thought I needed was a working environment. I'm not keen on investing in an entire storage array (at this stage) to do the testing (unless some *very* generous vendor out there wants to let me "loan" one) so I've build a virtual environment based on a number of free components.

I've a dedicated VMware testing machine recently built which has a dual Core Intel processor, 2GB of RAM and a SATA drive. Nice and simple. It runs Win2K3 with the free VMware server, onto which I've created another Win2K3 R2 partition and a Linux partition running Fedora Core 6. This is where my iSCSI "target" will sit.

For those unfamiliar with SCSI terminology, the source disk or disk system presents LUNs which are referred to as targets. The host accessing those LUNs is the initiator; simply put the host initiates a connection to a target device, hence the names. My iSCSI target in this instance is a copy of the Netapp simulator running on Linux.

Most people are probably aware of the simulator. If not, Dave Hitz talks about it here. I've created a number of disks into an OnTAP volume and out of that created a LUN. LUNs can be presented out as FC or iSCSI, in this instance I've presented it out as iSCSI.

By default the simulator doesn't enable iSCSI so I enabled it with the standard settings. This means my target's iSCSI address is all based on Netapp defaults. I'm going to work on what the best practices should be for these settings over the coming days. Anyway, I've presented 2 LUNs and numbered them LUN 4 and LUN 9.

At the initiator (host) end, I've used my Win2K3 Server and installed the iSCSI initiator software from Microsoft. This gave me a desktop icon to configure the settings. Again, I've ended up with the default names for my iSCSI initiator, but that doesn't matter; all I had to do was specify in the iSCSI initiator settings the IP address of my target, log on and it finds the LUNs (oh, one small point, I had to authorise the initiator on the simulator). Voila, I now have 2 disks configured to my Windows host which can be formatted as standard LUNs.

As a performance test, I ran HdTach against the iSCSI LUNs on Win2k3. I got a respectable 45MB/s throughput, which isn't bad bearing in mind this environment is all virtual on the same physical machine.

All the above sounds a bit complicated, so I'll break it down over the coming days as to what I had to do; I'll also explain the iSCSI settings I needed to make and my experiments with dual pathing and taking the iSCSI devices away from Windows in mid-operation.