Get the Balance Right

It's not very often I side with one vendor or another however after BarryB's recent post regarding "Benchmarketing" I feel obliged to comment. Have a read of Barry Whyte's rebuttal too.

We see technology advancements because "concept" devices are used to drive innovation but don't necessarily translate directly to end-user products. Look at the fashion industry - some of the most outrageous outfits are paraded down the catwalk but the same dress, coat, hat or whatever isn't sold in the shops. Instead it influences the next fashion season.

Look at the motor industry - concept cars appear well before actual consumer products. We may laugh at some and marvel at others - take the Bugatti Veyron. It is well known that Volkswagen make a loss on each car produced, however what counters this is the publicity, the research, the kudos of being able to claim Veyron technology (disputably the fastest car in the world) is deployed in the standard VW range. Lexus is another good example of a brand created by Toyota to perform the same function. Much the same can be said for Formula 1.

Now, I'm not endorsing IBM per-se here, however I don't see the harm with IBM marketing a "concept" piece of technology which could lead to innovation in the future. After all, IBM is well known for research of this kind; the disk drive and the tape drive spring to mind.

Even EMC's own bloggers question whether EMC is known for innovation and other than Symmetrix, I can't think of one thing I view as an EMC "idea".

Anyway, 'nuff said. As previously offered - I would love to take the position of moderator in developing real world benchmarking - bring it on!!

IBM's Storage Announcements

Today IBM made a slew of new product announcements. None of it was a surprise due to lots of pre-announcement leaks and the fact the IBM session in Montpelier was to a certain degree the formal announcement of products already mentioned in standard press releases.

I can only describe the new product releases as slow and steady with nothing radical or earth shattering. What it does do is consolidate IBM's product range - slightly better tape libraries, slightly better tape drives; midrange SAN appliances, archiving appliances and a reconfirmation of the latest releases on SVC.

The release is here.

One point of note, IBM list their acquisitions in a PDF at the bottom of the release (get it directly here). IBM have acquired a lot of both hardware and software companies and this echoes somewhat of EMC a few years ago. What's interesting is to see how IBM will integrate and harmonise these products, rather than sell them as a disparate set of technologies. Should be fun...

Could IBM be buying Netapp?

Over at Tech Trader Daily, Eric Savitz has picked up on a 6% rise of Netapp shares today. There are no theories as to why, but I have my own. Could IBM be planning to buy Netapp?

If you think about it, the purchase would make sense. IBM is a huge reseller of Netapp as the N-Series. IBM can give Netapp access to a massive sales force, accelerating the plans Netapp has to move their sales channel to a more direct model. At $8.7 billion, it's a snip!

Then there's XIV. Could Netapp add the extra touch required to make XIV an enterprise product?

Remember you heard it here first.

P.S. I don't own shares in either Netapp or IBM

Off The Grid

I've been on holiday for the last week (sunning myself and the family in Cyprus). I had no Internet access - not even TV! Although I had no laptop (or Blackberry this time) I did take my iPod Touch, now configured with the mobile version of NewsGator. As I've mentioned previously, I have a 100+ RSS feeds (which I'll publish once I get around to it) on storage and others. My backlog was about 2500 entries, so I decided to challenge myself to get up to date and read as much as possible. Clearly I didn't read them all (there were plenty that could be skipped) but I read most and it provides for an interesting cross section...

EMC - blogs are run like a military machine; co-ordinating the news relating to new product releases and mercilessly hammering the competition. EMC have more storage bloggers than any other storage company and there are some good ones out there - one of my favourites is Information Playground by Steve Todd, where he discusses the design of Clariion.

Netapp - follows a close second to EMC with lots of bloggers and lots of competitor bashing. I particularly like Alex McDonald's postings.

IBM - doing a great job running the "resistance", fighting back against the continual onslaught of Barry (A Burke). Check out Barry (Whyte) and Tony Pearson. I'd like to see more from IBM though, especially their product developers working on DS arrays and XIV.

HDS - A jolly good bloke, but not really a player in the blogosphere. Only Hu contributes regularly, but doesn't engage in any serious debate.

Sun - quite literally on another planet with their storage strategy!

Dell - bought some toys, but doesn't know how to play with them. Unfortunately the older boy who could help them play with them has left...

Now there are more companies out there and I don't think I have any blog links from Brocade, 3Par, Compellent, Emulex, Qlogic, Pillar and others although I may be wrong (it is getting late). Any RSS link offerings gladly welcome - although I might not get around to reading them before my next holiday!

Why Tape Technology Just Doesn't Cut It

There have been a raft of tape announcements in the last week, most notably the two 1TB wannabee's IBM and Sun. For around a mere $37,000, plus the cost of a cartridge, I can backup 1TB of my most precious data. HP have also announced plans to extend the life of the DAT/DDS tape drive.



If you are a large enterprise customer then the cost of these drives may be justified (although I struggle to see how, when LTO4 drives can be had for about $5000 a piece) and I'm sure actual versus list price will be much lower.



The thing is, hard drives just continue to outpace tape growth. With 1.5TB drives on the way, and 1TB SATA drives available for less than $200, then disk-to-disk is much more appealing than tape at this rate. Obviously I'm riding roughshod over the issues of disk power consumption and portability but my point is that tape just isn't keeping up the pace in either capacity or throughput.

The whole issue is especially true in the small business area where it is easy to purchase terabytes of primary storage but backup to tape is really time consuming.

Why can't tape produce the equivalent bit density of disk? Is it the more fragile nature of the medium? Clearly tape is more flimsy than a rotating sheet of metal; the T10000 cartridge tape is 6.5 microns thick and the tape itself covers approximately 11.5 square metres, much more than the total surface area of the spinning plates in a hard drive.


I guess we will just have to accept tape capacity will never be good enough. That's just the way it is.


By the way, Sun get a big fat zero in the RSS ratings for not providing their news in RSS feed format....

FCoE

Fibre Channel over Ethernet has been back on my radar recently, especially as it was touted again at Storage Networking World in Orlando last week. Unfortunately I wasn’t there and didn’t see for myself, although I was in Orlando the week before on vacation. I can imagine if I’d extended or moved the holiday to include SNW that I’d be none too popular with Mrs E and my sons.

Any hoo, I looked back over my blog and I first briefly mentioned FCoE back in April 2007, a whole 12 months ago. Now, we know 12 months is a long time in the storage world (in which time iSCSI will have claimed another 3000% market share, EMC will have purchased another 50,000 storage companies of various and dubious value, HDS will have released nothing and IBM will have developed 2 or 3 new technologies which won’t see the light of day until I’m dead and buried). I expect then that FCoE should have moved on somewhat and it appears it almost has. Products are being touted, for example, Emulex with the LP21000 CNA card (not an HBA card, please note the new acronym) and Cisco with their Nexus 5000 switch (plus others).

At this stage I don’t believe the FCoE protocol has been fully ratified as a standard. I have been spending some time wading reading through the FC-BB-5 project documentation on the T11 website, covering FCoE to understand exactly how the protocol works in more detail and how it can be compared to native fibre channel, iSCSI, iFCP and FCIP. In the words of Cilla, here’s a quick reminder on storage protocols in case you’d forgotten.

Fibre channel and the Fibre Channel Protocol (FCP) provide a lossless, packet based data transmission protocol for moving data between a host (initiator) and a storage device (target). FCP implements SCSI over fibre channel. To date, fibre channel has been implemented on dedicated hardware from vendors including Cisco and McDATA/Brocade. iSCSI uses TCP/IP to exchange data between a host and storage device using the SCSI protocol. It therefore includes the overhead of TCP/IP but provides for lossy and long distance connectivity. iFCP and FCIP are two implementations which encapsulate FCP in TCP/IP packets. FCIP extends an existing fibre channel SAN, whereas iFCP allows data to be routed between fibre channel SANs.

FCoE will sit alongside fibre channel and allow the transmission of FCP packets at the Ethernet layer, removing the need for TCP/IP (and effectively allowing TCP/IP and FCP packets to exist on the same Ethernet network).

So hurrah, we have another storage protocol available in our armoury and the storage vendors are telling us that this is good because we can converge our IP and storage networks into one and save a few hundred dollars per server on HBA cards and SAN ports. But is it all good? Years back, I looked at using IP over fibre channel as a way to remove network interface cards from servers. The aim was to remove the NICs used for backup and put that traffic across the SAN using IPFC. I never did it. Not because I couldn’t; I’m sure technically it would have worked, but rather because the idea scared the willies out of “the management” for two reasons (a) we had no idea of the impact of two traffic types going over the same physical network and (b) the Network Team would have “sent the boys round” to sort us out.

Will this be any different with FCoE? Will anyone really be 100% happy mixing traffic? Will the politics allow the Networks teams to own SAN traffic entirely? Let’s face it, in large environments I currently advocate the separation of host, tape and replication traffic to separate fibre channel fabrics. I can’t imagine reversing my position and going back to single consolidated networks.

So then, is FCoE going to be better in smaller environments where the consolidation is more practical? Well, if that’s the case, then surely that makes FCoE just another niche player to FC, just like iSCSI.

It’s early days yet. There are a million-and-one questions which need to be answered, not least of which will be how FCoE will interoperate with standard FC, how drivers will interact with the existing storage protocol stack on a server and how performance/throughput will be managed. Some of these issues have been answered, however this blog entry is already far too long and rambling to include a discussion on these points this time and I will save them for another time.

HDS and Hardware

I’m away on holiday at the moment (which is a blessed relief, looking at the Easter weather in the UK, no pun intended) however catching up with some emails I noticed the supposed post on Storagezilla’s site that IBM are to purchase Diligent.

I nearly worked for the UK arm of Diligent many years ago when they were getting started, but that’s another story. What struck me about the comments ‘zilla made with reference to IBM, Moshe and IBM’s acquisitive streak was the predicament HDS finds itself in.

HDS traditionally partners and doesn’t purchase companies however they don’t own their underlying hardware – the product is produced by Hitachi and resold by HDS and HP. The recent IBM acquisitions and EMC’s relentless purchase of anyone with a storage product mean HDS is falling behind the game. They currently have 2 product lines; enterprise hardware and modular hardware, neither of which they own. With IBM and EMC making inroads into acquiring more hardware and software tools to build entire solutions, how could HDS hope to compete going forward?

I’m not sure if HDS have a relationship with Hitachi which guarantees them access to storage products and a say in product design, however if they don’t then I can’t see how they will compete with the big boys. HDS need to look to acquire technology and start producing integrated solutions pretty quickly or they are going to get squeezed out of what looks to be an increasingly competitive and solutions’ based market.

Oh, one other little footnote; my 10-year old son bought a iPod Touch at the airport and fortunately for him I’d brought my laptop with me. Unfortunately, he didn’t understand why I couldn’t get online and download iTunes to configure his new toy while flying along at 35,000 feet…

XiV Part II

Following on from BarryW's comment to my XiV post, I've been thinking over how the XiVarchitecture works. When a disk fails and the missing mirrors need to be recreated, then the data is likely to exist across all or most of the configured drives. Logically it would make sense that the target for the new lost mirrors would be all drives. So, as a drive fails, all drives could be copying to all drives in an attempt to ensure the recreated lost mirrors are well distributed across the subsystem. If this is true, all drives would become busy for read/writes for the rebuild time, rather than rebuild overhead being isolated to just one RAID group. Whilst that seems like a good thing for rebuild time, it seems like a bad thing for performance. Perhaps this isn't the case and in fact the failed device is re-created on a spare drive by copying all the mirrors back in from their other location.

Following the same line, in order to recreate a failed drive and rebuild the lost data across the array, then each drive must have spare capacity; in say a 200 drive system, that would require about 1/200th of all drives to be free at any one time, ready to receive rebuilt mirrors. Obviously the alternative option is just having spare drives, but that sounds less interesting!

What about when the failed disk is replaced? There must be another algorithm which ensures the replaced disk is not a target for all new writes, so presumably, static mirrors are pro-actively moved onto the replaced device.

This architecture throws up some interesting questions, especially when trying to understand performance. I am starting to get excited about messing about with one!

2.5" is cool

Hitachi Global Storage Technologies (the old IBM Disk Division) have announced a 500GB 2.5" drive for laptops. Not particularly interesting I hear you say, however these latest drives from HGST now come in Standard and "Enhanced Availability" versions. The enhanced version claims to be able to run a 24/7 workload rather than standard day to day access required in a laptop.

So does this mean these SATA drives are suitable to use to build an array? Well, first of all they are not fast ast 5400RPM. It would also be nice to have the SAS rather than SATA interface, but just look at the power consumption figures; 1.9W in read/write mode!

How about IBM taking the XiV technology and using 2.5" drives with this power consumption and capacity - now that would give EMC something to complain about!

Two for the price of one

The holidays are over and it's back to work for me. In fact I returned yesterday; the break was good however it is also good to be back.

It seems that I've returned to a flurry of acquisitions. Yesterday there was the heavily reported (on the blogosphere) purchase of XiV by IBM. Tony Pearson gives a summary of the features on his post. One thing that interests me is the use of distributed writes across an entire array by creating 1MB blocks from (presumably) LUNs and filesystems. If a drive fails, then the data is still available on other disks in the system and spread across a great number rather than a single drive (RAID-1) or potentially a small number of drives (RAID5/6).

I've been trying to get my head around what this means. On the one hand it sounds like a real problem, as a double drive failure could impact a wide number of hosts; it all depends on how well the 1MB chunks are distributed. However maybe it isn't that much of a problem as the issue only arises when both of the chunks that mirror a 1MB block both occur on failing drives. I would expect that as the number of physical drives increases then the impact of double failure reduces, as does the number of 1MB blocks affected. In addition, a drive may fail only in one area rather than on the whole device, so the affected blocks could be quite small; the remainder could be perfectly readable and be quickly moved. No doubt Moshe and the team have done the maths to know what the risk is and compared it to that of standard arrays and wouldn't be selling the product if it was not inherently more safe.

The only other issue I can see is what market the product will slot into; Tony mentions that the product is not for structured data (although I guess it supports it) but was designed for unstructured data of large binary file types. So, why use RAID-1 compared to say a 14+2 RAID-6 configuration which would be much cheaper in terms of the disk cost? Presumably another selling point is performance, but I would expect the target data profile (medical, large binary objects) to be more sequential than random access and not be that impacted by using SATA.

I guess only time will tell. I look forward to seeing how things go.

The other purchase announced today was that of Onaro by Netapp. Onaro sell SANScreen, a tool to collect and analyse fibre channel SANs and to highlight configuration issues. Whilst I think it is a good product, I don't see the fit with Netapp's business in the NAS market (in fact I'm sure SANScreen doesn't currently support NAS), so where's the benefit here other than buying up a company which must be close to or is making money.

I wonder who will be bought tomorrow?

SAN Virtual Appliances

LeftHand, FalconStor, Arkeia and Datacore all now offer VMware appliance versions of their products. I'm in the process of downloading them now and I'm hoping to install over the next few days and do some testing. I've previously mentioned some VM NAS products which I've installed but not reported back on. I'll try to summarise all my findings together.

It seems that the market for virtual appliances (certainly in storage) is getting bigger. I think this is a good thing but I'm not sure that the virtualisation technology today provides capabilities to allow all vendors to virtualise their products. I suspect that the iSCSI brigade will get best benefit out of this wave of technology but fibre channel will not, as (from my experience) VM products don't directly pass through fibre channel hardware to the VM guests (I'm aware of how RDM works in a VMware environment but I don't think pass-through of target devices is sufficient).

Will IBM produce an SVC Virtual Appliance? I doubt it, but products such as Invista should be perfect candidates for virtualising as they don't sit in the data path and the controller parts aren't critical to performance. So EMC, show us your commitment to Invista and make 3.0 the virtual version!

Netapp/Cisco

I've been a little quiet on the blog front over the last week, mainly because I've been away on business and I didn't take my laptop ( :-( ). I travelled "lite", which I'm not normally used to doing and that meant taking only the essentials. In fact, as I didn't have any checked baggage, I forgot about a corkscrew in my washbag, which was summarily extracted from me at the security checks at Heathrow.

Anyway enough of that, I've also had another issue to resolve attempting to link two Cisco fabrics via FCIP. It's a frustrating problem which has taken up more of my time than I would like and I still haven't managed to resolve it. Both fabrics already successfully move data via FCIP links, will connect to each other (and the end devices are visible and logged in) but the initiator HBA can't see any targets in the same zone.

These sorts of problems become annoying to resolve as most vendors take you through the level 1 process of problem determination (which translates to "you are an idiot and have configured it wrong") then level 2 ("Oh, perhaps there is a problem, send is 300GB of diagnostics, traces, configurations, date of birth, number of previous girlfriends etc") who get you to "try this command" - usually things you've already tried to no avail, because you actually know what you are talking about.

I'm almost at level 3 ("we've no idea what's causing the problem, we will have to pass to the manufacturer"). Hopefully at this stage I will start to get some results. Does anyone out there have a way to bypass all this first level diagnostics nonsense?

The other thing that caught my eye this week was the comment on Netapp and their targets miss. There is lots of speculation on what has occurred; here's my (two penn'orth/two cents).

Netapp had a great product for the NAS space, there's no doubting that. They made a great play of expanding into the Enterprise space when NAS-based storage became widely accepted. Some features are great - even something as simple as snapshots, replication and flexclones. However I think they now have some fundamental issues.

1. The Netapp base product is not an Enterprise storage array for NAS/FC/iSCSI. It doesn't scale to the levels of DMX-4 and USP. I think it is a mistake to continue to sell the Netapp appliance against high end arrays. Those of us who deploy USP/DMX technology regularly know what I mean.
2. The original Netapp technology is hitting a ceiling in terms of its useful life. The latest features customers demand, such as multi-node clustering can't be achieved with the base technology (hence the Spinnaker acquisition).
3. The product feature set is too complicated. There are dozens of product features which overlap each other and make it very difficult to determine when developing a solution, which is the right to choose (some have fundamental restrictions in the way they work that I found even Netapp weren't clear about).
4. Netapp developed a culture of the old IBM - that is to say expecting their customers to purchase their products and deriding them if they didn't choose them, attempting to resurrect the old addage "No-one Ever Got Fired for Buying IBM" to "No-one should get fired for buying Netapp".

I think I found point 4 most difficult to deal with; Netapp seemed to think they had a right to be No. 1 selection, almost forcing technical people to have to justify why *not* to buy Netapp.

Perhaps a little humility is long overdue.

Mine's bigger than yours - do we care?

Our resident storage anarchist has been vigorously defending DMX - here. It's all in response to discussions with Hu regarding whether USP is better than DMX. Or, should I say DMX-3 and Broadway (whoops, I mentioned the unmentionable, you'll have to shoot me now).

I have to say I enjoyed the technical detail of the exchanges and I hope there will be a lot more to come. Any insight into how to make what are very expensive storage subsystems work more effectively has to be a good thing.

But here's the rub. Do we care about how much faster DMX-3 is over USP? I doubt the differences are more than incremental and as I've both installed, configured and provisioned storage on 9980V/USP/8730/8830/DMX/DMX2/DMX3, I think I've enough practical experience to qualify it. (By the way, I loved StorArch's comment about how flexible BIN file changes are now. Well, they may be, but in reality I've found EMC cumbersome to release configuration changes).

Finally I'll get round to the point of this post; most large enterprise subsystems are of the same order of magnitude of performance. However I've yet to see any deployment where performance management is executed to such a degree that, hand on heart, the storage admins there can claim they sequeeze 100% efficient throughput. I'd make an estimate that things probably run 80% efficient, with the major bottlenecks being port layout, backend array layout and host configuration.

So the theoretical bantering on who is more performant than the other is moot; now, EMC, HDS or IBM, come up with a *self tuning* array then you've got a winner...

Buy Buy Buy

More acquisition news; Brocade and McDATA are finally just Brocade after the purchase completes. There's a snazzy new logo

which I can't decide is like a pair of red angel wings, or some convoluted join of the M and B from the two companies. Personally, I preferred the old logo depicting a fabric, it worked for me. I even had a lovely "personal" email from Michael Klayko (as I'm sure lots of other people did).

Anyway, moving on, what's more interesting is IBM bought SoftTek. Now they have a great product, TDMF, a piece of software that really made me go "wow!" when I first saw it in action. It literally is a transparent data mover. With the mainframe version (I was a proper storage admin when I started out on the mainframe) you could sync up many volumes in real time and perform an instant swap on them, allowing migration of TB of storage in seconds. I used the product to migrate data from an old storage array to new without any user impact or input at all. I hope IBM uses its new jewel well.