The Right Way to do Vendor Comparisons

Good old Chuck Hollis has stirred up the vendor vitriol over the last week with two posts comparing the capacity efficiency of EMC's CX4, Netapp's FAS and HP's EVA products. See "Your Usable Capacity May Vary" and "Updates to Capacity Post".

Unsurprisingly, Chuck's conclusion is that EMC comes out on top (if it hadn't, would he still have posted - I think not). Now, not content with the statements Chuck made, HP have responded in kind. Check out the blog here (HP). There have also been plenty of comments on Chuck's posts, many of them non-too positive.

Whilst the opposing sides continue to score points off each other by highlighting the merits of their own technology, my mind drifts to the subject of exactly how vendor comparisons can be made. In some of my previous roles, I've had to help bring quotes for storage and SAN switches into line to make them as equivalent as possible in terms of their capacity. The trouble is, it isn't that simple.

Think of the difference between the switch vendors. Until recently, some vendors had full line speed blades in their hardware, however others followed the over-subscription model, sharing bandwidth between physical ports. If you're being charged by the port, then there's a clear difference in what you are getting for your money with these two technologies. My view was to work out the bandwidth per port as another comparison and to break down the cost of individual components, creating a more detailed cost model.

The same thing applies to arrays, whether enterprise or modular. Inevitably, most users don't follow the vendor best practices, choosing to use their own design (whether tested in their labs) or using a customised best practice model. There are also those who don't follow any model at all. :-)

Why do users do this? Easy - they all have different requirements and customise their hardware to match this.

Back to testing. We need some real world independent testing. So rather than vendors submitting their hardware to SPC tests in which they set the configuration, the independent testers need to set the hardware specification based on common sense configurations. Now you may say common sense isn't that common and this would allow "interpretations" of configurations but I disagree. I think common sense configurations would more likely get user approval and any vendor who believes their hardware is best would have no reason not to take part.

So, vendors out there - got any hardware you want to loan out?

Could IBM be buying Netapp?

Over at Tech Trader Daily, Eric Savitz has picked up on a 6% rise of Netapp shares today. There are no theories as to why, but I have my own. Could IBM be planning to buy Netapp?

If you think about it, the purchase would make sense. IBM is a huge reseller of Netapp as the N-Series. IBM can give Netapp access to a massive sales force, accelerating the plans Netapp has to move their sales channel to a more direct model. At $8.7 billion, it's a snip!

Then there's XIV. Could Netapp add the extra touch required to make XIV an enterprise product?

Remember you heard it here first.

P.S. I don't own shares in either Netapp or IBM

The Conserve IT Con

There's been a lot of blogosphere talk recently about the Wikibon "Conserve_IT" initiative and how California's Pacific Gas and Electric Company are taking the initiative and offering organisations rebates for demonstrable IT efficiencies. You can find comments here, here and here.

Unfortunately as with so many of these kinds of initiatives, the devil is in the detail. It is worth spending time reading the Conserve IT wiki page and Wikipedia's page on the California Energy Crisis of 2000/2001. To quote from Conserve IT: "Power in California is severely constrained". Why is that? The answer is pretty simple; California botched their deregulation of the energy markets and were screwed over by many companies, including Enron. In addition, significant underinvestment has placed restrictions on power distribution within the state (some of which is starting to be remedied).

It is pretty obvious that companies such as PG&E need other methods of controlling energy growth (Wikibon almost claim a virtue of the way per capita energy growth has remained flat in California compared to the rest of the US and Europe) and rebates for efficient energy use is their approach. Don't think of this as an altruisic green initiative - PG&E and others can't provide the electricity required.

So what about the Conserve IT initiative?

"As part of the qualification process, Wikibon has launched the Wikibon Energy Labs, an independent verification service that validates energy measurements of vendor products. "

Do we have so little faith in our vendors? When companies deploy equipment into their datacentres do they not measure the increased load on their PDUs? HDS and EMC provide tools for calculating the power demands of their products. These are detailed methods of analysis, down to the component level. Cisco provide significant information on power draw for their equipment (I've talked about this before). I'm sure other companies do to - feel free to mail me links or tools for calculating power demands for other manufacturers (I've asked HP about this for EVA but had no response so far).

The truth is, most eletricity generating companies are out to make money and giving organisations rebates for using less of their products makes no business sense. Where there are constraints on offering a service, then perhaps rebates or other incentives can do good, but this is not the norm and won't be replicated as a model across the rest of the US or the world.

As an organisation, you can do all the power calculations yourself. It isn't hard and it isn't hard to validate expected power draw against reality by getting one of your electricians to double check power consumption of a particular piece of hardware.

I'd urge all the vendors to make their power tools transparent and available. I would also urge them to incorporate KVA/cooling figures into their management tools.

In the meantime, read the links above and see how pointless Conserve IT is.

Off The Grid

I've been on holiday for the last week (sunning myself and the family in Cyprus). I had no Internet access - not even TV! Although I had no laptop (or Blackberry this time) I did take my iPod Touch, now configured with the mobile version of NewsGator. As I've mentioned previously, I have a 100+ RSS feeds (which I'll publish once I get around to it) on storage and others. My backlog was about 2500 entries, so I decided to challenge myself to get up to date and read as much as possible. Clearly I didn't read them all (there were plenty that could be skipped) but I read most and it provides for an interesting cross section...

EMC - blogs are run like a military machine; co-ordinating the news relating to new product releases and mercilessly hammering the competition. EMC have more storage bloggers than any other storage company and there are some good ones out there - one of my favourites is Information Playground by Steve Todd, where he discusses the design of Clariion.

Netapp - follows a close second to EMC with lots of bloggers and lots of competitor bashing. I particularly like Alex McDonald's postings.

IBM - doing a great job running the "resistance", fighting back against the continual onslaught of Barry (A Burke). Check out Barry (Whyte) and Tony Pearson. I'd like to see more from IBM though, especially their product developers working on DS arrays and XIV.

HDS - A jolly good bloke, but not really a player in the blogosphere. Only Hu contributes regularly, but doesn't engage in any serious debate.

Sun - quite literally on another planet with their storage strategy!

Dell - bought some toys, but doesn't know how to play with them. Unfortunately the older boy who could help them play with them has left...

Now there are more companies out there and I don't think I have any blog links from Brocade, 3Par, Compellent, Emulex, Qlogic, Pillar and others although I may be wrong (it is getting late). Any RSS link offerings gladly welcome - although I might not get around to reading them before my next holiday!

Drobo Again

As previously discussed I've been gradually moving data over to my Drobo. I recently invested in a DroboShare, which lets me share the contents of the Drobo as NAS.


Although I run my data through a Windows Server, I have been toying with the idea of having a large majority of the content available through a NAS share rather than let the Windows box do the work. In this way, if the server is down, then I'm not affected. This is effectively what the Drobo can achieve, however configuring it wasn't as simple as it seems.

The first issue I encountered was understanding whether, when plugging a Drobo with data on it, I would need to reformat the Drobo to work with the DroboShare (turns out you don't reformat and the data just gets passed through). This "feature" wasn't at all clear in the manual so I decided not to use my primary disk storage and data to test the theory and I just plugged the Drobo alone into the network.


At this point, I'd expect the Drobo to get a DHCP address and be visible in the Drobo dashboard software, but it wasn't. A quick check of DHCP on the server and a trace from Ethereal and I was sure the Drobo had requested and successfully picked up an IP address, but it wouldn't appear on the dashboard.

With fingers, toes and anything else I could find, all crossed, I decided to plug the Drobo into the Droboshare. As if by magic, the drive appeared, presented through the Drobo - with all data intact.

I wouldn't advise anyone to try this kind of cavalier approach but I guess that is part of the underlying Drobo design of simplicity, however I think I like the idea of having a bit more control of my storage.

After a few days of use, I started to notice problems; firstly the DroboShare wasn't always visible; it wasn't clear in the first place how the Drobo would present itself and after tweaking the settings I managed to get it named as \\DROBO\ on my network, but it wasn't always accessible and it wasn't clear why.

The second problem was to do with security - or more precisely the lack of it. Anyone with children will recognise the desire to prevent little fingers tinkering where they shouldn't. For me, that means making my music, video and picture shares all read-only to everyone except me. I can't achieve that with DroboShare.

Eventually the teething problems caused me to take the DroboShare out and resort to my previous configuration. I'm disappointed but perhaps I shouldn't have expected anything over and above a basic NAS setup. Since reverting to plain Drobo, I've had seamless operations once again.

Second hand DroboShare anyone?

The cuts are happening again

The Register has reported that some UK banks are forcing their contractors to take a 10% pay cut or take a hike. It is usual for contractors to take the hit first as they are an easy target compared to terminating permanent staff and to be fair, as a contractor/consultant myself, if the chop comes then so be it, it's part of the nature of the business.

However, the downside is that the most able contractors are likely and will surely leave first. Many of those who remain will have a lethargic attitude to work and overall, the company suffers.

In addition, Barclays are offshoring more jobs, presumably to further save costs. In my experience, moving jobs offshore doesn't work. It simply moves skills away from the business and causes the customers to lose control over their IT operations. This isn't necessarily a reflection on poor quality of work from the outsourcing suppliers; far from it, I'm sure most are equally as competent or skilled as their UK counterparts. The issue is more to do with operating remotely from the customer; not understanding the "local" issues; not understanding local culture and not being part of the team who see and chat with each other every day. As someone who currently works with multiple clients for 1-2 days a week, I experience this problem even though I know my customers well. Imagine if you've never met most of the people you are providing complex services for.

One final note; Barclays claim they worked out they were paying "over the market rate". That's a phrase I've never understood. It assumes all contractor's skills are equally good (or bad) and so they can be paid, or have their pay cut to the same degree. It also assumes that all contractors were taken on and overpaid by an equal amount. I wonder if they did the same market comparision for permanent staff and considered cutting their salaries by an equivalent amount? Somehow I think not...

Size of the Storage Blogosphere

I use Feed Demon to track all of my RSS subscriptions. The product integrates with Newsgator online and now with the iPod Touch. The benefit of having an online location for synchronising my feeds means I can run multiple Feed Demon copies from the different laptops I use on a daily basis.

I need this level of integration as I now have over 110 feeds just focused on storage (111 in fact).

It is incredible that the number of blogs just related to storage is increasing on an almost daily basis.

How many feeds do you read?

Enterprise Connectivity with iPod Touch

Last year I blogged about my replacement iPaq after my previous one failed. As I said at the time, it was dull, dull and more dull.

Earlier this year I bought an iPod Touch while in the US. It is a wonderful device. The screen is fantastic, the quality of the applications is great and the video is so watchable. Unfortunately, email integration with my Exchange Server was only via IMAP. The integration worked fine, however it didn't integrate my calendar or contacts. For a while I used the notes application until I jailbreak'ed and installed a to-do list application, although it wasn't that good.

Now I've installed version 2.0 of the 'Touch software (following on from its release on the iPhone). I have full Exchange integration and the ability to install applications (Omnifocus for time management being the first).

Finally my 'Touch has become indispensable. I can't see anything else on the market to touch it (no pun intended)! Now I have a problem...do I invest in the new iPhone....

A Dying Industry (slightly off topic)

The BBC reported today of the next stages of the government and record companies' attempts to crack down on illegal file sharing. Persistent offenders will have their internet connections cut off. Unfortunately, the genie is out of the bottle in terms of the ability for music to be copied and shared.


In the late 90's, I founded a music company that distributed music by CD. You chose the tracks (from our catalogue) and we cut them to disk and shipped them to you. We also had digital downloads DRM protected with Microsoft's encryption. This was *before* Apple release their iTunes store. At the time, the record labels would not give us any new material and we were restricted to back catalogue and inferior content. This ultimately was the demise of the business as our customers couldn't understand why we didn't have access to any music track in the world.

At the time, the record companies' view was that providing us content that could be distributed on CD was risky and would allow people to copy the music. Well, of course it would, in exactly the same way people were *already* copying CDs purchased from record stores! Their principled approach changed when Apple rocked up with large volumes of cash for access to their catalogue and the digital download era started for real.

Now I see a music industry which for years acted in a protectionist fashion, overcharged for their content and milked the customer with countless re-releases and compilations, trying to target anyone they can in order to protect their revenue stream (a bit like when SCO sued IBM).

Unfortunately the genie I mentioned earlier is out there, alive and well and doesn't need the Internet. Music content can easily be swapped on portable hard drives, CD & DVD-ROMs, memory sticks, flash memory cards and so on. The Internet just provides the opportunity to spread music wider than physical media can.

Ultimately I think all copyrighting and protection systems, whether electronic or legal, will fail or be circumvented. We already know that artists are changing their revenue models away from the recorded content into concerts and other revenue streams. Most record companies have moved back from DRM protected content on digital downloads and Sony scored the biggest faux pas with the rootkit they deployed on CDs in 2005.

There will always be people who want something for nothing, as the release of Radiohead's recent album "In Rainbows" shows. However it also shows that people will pay (the average pay price for the album in a survey was believed to be around £4) for decent content and perhaps that represents the crux of the problem - not selling stuff people don't actually want.

Recycling Drives - Update

Last week I posted about wasted hard drives, removed from arrays and crushed to prevent the leak of sensitive data.

I contacted HGST and Seagate to get some additional background. Here are their responses, slightly edited to correct any spelling mistakes but otherwise intact.

Seagate

(a) when will the technology be deployed in Enterprise FC drives? Our OEMs are currently developing with the Cheetah 15K.6 FDE, a drive that Seagate has already in production.

(b) is the technology proprietary to Seagate? - No, this will becompliant with the Trusted Computing Group's spec. All hard drive vendors are participating in this Trusted Computing Group and we expect that they will have self-encrypting drives that will be inter-operable with ours.

(c) is DriveTrust accepted by the US Government and other similar organisations as secure enough to treat a drive as "wiped" if the encryption keys are removed? Endorsement from National Security Agency (NSA) has already been received for the 1st Self-Encrypting Drive Model-the Momentus(r) 5400 FDE hard drive, for protection of information in computers deployed by U.S. government agencies and contractors for national security purposes.

(d) are any of the "big" manufacturers (EMC/HDS/IBM) looking to deploy DriveTrust enabled drives in storage arrays? IBM and LSI have both publicly announced that they will do so. Note that Hitachi has also just announced a self-encrypting drive, the Deskstar E7K1000, a drive designed for business critical storage systems.

(e) Where do the drives go when they're wiped for final disposal? Extra shipping is involved to ship a drive to a special data destruction service facility, where it can be degaussed or shredded, and then the drive must be shipped to [be] environmentally disposed of. Alternatively, a drive may be over written, a process that takes hours and hours, using energy and tying up system resources, and then may be re-purposed.


HGST

My name is Masaru Masuda, working on product planning for Hitachi GST. Let me try to answer your question. Like Raj mentioned below, we have already supported bulk encryption feature for 2.5" and 3.5"and will support it to Enterprise product next year. With the bulk encryption feature, user data on the HDD media is automatically and always encrypted by the SoC inside [the] HDD. The security feature has two basic functions. One is active protection of data (encryption with secret key) and secure erase of the drive by deleting the encryption key for repurposing or disposal. As you pointed out, Standardization is a key for security. Therefore, a non profit security organization called TCG (Trusted Computing Group) was formed as described in the page 5 and 6 of the attached package. We have been very actively involved in the activities of TCG and plan to pick up security feature based on TCG standards which will be implemented from next year.The security market is still small but it has been growing steadily due to the data security concern and also as a fast and cheap solution for repurposing of drives in Server applications or disposal of failed drives. Also we have had a recycling process for drives failed in the internal testing and for drives returned from the field.

Thanks to both companies for their responses.

So it seems to me that in the future there will be no excuse for scrapping drives. I think the retirement process for HDDs should form part of the "green measurement" of storage.

Five Storage Strategies that *May* Save You Money

Infoworld have a great article here discussing how to save money on storage in these tight times we are experiencing. Here's a summary - with my opinions of course!

1. Play Hardball with Vendors. Ah if it was only that simple. It may be possible to find another vendor selling hardware marginally cheaper, but are you ready for it? There are few companies who have got their storage deployment to a level where interoperability allows them to take storage from any of a range of vendors. Fewer still who have calculated the cost of migration or the full operating expense for each vendor's product; it isn't all about the hardware cost alone.


2. Avoid New Purchases By Reclaiming What You Have. So what tools will you use to achieve this? Do you know how and why you are missing storage which can be reclaimed? Storage reclamation is usually an ad-hoc process run when storage gets tight or when admins have the time. Some places may have written scripts to automate the discovery of wastage but it isn't easy. I use my own software tool and can highlight about 10 separate categories but you need to be careful of the law of diminishing returns.


3. Audit Backup and Replication Configurations To Cut Waste. Right, so throw away some backups. Are you *sure* you can do that? Surely the data owner needs to validate whether that backup copy isn't needed any longer...


4. Rethink Storage Network Decisions. I like this one. Basically, find a cheaper piece of hardware - which may include DAS! Again, interoperability and migration costs have a big impact here. Any marginal savings may be wiped out by the cost of moving to another platform.


5. Use a Tiering Methodology That Delivers Results Simply. Finally a point I agree with. Tiering can be implemented easily by taking a common sense approach to moving data to a cheaper layer of disk. This doesn't have to involve a migration project but can be achieved as users request more storage.

All of these options are great, however they fail to attack the underlying issue of rising costs - that fact that more data is being generated each day. Simply asking users to question whether new storage is needed or existing allocations can be re-used is as easy as implementing new technical solutions.

Remember - keep it simple!

The Defrag Debate

I was asked again this week whether defragging of hard drives on Windows servers is really necessary. This is quite pertinent as the cost of deploying an enterprise-wide defrag tool can be significant and any opportunity to save money has to be a good one.

I discussed fragmentation last year (here) when looking into a problem which turned out to be the lack of free space consolidation. however I didn't discuss the potential performance impact.

So, reflecting on whether defragmentation is required or not, I'd say that in most cases the benefits are minimal. Here's why...

Firstly, hard drives have on-board cache; the larger and more "enterprise" the drive, then the larger the cache. We are also likely to see more and more hybrid drives on the market, which will have large amounts of fast memory fronting access to the old moveable components. Cache will mask the impact of fragmented files during writes as data will be written to cache and confirmed as written by the drive. The data can then be written to disk asynchronously afterwards. Obviously if cache becomes overrun, then the benefit will be negated.

Second, operating systems have built in file system performance techniques, including lazy writing and file prefetch. These features will try and minimise the latency issues of reading and writing to disk.

Third, if the data being accessed is a database, the database itself will also have lazy writer processes to asynchronously write data to disk.

Now all of the above applies directly to "traditional hard drives". In systems which have RAID controllers with onboard cache, then the issues will be less. Where storage is taken from a SAN with an enterprise or modular array, all reads and writes will occur through cache, giving the best options for performance and masking the underlying hard drive mechanics.

So, can fragmentation actually help? In fact, I have seen one instance when this occurred and it required slightly special circumstances. The file system was made up from a number of concatenated LUNs using a volume manager. As the server had multiple CPUs and the storage was SAN connected, then multiple I/Os could be issued to the volume. With more fragmentation, then the I/Os are spread across all of the LUNs and performance was increased.

Why Tape Technology Just Doesn't Cut It

There have been a raft of tape announcements in the last week, most notably the two 1TB wannabee's IBM and Sun. For around a mere $37,000, plus the cost of a cartridge, I can backup 1TB of my most precious data. HP have also announced plans to extend the life of the DAT/DDS tape drive.



If you are a large enterprise customer then the cost of these drives may be justified (although I struggle to see how, when LTO4 drives can be had for about $5000 a piece) and I'm sure actual versus list price will be much lower.



The thing is, hard drives just continue to outpace tape growth. With 1.5TB drives on the way, and 1TB SATA drives available for less than $200, then disk-to-disk is much more appealing than tape at this rate. Obviously I'm riding roughshod over the issues of disk power consumption and portability but my point is that tape just isn't keeping up the pace in either capacity or throughput.

The whole issue is especially true in the small business area where it is easy to purchase terabytes of primary storage but backup to tape is really time consuming.

Why can't tape produce the equivalent bit density of disk? Is it the more fragile nature of the medium? Clearly tape is more flimsy than a rotating sheet of metal; the T10000 cartridge tape is 6.5 microns thick and the tape itself covers approximately 11.5 square metres, much more than the total surface area of the spinning plates in a hard drive.


I guess we will just have to accept tape capacity will never be good enough. That's just the way it is.


By the way, Sun get a big fat zero in the RSS ratings for not providing their news in RSS feed format....

Destroying hard drives, what a waste

Many financial and government organisations choose to destroy the hard drives that are declared as failing and removed from their arrays. They use products like this which make the hard drive unusable.

What happens to these hard drives? I presume they just end up in landfill and aren't recycled. Is it beyond the wit of man to find another solution?

First of all, a large number of these drives haven't actually failed. They've been marked as having a potential to fail by the array and before a hard failure occurs, the data is moved off to a hot spare. Naturally it is more efficient to copy parity generated data like RAID-5/6 to another drive than to read all drives in the parity group and rebuild the data.

Second, we are imbedding encryption directly into the drive itself. Can't we simply create a drive where the keys can be wiped in the event that the drive needs to be recycled? This seems to me to be the simplest and most elegant solution.

Incidentally, I checked the Hitachi Global Storage Technologies (HGST) website and could only see some nice words about caring about the environment but nothing specific relating to recycling itself.

Green IT with HP

Last Wednesday I passed a pleasant evening chatting with a number of people from HP on the subject of "Green IT". I happen to think that "Green IT" is an oxymoron as IT is never going to deliver computing power using 100% recyclable energy and components. However, IT can certainly improve its green credentials from the position it occupies now.

The HP representatives included the EMEA VP for Marketing, one of the Sales Managers in HP's Power and Cooling Solutions division, the EMEA Environmental Strategies and Sustainability Manager and the UK and Ireland head of Innovation and Sustainable Computing. As you can imagine this gave the opportunities for plenty of lively debate.

For me, there were a number of highlights; firstly HP admitted and recognises that almost all organisations are attacking the green issue not for a sense of altruism but because being green has a direct impact to the bottom line, whether that is in reducing costs or in acquiring new business.

Second, there's the degree of how complex and unstructured the whole green debate is. Is the aim to reduce carbon footprint or to recycle precious resources (like metals)? How should all of these initiatives be measured? What's a good or bad measure? I think I need more time to mull it over.

An interesting side issue of the discussions relates to how HP have selected the bloggers with which to interact. This is being achieved in conjunction with external agencies who obviously follow the market. My concern is how to HP will determine who is an influencer and who is simply spouting hot air. There's got to be a scientific (ish) basis for this; perhaps it's readership size, perhaps it is references to their blog, perhaps it's the level of comments. Perhaps it is based on keyword count and/or other semantic scanning.

However it is achieved, companies like HP will need to ensure that the tranche of their marketing spend directed at bloggers is appropriately spent. It will be really interesting to see how this develops.

Three Degrees of Bill Gates

I heard on the wireless today that Bill Gates has a LinkedIn Profile (well hasn't everyone). Just out of interest, I thought I'd see how far I am from the man himself and I was surprised to find I was only 3 steps away, despite Bill only having 5 connections. I was impressed. Much better than the Six Degrees of Kevin Bacon.

On a related note, I've set up a "Storage Bloggers" group on LinkedIn for those of us who run Storage Blogs online. Drop me a note/comment if you want to join the group (I won't publish those comments). I'll also be putting up a page to list and link the blogs too.

Is the Oyster Failure a Red Herring?

This weekend there was a "fault" on the UK London Underground Oyster card system which caused chaos for many travellers and rendered a large number of cards inoperable and requiring replacement.

This follows the recent cloning of the Oyster card by a group of "Dutch Boffins" (The Register's term, not mine) who had previously cracked the encryption used on the Amsterdam metro (GVB).

I found this story interesting as I travel regularly using both services. It also piqued my curiousity because of the implications on data encryption. It shows that given enough time and ingenuity, it is possible to hack/crack almost all, if not all encryption methods. This has significant impacts for the use of Cloud Storage as a location to store what I would term "Data at Risk" (not Data in Flight or Data at Rest) and re-inforces the need for organisations with valuable data (governments etc) to store their information in secure locations.

Anyway, getting back to this weekend's failure, the conspiracy theorist in me says that TFL decided to change the encryption method used by Oyster to fix the successful Duch crack. This inevitably rendered a number of cards invalid as the cards were/are fixed in nature and couldn't be changed. TFL decided it was better to take the bad publicity of a number of rejected cards than compromise the entire system, because you can be sure the workaround for getting the card recharged would eventually find itself in the public domain.

What do you think?

Seagate Raises the Bar with 1.5TB Drive

Seagate announced yesterday version 11 of their Barracuda hard drive range, to be released next month (August 2008) with a maximum capacity of 1.5TB. The news link has all the speeds and feeds if you're interested in how they have achieved this remarkable milestone.

I've trawled the 'net to plot the release of previous versions of the drive and their capacities at the time. Here's a graph of the releases I could find, going back to 2002. Trending the growth (totally un-scientifically of course), then we can expect to see 2TB drives by December 2008, 3TB drives by November 2009 and 4TB drives by June 2010. This may be a little optimistic as the trending is skewed slightly by the recent advances perpendicular recording has brought to capacity growth, but maybe not, as my recent post on Hitachi 5TB drives shows.

Unfortunately, sustained transfer rates for these drives have remained around the 100MB/s mark, so offloading a complete drive sequentially takes around 250 hours, by my calculations. I'd love to know how long a RAID rebuild would take (Seagate if you fancy loaning me some drives, I'll find out for you!).

As previously discussed, the increased capacities are good as they increase the GB/Watt and GB/cm3 density but we're going to be increasingly challenged by how we get data on and off them - especially when the drives fail.

Distributed Computing Nirvana

Last year I blogged about the concept of storage futures (or options - I'd have to go back and check which) which would allow storage charges to be based on a forward pricing model. The logic for this was to penalise those "customers" who don't bother to take the time and plan their storage demand requirements. By charging more closer to the delivery time of the storage, customers are dis-incentivised to ask for new storage at the last minute.



The evolution of cloud computing has the opportunity to deliver on my original idea. I'm sure I don't have to explain Cloud computing to anyone, but in case you're not aware, it is effectively distributed computing for the Web age. Amazon Web Services is probably the most popular service (in terms of awareness). The Amazon services provide the ability to create a virtual machine, perform database services, manage queues and of course store data in their S3 Simple Storage Service. I'll discuss my experiences on Cloud Storage in more detail in another post.



Many large organisations are facing issues with meeting the power and cooling demands in their datacentres. This is being driven by the increase in computing power and storage density, achieved by the use of blade server technology and virtualisation. Although more computing is being achieved in the same physical space, for some organisations their business model demands more computing to take place in order to gain business advantage. Think of pharmaceutical companies who are using software to model organic chemical interactions rather than perform the experiments in the lab.



I suspect if you investigate the use of computing in a lot of these datacentres, then only a small percentage of the computing power will be dedicated to core business operations. There will be many applications providing anciliary services such as reporting, financials, batch processing, reconciliation, inventory and so on. Many will not be time or location dependent and could easily be removed from the core datacentres for processing elsewhere.



Obviously this change of processing requires a different operating model. A key trend in the industry is to consolidate into a small number of large (and expensive) datacentres, but by operating in this way, companies are artifically constraining their growth into the size of these datacentres and setting a timeline which will require new datacentres to be built before expansion can continue.



So what is the answer? Computing could to move to be location independent except for only those critical components which can't suffer the effects of latency. As an example, take file archiving. If data has been unreferenced for more than a specific time (say 3-6 months) then move it into the storage cloud. The data can be duplicated in multiple locations automatically to provide redundancy. Note that I'm assuming all the issues of security have been investigated, discussed, resolved and implemented.



Immediately redundant data is out of the datacentre and the cost of storage reduced to a service charge, which is likely to be signficantly lower than the cost in the primary location.



Some organisations may decide that the cloud is too unsafe for their data. In this instance this is where a more appropriate datacentre strategy needs to be developed. Rather than having a small number of "megacentres", smaller location critical sites can be built for primary data, with other sites developed in locations offering the cheapest space/power costs. In this way, large organisations could effectively operate their own computing (and storage) cloud.

I think the options for real distributed computing are really exciting. They provide the opportunity to "green" the storage environment over and above the simple task of deploying larger disk drives and bigger storage systems.

Shock - Escargots n'est pas francais!

The BBC has reported that prices in France for snails are set to rocket. I lived and worked in France for a few years and loved tucking into the garlic covered gastropods. But horror of horrors, apparently most snails served in France are now not French! Is nothing sacred?

5TB drives

I just read this on The Register. 5TB drives! Can you imagine it! The HDD manufacturers continue to push the envelope even further.

Now I have a concern about drives getting to this size and that's the ability to get data on/off the drive itself. With 73/146/300GB drives, the capacity to response time ratio is still within a tolerance that means adequate random access throughput can be achieved. But with larger drives the number of different concurrent accesses will increase and if response time doesn't decrease then very large HDDs will start to operate like sequential devices.

I think I need an illustration to make my point. Imagine a 73GB drive is receiving 200 random I/Os per second, each with an average 5ms response time. Scale the capacity up to a 5TB drive and that's about 69 times the capacity. The scaled up drive would have to cope with 13800 I/Os a second and provide an average response time of 0.07ms!

Firstly, it is unlikely 5TB drives will be expected to perform like today's 73GB drives but it serves to illustrate that we can't expect to simply consolidate and shrink the number of drives installed into an array. We need something more.

I think we need a more innovative approach to the design of the drive interface. This may simply be shed loads of cache, to improve the overall average response time, or perhaps multiple virtual interfaces per drive or independently mobile read/write heads which don't need to read/write a cylinder at the same time. It could even be drives that dynamically reallocate their data to make read/write quicker (for example, put frequently read/write blocks in the same physical area of the drive).

Who knows what the solution is, but rest assured something needs to happen to make 5TB drives useful devices.

Monster Mash (Up)

The UK Government is running a competition offering participants up to £20,000 if they can create new uses for existing free sources of government data by combining the data into new and useful information (a mash-up). You can find a link to the data here.

The volume of data is immense. I started trawling the UK census from 2001, burrowing down to the small village I live in. Of the 4700 or so residents, there are three buddhists but disappointingly no Jedi (there's an urban myth that Jedi is put down by so many people who have no specific religious persuasion). Apparently there are no 1-room properties, so no studio flats and no basement flats (no-one lives below ground level). 2.4% of houses have no central heating and only 5.5% of the population are not in good health.

I'm not sure what this indicates other than don't try selling central heating or life insurance policies where I live! But being less facetious, the power in this data will be in deriving new value. I can see two immediate uses/methods.

Firstly, most of this data is useful to people looking for new places to live; schools information, services information, crime levels and so on. So, develop a website and put a postcode in to see how your prospective area rates.

Second, trawl the data and find the ends of the spectrum - the good and bad, best and worst of each metric. For example, which area has the highest population density? Which has the worst crime? This information could be great for business planning; don't set up a locksmiths in an area with the lowest crime rate and so on.

Of course the hardest part will not be to correlate different data sources but in bringing together a consistent view of the information. Some data is accessed via APIs, some by XML, some in Excel format. What "common" point of reference can be used? Postcode? Address?

Whatever, the availability of more and more data content will be absolutely invaluable, but for me, I'd like to see more real time information to be mashed up. For instance, at the airport a live XML feed of flight arrivals/departures that I can read in the taxi when I'm running late (I don't want to have to log onto their website); same for the train; a feed showing my nearest tube station or restaurant as I travel; a feed of the waiting time at all the Disnet rides, so I can pick the shortest queue without having to find a status board; I'm sure there are many many more.

Rich data is a wonderful thing; bring it on!

Keep Your Data for 200 Years - Why?

Courtesy of The Register, I followed their link to a company called Delkin (the data Belkin?) who are touting their premium Blu-Ray disks (BD-R) with a lifetime of 200 years (and 100 years for their DVD-R disks).

Now, this all sounds wonderful; a "guaranteed protection over time" (whatever that means) for your "wedding photos, tax documents etc". The trouble is, and we've been down this road before, having media that survives 200 years is great, but (a) what's going to be around to read it and (b) will the data format still be understandable by the latest software?

Attacking the first, it is conceivable that Blu-Ray compatible drives will be around in 10-20 years' time. After all, we can still read CD-ROMs 20 years after they were introduced and Blu-Ray is already a mass-marked storage platform, not just for data, but for media content too. However, 200 years is a bit hopeful. The 20 years since the introduction of the CD format has seen DVD, Blu-Ray, HD-DVD (!) plus countless other solid state formats.

Data format is more of an issue. I discussed this issue in a recent post.

Another great example of hardware and data compatibility issues can be seen with the BBC's Domesday Project, which used Laserdiscs and non-standard graphical images for displaying information. What's to say that in 50 years time we won't think JPEG just as archaic?

So, don't waste your money on $27 BD-R disks. Buy them cheap, keep multiple copies and refresh your data regularly.

Incipient Revisited

You will remember that I recently posted a comment about migration costs, specifically with relation to Incipient. My view was (and still is) that the majority of migration costs come from preparatory and remedial work rather than execution of the migration. Well, Incipient asked for the right of reply and I had a call last week with Robert Infantino, their Marketing and Alliances Sr VP.

The $5000/TB figure they were quoting was an average they had seen in the industry for certain vendors' professional services time to come in and perform the migration work on behalf of the customer. Incipient's take was that they could provide their appliance/software expertise to provide the same service but at a significantly reduced cost (I won't quote specific numbers here, but the number quoted was much lower than the equivalent cost from "a vendor"). So, I guess with clarification, it is more clear that Incipient were comparing the vendor costs versus their product costs and not including any internal customer costs (project management, preparation work etc) in the calculation. This seems a more appropriate comparison in my opinion.

Getting back to the vendor discussion, there's a real issue here. If vendor X wants to sell you their latest technology, they need to accept and take the hit on helping with migration to their new array. This should be even more so where the vendor doesn't change as this should be a "no brainer" and built into the technology.

In a world where hardware is becoming a commodity, one differentiator will be the vendor who can minimise the effort/cost and impact of migrating from one technology to another. Until then, products like SVC and those from Incipient will continue to have a market position - oh and humble consultants like yours truly!

The Rise of SSDs

Sun recently announced that they will be putting solid state disks into all of their server and storage range of hardware. EMC already have solid state drives for DMX-4, which was announced in January this year. EMC have also stated that they think SSDs will reach a price parity with high end FC drives by 2010.

All of a sudden (and I'm sure plenty of people will claim it isn't sudden) solid state disks are all the rage. For servers, I can see the logic. It's another step in keeping the power and cooling demands of servers down; it also extends primary memory further and will definitely increase performance.

But what about storage arrays? I can see the benefit of putting a tier of SSD drives into DMX arrays, especially in the way EMC have chosen to implement it. It allows those targeted applications to get the performance they require at a manageable price point without a drastic reconfiguration of the array. But an entire array of SSD? That's just the same as existing products like Tera-RamSan.

If SSD prices are driven down, then surely so will the price of standard hard drives. HDD manufacturers aren't going to lie down and let solid state take away their business. We've seen their response already with Seagate taking STEC to court over patent infringements.

So where will it end? Well, tape didn't go away as many forecast it would. I don't see spinning drives going away any time soon either. What I'd like to see is the rise of intelligent storage systems that learn the busy and quiet blocks and move the data between SSD and HDD to keep optimal performance. Meantime, HDD prices will continue to fall and the battle will be between cheap (but fast) HDDs and balancing their cost against the power/cooling they need.

FC Enhancements

A comment posted to my previous blog entry reminds me of a requirement I've had for some time from Fibre Channel. In the "Good Old Days" in my first working life as a mainframe systems programmer, I could very easily see a breakdown of response time against each storage device on an LPAR. Now, the passing years may have given me "rose tinted spectacles" (or more accurately now, contact lenses) of that time, but I seem to remember the reason that I could see seek time, disconnect and connect time was due to the design of the (then) MVS I/O subsystem. As each I/O (CCW) was processed, the hardware must have been adding a consistent timestamp to each part of the process; the I/O initiation, the connect, the disconnect and subsequent seek and then the reconnect and data transfer time to complete the I/O (if none of this makes sense, don't worry, it probably means you are under 40 years old and never wore sandals to work).

Nowadays, the I/O infrastructure is a different kettle of fish. Each part of the infrastructure (host, HBA, fabric, array) are provided by different vendors and have no consistent time reference, therefore tracking the time to execute a storage "exchange" is very difficult. There is (as far as I am aware) nowhere within a fibre channel packet to track this response time at each stage of the journey from host to storage.

If we want the next generation of storage networks to scale, then without a doubt we need to be able to track the journey of the I/O at each stage and use this information to provide better I/O profiling.

Now, just how do I become a member of the t11 committee.....

Storage Migration Costs

I’ve not paid much attention to Incipient (their news page doesn’t provide an RSS feed, so there’s no chance of me seeing their press releases easily), but my attention was recently drawn to a recent release relating to their iADM and iNSP products (catchy names, those).

Now, if you want to know about their products, have a look at their website for yourself. Rather, my interest was sparked by a claim in their press release, quoted below:


The High Cost of Today's Data Migration

Industry estimates and field data captured by Incipient indicate that SAN storage is growing at 40 - 60 percent annually and 25 percent of data under management is moved annually at an average cost of $5,000 per terabyte. Based on these estimates, a data center with one petabyte of storage under management today spends $1.25 million annually on data migration operations. Two years later, the data center is likely to grow to nearly two petabytes increasing the annual data migration cost to nearly $2.5 million.

Source: Incipient Press Release 11 June 2008

So the estimate is $5000 per TB of data movement and 25% of data being moved each year. I can understand the latter; it’s simple logic that if you have a 3-4 year lifecycle on technology then on average 25% of your estate will be being refreshed each year (although that figure is slightly distorted by the fact that you’re also deploying an additional 40-60% each year). Now, how to get to a $5000 per TB calculation...

Excluding new storage acquisition, network bandwidth, etc, I’d assume that the majority of migration costs will be people time. That would include planning and execution of migrations. In environments of 1PB or more, I could (almost) bet my house on the fact that there will be a significant amount of the storage infrastructure which is (a) not understood (b) badly deployed (c) backlevel amongst many other issues. $5000/TB would therefore seem quite reasonable, based on the amount of work needed to refresh. The only problem, though, is that a majority of the manpower cannot be solved by software alone. This will include documenting the environment, bringing server O/S, firmware and drivers up to date, negotiating with customers for data migrations, migration schedule planning, clearing up wastage, new server hardware and so on.

It would be an interesting exercise to determine what percentage of the $5000/TB cost is actually attributable to data movement work (i.e. having someone sitting at a screen issuing data replication commands). I suspect it is quite low. From experience, I’ve been able to move large volumes of data in quite short timespans. In fact assuming sensible preparation and planning, most of the time doing migrations is sitting around (previous employers disregard this statement).

So how much money would Incipient save? My bet is not much.

Ah this is so accurate...

As a contractor/consultant I can totally relate to this list; I'm sure many of you can too.

Simple is Good

I've been doing a lot of travelling recently (rather a lot in fact), mostly in Europe, with a little in the UK between airports. European trains are much better than their UK counterparts - they are reliable, clean, comfortable (note I didn't claim they were fast) and their cost structure is simple to understand. No restrictions about time of day travel, booking in advance or all that nonsense. No. Simply turn up at the station and buy a ticket.

The UK on the other hand must have one of the most complex ticketing systems, especially around London. As an example, if I travel into London from where I live and want to return home between 4pm and 7pm then I can't buy a cheap day return. Presumably that's because they can fleece travellers who don't realise this rule exists. However if I am already in London and want to travel out, I can buy a cheap day single and travel on it between 4pm and 7pm! Even the people selling the tickets think it is crazy. I could give you dozen's of other similar examples, but life's too short.

So it is with storage. Keep it simple. Take tiering as an example. You could spend days and weeks developing the most finely detailed tiering strategy but in reality you will find most data will sit on a small number of tiers, the bulk of it being in the middle range. Developing complex tiering structures, just like complicated train pricing structures just leads to confusion and in the end additional cost. All that's needed is a simple strategy with most of the data on cost efficient storage.

Remember - simple is good.

Dealing With The Consequences

In a remarkable piece of coincidence, two WWII unexploded bombs were found today and caused air traffic delays. The first, at Stratford in east London, temporarily closed London City Airport. The second closed a runway at Amsterdam’s Schiphol airport. It’s amazing that over 60 years after these bombs were dropped, they are still being found; fortunately this time with no injury to anyone. I found out about both incidents because colleagues of mine flying from both City Airport and into Schiphol were delayed or had to alter their plans.

On a less serious level, but important nonetheless, we are having to live with the consequences of our data storage policies. We will have data being stored now which must be accessed in 60 years time. This will include medical and financial information, directly affecting individuals if the information cannot be retrieved. Obviously good data management practices are essential, but they will go beyond the normal storage of data we’ve been used to up until now.

Take the use of medical imagery; x-rays, CAT and PET scans, MRI scans. These all now produce complex digital images. If we store them using today’s format, how will the format and presentation of the images change in the future as we move to more complex display technology, higher resolutions and possibly 3D television screens?

If you want examples of what I mean, think of your old word processing documents. They could have been stored in an early version of Microsoft Word, WordPerfect, WordStar or other similar now defunct technology. You may be lucky and still be able to read them; you may not. Fortunately, apart from some formatting codes, most word processing documents can be opened in Notepad, WordPad or some raw file viewer which will at least allow you to recover the content. Things won’t be so easy with imaging files as their binary nature will render them useless if the software isn’t retained to open them.

I can see one of our future storage challenges will be to ensure all of our data is retained in a readable format for the future. XML addresses some of these problems and the adoption of file format standards will help. However, data will need to be refreshed as it ages. More metadata referring to the content format of files will have to be produced and software written to detect and convert unstructured files as file formats become defunct.

For many of us, we can afford to lose a few files here and there or perhaps print out the most important of our documents. For large organisations, the data management lifecycle has only just begun.

HP Give It Large

Yesterday afternoon I had an opportunity to meet with HP as part of an informal session to make contact with storage bloggers. HP are obviously interested in the possible benefits keeping the blogging community well informed could bring, however my blog is not to act as a mouthpiece for the HP marketing department and I'd suggest if you want to keep abreast of their technology releases, use this XML link.

What's more interesting is where HP storage is headed. Take for example their new Extreme Storage solution. A scalable NAS product which reaches the heady heights of 820TB in a single unit. Fantastic you may think, and I guess if you have a real need for this volume of data in a single unit, then it's the one for you.

However, apart from the obvious issues like whether your raised floor can actually take the weight of a fully configured device (and how do you cool this kind of beast), what troubles me more is how much data on a system like this is actually of any use.

Although the ExDS9100 is aimed at delivering storage for high performance solutions, I think there is a risk of arrays like this being deployed to defer the hard work of actually classifying and setting sensible deletion policies, which, let's face it, for most companies has sat as a task in the "too hard box" for as long as NAS storage has been around. It may well be that some customers see this product as a way to defer the inevitable and actually start managing their data.

Anyway, fair play to HP for entering the market and making use of their Polyserve acquisition and fair play to them for wanting to talk to the blogging community too. If I get any juicy nuggets of information (like whether HP have a position on cloud storage), you can be sure I'll share it here.

UK Email and Phone Database

The BBC reported today that the UK government is planning a tracking database for all phone calls made and emails sent in the UK.

This strikes me as an unbelievably stupid plan. In the first place, every UK citizen will be baulking at this incredible intrusion into civil liberties. Second, it is highly unlikely that the government could ever deliver such a database based on their previous track record with developing and deploying large scale IT projects - think of the UK tax system and new NHS IT system (which will apparently be 4 years behind schedule). Will this database contain all the content of the calls or just a list of who called/emailed who?

Assuming the former, let's do some "back of the fag packet" calculations...

According to the Mobile Operators Association, there were 70 million mobile subscribers at the end of 2006, with an average 100 minutes per user per month and 12 text messages per week. That's 84,000,000,000 minutes and 43,680,000,000 text messages in a 12 month period, the timescale the database is expected to hold data for.

Using the following forum posting referring to Skype, a conservative estimate of bandwidth is 30kb/s or about 225KB per minute. A text message is a maximum of 160 characters. This means the government database would need a 12 month capacity of only 6.36TB to store the text messages (assuming no database or filesystem overhead) but a whopping 17.6PB of storage to hold the voice calls. Let's assume those calls are made consistently over the course of a year, then the system would need to ingest about 600MB/s of data.

Now, I would imagine even the UK government wouldn't be stupid enough to attempt to store the content of all those calls (I didn't even attempt to calculate the email traffic). If they don't, let's face it, you have to question what the point is, if the content isn't being recorded as anyone with any nefarious intent will simply use anonymous pay-as-you-go SIM cards and bypass all the tracking. Still, at an efficiency of say, 40%, the 44PB potentially needed could set up the lucky EMC or HDS salesperson for life! I wonder if any of them have done the same calculation as me....?

Nexsan going public

I've been reading the Nexsan statement announcing their intention to go public. I don't know much about their products other than they sell high density storage systems. What intrigued me about reading the announcement is the amount of exposure to the internal operation of the business occurs when statements are made to SEC as part of a flotation.

For instance, Nexsan have never made a profit, although their losses are declining year on year. They list their competitors, including all the usual suspects like EMC, HDS and HP, but quote Dell and Equallogic (now Dell) at the beginning of the list; this gives you an idea of the position in the market they see themselves in.

Nexsan don't run a direct sales force; they rely on Kodak for their support arm; they are reliant on Bell Microproducts for their disk drives; they rely on three subcontractors to manufacture their equipment; they are reliant on channel partners to install products at client sites.

So if I invested, what exactly would I be getting? Well I guess IP is the key value and that's exactly what the company will be rated on the value of. The more Nexsan can reduce costs by removing the dependency on external suppliers and the more they can acquire and use their own IP, then the more chance the company has of reaching that profit goal.

Remember 3Par? They launched last November at $14 a share and are now trading at a more $8.37. I wouldn't be betting the kids' college fund just yet....

Techdirt

I recently joined TechDirt; here's my first post if you're interested!!

http://thefutureofstorage.com/archives/23

Drobo Update

I’ve had my Drobo for a few months now. For those of you not familiar with the technology, the Drobo is a storage device from a company called Data Robotics. Follow the link above to their website for full details.

I’d been looking for a decent home/home office storage device for some time. RAID support was a must and initially I thought I wanted NAS because my solution at the time was to keep a server running continuously. The server performs other tasks and I was using it for file serving too.

Previously I had taken the plunge with the Linksys NSLU2 which runs a modified version of Linux. Unfortunately at the time, the device only supported ext3 filesystems and as I loaded the device with more data, responses became erratic and I found the exported systems going read-only and losing content. Lucky for me the problem seemed to be the device rather than the actual data on disk and I was able to recover everything using a little software utility which allowed me to read ext3 devices on Windows. This experience scared me and it was time to look for something else.

The Drobo hit the spot for a number of reasons; firstly it was a dedicated device which took SATA II drives. It has a USB connection, allowing me to plug it directly into my existing server and third (and at the time most importantly) Data Robotics had just released a NAS head which could be used with the standard Drobo, or removed without affecting the format of data on the device itself.

So, as I said, I’ve had it for a few months and what is there to say about it? Well, not a lot. It works – and so far has worked flawlessly. But there are a few things of note.

Firstly, I have a BIOS incompatibility issue; when my server reboots, if the Drobo is plugged into a USB port, it hangs the server. I haven’t bothered to resolve this yet; weighing up the relative merits of just living with this problem or upgrading the BIOS on my server, then I fall squarely on the side of accepting the workaround of unplugging the drive at boot time and plugging it back in as the system comes up. If I was using a standalone PC, then I would obviously have fixed the problem.

Second, I was interested to see that despite my system having two 1TB drives and RAID protection, the X: drive I’d created reported back a 2TB file system. Was RAID on or not? Well, yes it was; the Drobo presents a 2TB file system regardless of the drives you have installed. It’s virtualisation in action! As you allocate all of the physical storage available, you get prompted to add or swap drives to match the physical demand. I like this feature as it’s a painless way to upgrade your storage over time and as terabyte drives drop in price (currently I’ve seen them at 1TB for £99) it helps smooth out the cost of upgrade because drive sizes can be mixed and matched.

Last, there’s the issue of firmware upgrade. Version 1.1.1 of firmware is available and it was a simple task to upgrade, however I can’t implement the code without rebooting the Drobo and that requires closing all the active files accessing the Drobo on the server. This is not a major problem though and wouldn’t be a problem on a standalone PC.

All in all, the Drobo looks good and does the job. Having 1TB of new capacity has encouraged me to spend time moving my data over in a controlled and structured fashion. The process will take months (a subject I will return to), but in the meantime I have bags of spare capacity and an easy upgrade path for both additional capacity and NAS connectivity.

Now, if anyone out there would like me to review their NAS product, then I’d be only too happy….

FCoE

Fibre Channel over Ethernet has been back on my radar recently, especially as it was touted again at Storage Networking World in Orlando last week. Unfortunately I wasn’t there and didn’t see for myself, although I was in Orlando the week before on vacation. I can imagine if I’d extended or moved the holiday to include SNW that I’d be none too popular with Mrs E and my sons.

Any hoo, I looked back over my blog and I first briefly mentioned FCoE back in April 2007, a whole 12 months ago. Now, we know 12 months is a long time in the storage world (in which time iSCSI will have claimed another 3000% market share, EMC will have purchased another 50,000 storage companies of various and dubious value, HDS will have released nothing and IBM will have developed 2 or 3 new technologies which won’t see the light of day until I’m dead and buried). I expect then that FCoE should have moved on somewhat and it appears it almost has. Products are being touted, for example, Emulex with the LP21000 CNA card (not an HBA card, please note the new acronym) and Cisco with their Nexus 5000 switch (plus others).

At this stage I don’t believe the FCoE protocol has been fully ratified as a standard. I have been spending some time wading reading through the FC-BB-5 project documentation on the T11 website, covering FCoE to understand exactly how the protocol works in more detail and how it can be compared to native fibre channel, iSCSI, iFCP and FCIP. In the words of Cilla, here’s a quick reminder on storage protocols in case you’d forgotten.

Fibre channel and the Fibre Channel Protocol (FCP) provide a lossless, packet based data transmission protocol for moving data between a host (initiator) and a storage device (target). FCP implements SCSI over fibre channel. To date, fibre channel has been implemented on dedicated hardware from vendors including Cisco and McDATA/Brocade. iSCSI uses TCP/IP to exchange data between a host and storage device using the SCSI protocol. It therefore includes the overhead of TCP/IP but provides for lossy and long distance connectivity. iFCP and FCIP are two implementations which encapsulate FCP in TCP/IP packets. FCIP extends an existing fibre channel SAN, whereas iFCP allows data to be routed between fibre channel SANs.

FCoE will sit alongside fibre channel and allow the transmission of FCP packets at the Ethernet layer, removing the need for TCP/IP (and effectively allowing TCP/IP and FCP packets to exist on the same Ethernet network).

So hurrah, we have another storage protocol available in our armoury and the storage vendors are telling us that this is good because we can converge our IP and storage networks into one and save a few hundred dollars per server on HBA cards and SAN ports. But is it all good? Years back, I looked at using IP over fibre channel as a way to remove network interface cards from servers. The aim was to remove the NICs used for backup and put that traffic across the SAN using IPFC. I never did it. Not because I couldn’t; I’m sure technically it would have worked, but rather because the idea scared the willies out of “the management” for two reasons (a) we had no idea of the impact of two traffic types going over the same physical network and (b) the Network Team would have “sent the boys round” to sort us out.

Will this be any different with FCoE? Will anyone really be 100% happy mixing traffic? Will the politics allow the Networks teams to own SAN traffic entirely? Let’s face it, in large environments I currently advocate the separation of host, tape and replication traffic to separate fibre channel fabrics. I can’t imagine reversing my position and going back to single consolidated networks.

So then, is FCoE going to be better in smaller environments where the consolidation is more practical? Well, if that’s the case, then surely that makes FCoE just another niche player to FC, just like iSCSI.

It’s early days yet. There are a million-and-one questions which need to be answered, not least of which will be how FCoE will interoperate with standard FC, how drivers will interact with the existing storage protocol stack on a server and how performance/throughput will be managed. Some of these issues have been answered, however this blog entry is already far too long and rambling to include a discussion on these points this time and I will save them for another time.

May The Force Be With You

Just had to share this with everyone if you haven't already seen it...

On a PC (or server) with Internet connectivity, type "telnet towel.blinkenlights.nl" (without the quotes).

Assuming you have telnet and your firewall allows it, sit back and enjoy!

When Storage Planning Goes Bad

I was chatting to colleagues today and we were reflecting on an installation which had just completed and needed another additional storage tranche installed. Ironically, the initial disk installation on the new array hadn't been fully implemented because the vendor "forgot" to install the full quota of cache in the array. Although this was a simple gotcha, it reminded me of others I've had along the way in my career including;

• An engineer was testing the Halon system in the newly completed computer room extension at my first site. Unfortunately he'd forgotten to turn the key to "test" before pressing the fire button and let off the Halon in the whole of the datacentre with both the equipment up and running and operators in the room mounting tapes. Needless to say, they were out of there like a rat up a drainpipe!
• During a recent delivery of storage arrays; one array literally fell of the back of the lorry. It had to be shipped back for repair...
• An array installation I managed in one site was mis-cabled by both the electricians and the vendor. When it was powered up, it exploded...
• On a delivery of equipment, the vendor arrived at the loading bay at the datacentre. As the loading bay door was opened, it jammed and broke, just too low for the arrays being delivered to be pushed under the door. The vendor had to return the following day after the broken door had been repaired.
• A tape drive on a StorageTek library I worked on took 12 hours and around 6 staff to complete. Half way through the upgrade, we took a go/no go point and checked both the MVS and VM installations to ensure the new drives worked. The MVS connected drives were fine; the VM drives had a "minor problem", so we proceeded, in anticipation of resolving the VM problem. The following day we discovered the VM problem was not correctable and had to purchase additional drives at considerable cost.
• After loaning out some disk space to a "temporary" project, we had a hardware failure 3 months later. It turned out that the team had forgotten to ask for backups for their data and 3 months of the work of a dozen people was lost.

Fortunately, most of the above were not life threatening (except the first, which I was not involved in directly). However one of these problems did result in data loss (albeit on a development environment). It shows how many times the unexpected and unplanned can happen and mess up the best laid plans.

Care to share any of your stories?

Multi-vendor Storage

Reading Chuck’s blog during my vacation, I stopped on his comment that multi-vendor environments are on the decline due to their complexity and the hassle of dealing with multiple vendors.

I have to say that firstly, I don’t believe this and secondly and companies with large storage environments would be mad not to consider a multi-vendor setup.

The reasons people have problems with multi-vendor environments are because they don’t spend time turning their storage into a commodity. EMC and HDS both recommend their own LUN sizes; each will sell you their management solutions; each will have their own support matrices.

But these things can and should be standardised. It is a simple task to define and migrate to consistent LUN sizes, regardless of vendor hardware. Software tools can be simplified; most people choose use command line or the basic configuration tools rather than the bloated EMC tools, so no problem there. In addition, scripting can be developed for failover and PIT/Snapshot management, making their use generic across vendors. Finally, driver/firmware/HBA/fabric standards can all be established to converge on a common set across all storage vendors.

Once storage has been established as a commodity, any new purchases can come from any of the vendors in your multi-vendor strategy.

Oh and one last thought; do you really believe HDS, IBM and EMC would give you the absolute best price if they know you can only use their product on most of your server farm environment? Competition within the storage market is a false premise; moving to another hardware platform to replace an existing one takes months (in some cases years). Vendors know that customers’ threats to move everything to another platform are only hollow unless you have a true multi-vendor strategy.

Hitachi and Hardware II

After my last post I think I should have been a little more specific. Whist I'm aware HDS is owned by Hitachi, I pointed out that HDS is effectively a reseller for their hardware. In my experience, HDS take the hardware as Hitachi deliver/design it. I've not seen a huge feedback loop where hardware futures are developed with HDS (although HDS do seem to feed back a lot of software requirements).

My point was, HDS don't purchase companies as they are just Hitachi's reseller and Hitachi don't purchase at all. Either way, whether Hitachi or HDS, the company needs an obvious strategy.

HDS and Hardware

I’m away on holiday at the moment (which is a blessed relief, looking at the Easter weather in the UK, no pun intended) however catching up with some emails I noticed the supposed post on Storagezilla’s site that IBM are to purchase Diligent.

I nearly worked for the UK arm of Diligent many years ago when they were getting started, but that’s another story. What struck me about the comments ‘zilla made with reference to IBM, Moshe and IBM’s acquisitive streak was the predicament HDS finds itself in.

HDS traditionally partners and doesn’t purchase companies however they don’t own their underlying hardware – the product is produced by Hitachi and resold by HDS and HP. The recent IBM acquisitions and EMC’s relentless purchase of anyone with a storage product mean HDS is falling behind the game. They currently have 2 product lines; enterprise hardware and modular hardware, neither of which they own. With IBM and EMC making inroads into acquiring more hardware and software tools to build entire solutions, how could HDS hope to compete going forward?

I’m not sure if HDS have a relationship with Hitachi which guarantees them access to storage products and a say in product design, however if they don’t then I can’t see how they will compete with the big boys. HDS need to look to acquire technology and start producing integrated solutions pretty quickly or they are going to get squeezed out of what looks to be an increasingly competitive and solutions’ based market.

Oh, one other little footnote; my 10-year old son bought a iPod Touch at the airport and fortunately for him I’d brought my laptop with me. Unfortunately, he didn’t understand why I couldn’t get online and download iTunes to configure his new toy while flying along at 35,000 feet…

Controls in Place

I work with many companies in which the storage and server/platform teams have to sign in blood in order to receive suitable access to perform their day to day work. In fact, for some organisations, it can equate to 50% or more of their time. I've been there and I understand and have experienced the frustrations it creates.

Compare this to our trader friends. Jerome Kerviel allegedly managed to gamble €50bn, more than the assets of his bank, Societe Generale. I wonder how many change records he had to raise in order to get access to that level of funds?

HMRC Get Better And Better

Following on from the "inadvertent" loss of 25 million personal records, it appears that good old Her Majesty's Revenue and Customs have decided that online tax filing is not as secure as they may have indicated in the past (see BBC link). High profile celebrities, MPs and the Royal Family will not be allowed to post online. I have a feeling that HMRC will be moving to force all of us in the UK who submit a tax return to submit the return online in the future, so the implication is that it's OK to lose the records and compromise the personal and finanical identity of the general public (without recourse, by the way).

Is data security that bad? I really hope not, as our future depends on it.

My Drobo Has Been Ordered

I've been mulling over for some time whether to purchase/evaluate the Drobo storage device. Up to now I've been put off by the lack of NAS support, as the device only supports USB2.0. However all that has changed; I read today that there's now the DroboShare, which converts the Drobo into a NAS device. I've yet to see the full details (and I doubt the DroboShare will not be available some time in the UK) however this now justifies my purchase (as no-one has offered me one so far!).

The order is in - for Drobo and two 1TB drives. I can't wait...

EMC Stretches DMX Both Ways

EMC made two DMX-4 announcements today. At the top end, the first details solid state drives which replace standard spinning HDDs. At the bottom end the second announces new 1TB SATA-II drives (and other miscellaneous stuff).

The solid state hard drive (or as Chuck Hollis describes it, the enterprise flash drive), isn't just a small 1-2GB device; this is a real replacement for 73GB or 146GB drives and will apparently slot right into a standard HDD bay. Assuming drive support is identical to standard HDDs, then it should be possible to create hypers from the drive(s) and present them to many separate hosts. I see this as significantly better than products such as Tera-RamSan. Firstly, the EMC device is more compact, second and most important, it integrates with the existing array, meaning better compatibility, support for existing functionality such as SRDF and TimeFinder and therefore more consistency.

Obviously a real issue will be cost versus standard HDDs, however I think there will be plenty of takers. Every large organisation has one or more application which needs more storage horsepower and the tradeoff between rewriting the application, purchasing more hardware or using SSD HDDs in DMX will make the latter option very appealing. Hopefully as time goes on, the cost of SSD drives will reduce significantly (think of how expensive SD cards used to be) and SSD technology will be more attractive to a wider audience.

Naturally the blog spin masters at EMC have been active; here's just a selection of the links I found:

• Chuck's Blog
• Confessions of an eBiz Junkie
• StorageZilla

Nothing yet from The Storage Anarchist though. Give it time...

The second announcement covered availability of 1TB for DMX-4. Fair play to EMC, they are offering storage technology across the spectrum of speeds and capacities, all in a single box. I still need to think through the implications of having even more choice in one array, not least of which is the impact on my design methodologies!

Oh and the second announcement re-announced thin provisioning for DMX-4. But everyone else is already doing that already. :-)

I Stand Corrected

In a previous post earlier this year I mentioned the Onaro purchase by Network Appliance. As I said at the time, I wasn't aware Onaro's SANScreen product even had a NAS module. It seems I was wrong, and thanks for Deni O'Connor for indirectly pointing it out. In fact, SANScreen now has NAS Insight which provides for NAS monitoring support. However this feature was only made general availability on 31 December 2007, so you can hopefully excuse my oversight for not realising it has been released.

(On a side note, why are large Enterprises such as Onaro still not using RSS to announce product releases? I haven't got the time or inclination to trawl their websites each day. RSS is so much easier.)

I had a quick look at the NAS Insight press release and details on their website. Although there's a demo, I couldn't ascertain what NAS products (other than Netapp filers which are in the demonstration) the product supports. That makes me think it supports nothing BUT Netapp (although I again stand to be corrected). If that's true then NAS Insight is a pointless feature for many customers who would want to use the product for cross vendor consolidation and certainly doesn't demonstrate Onaro's NAS credentials. Compared to the last release of DFM I saw, NAS Insight is pretty poor.

To date my SANScreen exposure has been based on one large "global installation" of the product and presentations from the Onaro marketing team. When an instance of SANScreen was enabled in one location of the global deployment, it created thousands of exceptions which then required manual intervention. When I last had a presentation on the product (in October) there was a large number of SAN scenarios SANScreen wasn't reporting on, a lot of these relating to non-EMC and replication support. There's still a long way to go yet.

Up to this point, Onaro may have developed relationships with the vendors which provides for ongoing access to new releases of their management tools in order to extract configuration information. Going forward, will those companies still be as keen to provide that information to Netapp, who may be their direct competitor in the NAS (and non-NAS) marketplace?

Memory Lane

I thought I would spend some time tonight looking back at my blogging history. I first started writing stuff on a website I created called Storage Eye. The first "article" I wrote was posted in February 2002, almost 6 years ago. It was pretty crude stuff (the presentation, not the content), simply putting static pages up on a website. I followed that up with postings in 2004 on ITToolbox and then in 2005 back to my own website (this time database driven), eventually cutting over to blogspot at the beginning of 2006.

My material hasn't changed much over time. The main difference is probably that I post more often and with longer content but that it probably because the tools available now make that really easy.

What has definitely changed are the tracking tools and the ability to locate bloggers who are out there. I track 69 storage blogs using RSS (and quite a few more non-storage related). Tools such as Technorati, Digg, Stumbleupon, allow blogs and their feeds to be tracked and rated. The blogosphere is active and vibrant and full of lots of great content.

Where's it all headed? Who knows, but I am sure it will be fun!

XiV Part II

Following on from BarryW's comment to my XiV post, I've been thinking over how the XiVarchitecture works. When a disk fails and the missing mirrors need to be recreated, then the data is likely to exist across all or most of the configured drives. Logically it would make sense that the target for the new lost mirrors would be all drives. So, as a drive fails, all drives could be copying to all drives in an attempt to ensure the recreated lost mirrors are well distributed across the subsystem. If this is true, all drives would become busy for read/writes for the rebuild time, rather than rebuild overhead being isolated to just one RAID group. Whilst that seems like a good thing for rebuild time, it seems like a bad thing for performance. Perhaps this isn't the case and in fact the failed device is re-created on a spare drive by copying all the mirrors back in from their other location.

Following the same line, in order to recreate a failed drive and rebuild the lost data across the array, then each drive must have spare capacity; in say a 200 drive system, that would require about 1/200th of all drives to be free at any one time, ready to receive rebuilt mirrors. Obviously the alternative option is just having spare drives, but that sounds less interesting!

What about when the failed disk is replaced? There must be another algorithm which ensures the replaced disk is not a target for all new writes, so presumably, static mirrors are pro-actively moved onto the replaced device.

This architecture throws up some interesting questions, especially when trying to understand performance. I am starting to get excited about messing about with one!

2.5" is cool

Hitachi Global Storage Technologies (the old IBM Disk Division) have announced a 500GB 2.5" drive for laptops. Not particularly interesting I hear you say, however these latest drives from HGST now come in Standard and "Enhanced Availability" versions. The enhanced version claims to be able to run a 24/7 workload rather than standard day to day access required in a laptop.

So does this mean these SATA drives are suitable to use to build an array? Well, first of all they are not fast ast 5400RPM. It would also be nice to have the SAS rather than SATA interface, but just look at the power consumption figures; 1.9W in read/write mode!

How about IBM taking the XiV technology and using 2.5" drives with this power consumption and capacity - now that would give EMC something to complain about!

Two for the price of one

The holidays are over and it's back to work for me. In fact I returned yesterday; the break was good however it is also good to be back.

It seems that I've returned to a flurry of acquisitions. Yesterday there was the heavily reported (on the blogosphere) purchase of XiV by IBM. Tony Pearson gives a summary of the features on his post. One thing that interests me is the use of distributed writes across an entire array by creating 1MB blocks from (presumably) LUNs and filesystems. If a drive fails, then the data is still available on other disks in the system and spread across a great number rather than a single drive (RAID-1) or potentially a small number of drives (RAID5/6).

I've been trying to get my head around what this means. On the one hand it sounds like a real problem, as a double drive failure could impact a wide number of hosts; it all depends on how well the 1MB chunks are distributed. However maybe it isn't that much of a problem as the issue only arises when both of the chunks that mirror a 1MB block both occur on failing drives. I would expect that as the number of physical drives increases then the impact of double failure reduces, as does the number of 1MB blocks affected. In addition, a drive may fail only in one area rather than on the whole device, so the affected blocks could be quite small; the remainder could be perfectly readable and be quickly moved. No doubt Moshe and the team have done the maths to know what the risk is and compared it to that of standard arrays and wouldn't be selling the product if it was not inherently more safe.

The only other issue I can see is what market the product will slot into; Tony mentions that the product is not for structured data (although I guess it supports it) but was designed for unstructured data of large binary file types. So, why use RAID-1 compared to say a 14+2 RAID-6 configuration which would be much cheaper in terms of the disk cost? Presumably another selling point is performance, but I would expect the target data profile (medical, large binary objects) to be more sequential than random access and not be that impacted by using SATA.

I guess only time will tell. I look forward to seeing how things go.

The other purchase announced today was that of Onaro by Netapp. Onaro sell SANScreen, a tool to collect and analyse fibre channel SANs and to highlight configuration issues. Whilst I think it is a good product, I don't see the fit with Netapp's business in the NAS market (in fact I'm sure SANScreen doesn't currently support NAS), so where's the benefit here other than buying up a company which must be close to or is making money.

I wonder who will be bought tomorrow?