Beating the Credit Crunch

Yesterday was probably the worst day for the credit crunch and the global problems being experienced by banks. Recession is biting and without a doubt many companies are looking at how they can reduce their storage costs.




Reducing costs doesn't mean buying new hardware. Ask yourself some searching questions. Am I sure my configuration is optimally configured? Can I release any overallocated storage? Can I reclaim any orphan LUNs?




The picture on the right shows how storage attrition occurs as raw disk is carved up, converted into LUNs and presented out to hosts. There's plenty of room for resource recovery.



This diagram doesn't even cover orphan storage. Typically I've found anywhere between 10-20% orphan resources reclaimable in large environments. This is free money you can get back tomorrow (or today if you're quick). Most SAN arrays have:

1. LUNs presented to storage ports but not masked to anything


2. LUNs presented to hosts which have "gone away" - decommissioned, HBAs changed etc


3. LUNs presented to HBAs/hosts for which no zoning is present


4. Snapshot (BCV/ShadowImage) volumes never synchronised

This is only a small subset of what can be reclaimed. With a bit of time and the right tool, you could even delay that next storage purchase and keep the boss happy!!

HP Power Calculators

A while back I mentioned a meeting I'd had with HP regarding Green IT (and in my case specifically storage). As part of the discussion, I asked Lucio Furlani (VP for Marketing in TSG) if we could have Power Calculations for HP storage.

Well, true to their word, HP promised and have now delivered the first of a number of storage Power Calculators, available here: http://egui-prod.houston.hp.com/eGlue/eco/begin.do.

The first calculator is for the EVA4400 and there are more to follow. Get on-line and have a look and feed back what you think!

On the power calculator front, EMC, HDS (and now HP) all produce power tools for their storage products. EMC and HDS need to step up and make their tools public to anyone - after all if you believe your product is superior, then you've nothing to hide.

Does anyone know of calculators for IBM, 3Par, Pillar, Netapp etc?

InVista - EMC's Problem Child

Remember when EMC offered a fabric-based storage virtualisation product called InVista? Apparently they still do, as a customer of mine mentioned recently. Unfortunately they had trouble getting any sort of decent reference site for the product and weren't that impressed.

I had a quick check on EMC's site and the last press release for the product I could find was dated December 8th 2007. Compare that to the news about DMX, Clariion and probably almost every other product put out by EMC and InVista doesn't get much press. In fact general opinion perceives InVista as EMC's problem child, finding trouble gaining acceptance with storage managers at large. Why?

To be honest, I'm not sure why. The *concept* of fabric-based virtualisation seems appealing and once the appliance has been integrated into a SAN, data migration should be simple. Perhaps there's a lack of trust in the scalability or performance, perhaps a lack of trust that the product actually works, perhaps a nervousness in placing yet another layer into the infrastructure which, if it fails, could be massively disruptive to normal storage operations.

That leads to the question - what should EMC do? I doubt they will drop the product, as that will give competitors (especially IBM & SVC) too much of a marketing opportunity. Maybe they will acquire a competitor (like Incipient) and slowly transform InVista into a new product. I don't have a crystal ball to answer that one.

I'll close with one thought - are there any other products with the word "Vista" out there that are also equally dismissed and derided?

How much is your hard drive worth?

Have you ever thought of renting out your hard drive?


So here's the problem. Everyone needs backup. Most people are too lazy or inept to perform backups regularly. There are tools out there to use. There are online tools too which will help with the problem - but there's also a fee for using them (more about that later).


The thing is, we all have plenty of free space on our hard disks and this is set to increase as we deploy larger drives. So why not "rent out" a chunk of your unused storage to store other people's backups and backup/restore using a peer-to-peer model?


Now, there are some drawbacks. If your backups are dispersed across multiple servers, when you want to perform a restore, your data may not be available online. You could store multiple copies on different servers, but that then becomes more costly. It may mean you have to give up more of your hard disk freespace than you currently store, in order to get access to duplicate backup copies. Why? Well, for a shared backup service to work, everyone needs to provide an equitable amount of storage - you provide as much space for backup as you use.


A solution to this is to de-duplicate. There is a big opportunity to remove redundant copies of data (think of all your operating system files) which, if files are broken up into chunks, could provide the ability to remove a significant amount of redundant information.


However, here comes the killer blow. De-duplication will not work. It will not work because you will want to encrypt your data if it is going to be stored on someone else's server or PC. Even worse, this data is going to be stored on the server of someone you don't know and you can guarantee that means some of those people will be less than honest and looking to gain criminal advantage from accessing your data.


As soon as encryption is brought into the mix, de-duplication can't be used to reduce the volume of data stored as everyone's encrypted data will essentially look like random data and therefore the peer-to-peer backup & de-duplication model becomes untenable.



I mentioned earlier using online backup services and this weekend I subscribed to Mozy, which does offer a free 2GB service. Unfortunately free also means restricted, so I couldn't backup any data on network shares. This made my test a little limited, but at least I could try the features Mozy offer by backing up some data from my laptop hard drive. After an hour watching my backup data being encrypted and crawling to the Internet at a snail's pace, I gave up.



To be honest I don't know why I was so surprised with the limitations of online backup. A little over 10 years ago, I was involved in a project at StorageTek where we looked into offering an online backup service. The software was there, however telecoms was the inhibitor due to the cost of connection. At the time (especially in the UK, but less so the US), there was a lot of "free" Internet access around, which although free as a subscription, relied on local rate 'phone calls to make money. This is great for infrequent browsing but no good for a permanent connection and the cost to the customer of being online overnight was too high, and so the project was eventually shelved.



Although the cost base has shifted and "all you can eat" broadband is common, services like Mozy are not free. The "Pro" service is $3.95 plus $0.50/GB, per month (I assume this means there's a monthly $3.95 although it isn't clear). For my almost 1TB of data, I'd be looking at close to $500/month or $6000 a year!



When I see a bill like that, two things come to mind:



(a) - what is all this data and do I *really* need it? What can I prune?

(b) - how can I structure my data better so I only back up the stuff I really need to secure online? What other ways are there of protecting my content that doesn't change much?



So, I intend to not waste my time writing a P2P backup service. Instead I will invest some time structuring my data, cleaning up the garbage, looking at one-off static content backups and other options for that data (like DVD/Blu-Ray backups kept in a firesafe) and perhaps use cloud storage backups in a limited fashion.

Lehman Brothers hits the rocks

The latest casualty of the credit crunch is Lehman Brothers who have filed for Chapter 11 Bankrupcy protection. See the report here from the BBC.

I worked for Lehman in the UK for a short period about 4 years ago. If their banking operations was run anything like IT, then there's no surprise they are out of business today.

I joined as a consultant after the entire storage team left for other jobs and within 6 months moved on myself after the outsourcing Sword of Damocles hovered over the team. The storage operation in place was fundamentally flawed, however based on the attention Lehman's will be getting, this may not be the best time for me to expand on those issues in writing. :-)

Cisco WAAS Deployments

I'm currently working on a project which is investigating Cisco WAAS deployments and I'm looking to validate successful deployments of the Cisco products in a geographically distributed environment.

Has anyone done this recently? More pertinent, is anyone prepared to share their experiences with me (in confidence if necessary)?

Large Hadron Collider, Data, Sweden and Nuclear Rockets

The testing of CERN's Large Hadron Collider has been widely reported over the last few days and weeks in the media. You can find more details here.

In simplistic terms, the project is looking to re-create the conditions experienced in the Big Bang at the start of the universe in an attempt to detect elusive atomic sub-particles, which should help to solidify and prove the Standard Model.

There has been a lot of speculation that the testing could lead to the creation of a black hole and implosion of the earth and the universe. In fact, my son came home from school yesterday and told me the rumour in the playground was that Sweden were intending to launch two nuclear missiles at each other that would lead to the destruction of the earth!! Obviously, I had to bring his astrophysics knowledge up to scratch. I guess I shouldn't be too harsh on him - he's only 10.

Anyway, the interesting storage angle is in how much data the project will produce on an annual basis. According to this rather helpful PDF, the project will produce data at a rate of 700MB/s or 15 petabytes (PB) per year (presumably the collider will not be run 24 hours a day). That's a lot of data, especially when you consider the project will take 2-3 years to produce enough data to detect the elusive higgs-boson particles!

The collected data will be analysed by the LHC Computing Grid (see here for details). Managing this data, which will be distributed down 4 tiers will be an incredible job, mainly for the complexities of maintaining concurrency of access to the data and all the analysis results.

Let's hope it works!!

Moshe Joins The Blogosphere

Moshe Yanai of EMC, XIV and now IBM fame (I'm sure I don't need to fill in the details) is now blogging. You can catch up with him here.

As previously requested, I'm going to start posting my RSS blog feeds, starting with IBM.

• Tony Pearson - Inside System Storage Blog - Homepage - RSS - Rating: *****
• Barry Whyte - Storage Virtualisation - Homepage - RSS - Rating: *****
• Moshe Yanai - Think Storage - Homepage - RSS - Rating: N/A

The Rating figure is an indication of how often and how useful I find this blog - it is *not* any comment on the quality of the writing or author, before I start to get comments on any sort of bias!

IBM's Storage Announcements

Today IBM made a slew of new product announcements. None of it was a surprise due to lots of pre-announcement leaks and the fact the IBM session in Montpelier was to a certain degree the formal announcement of products already mentioned in standard press releases.

I can only describe the new product releases as slow and steady with nothing radical or earth shattering. What it does do is consolidate IBM's product range - slightly better tape libraries, slightly better tape drives; midrange SAN appliances, archiving appliances and a reconfirmation of the latest releases on SVC.

The release is here.

One point of note, IBM list their acquisitions in a PDF at the bottom of the release (get it directly here). IBM have acquired a lot of both hardware and software companies and this echoes somewhat of EMC a few years ago. What's interesting is to see how IBM will integrate and harmonise these products, rather than sell them as a disparate set of technologies. Should be fun...

Whatever Happened to Hybrid Drives?

Over 18 months ago, I discussed in a post (here) the Hybrid Storage Alliance who were looking to increase the performance of hard drives by adding large amounts of flash to each device. But where are they now?

A quick check on their website shows precious little information. In fact what's on offer seems to point to using hybrids for just laptops and Vista. Unless they're working on new products, it seems to me that the manufacturers of hard drives have missed a trick.

EMC have made a clear stance to move towards flash (with their EFDs) but in reality they're targeted as specific applications requiring super-fast performance. In any case, the cost will make them prohibitive for most customers (initially, at least). Drive speeds are fairly much plateaued at 15K (although some manufacturers are discussing 20K spin speeds) as the power/cooling issue is obviously a problem when going at faster revolutions.

Taking these factors into consideration isn't there a gap in the market for Enterprise Hybrid Drives? I'd like to stake a claim for the term EHD at this point before anyone else does :-)

It seems to me like a logical step; manufacturers such as Seagate won't want to give up their dominance to flash in a hurry and in any case flash is too expensive for most usages.

Perhaps there are issues getting EHDs to work; you wouldn't want to spin them down (although you might accept them spinning slower); maybe there are problems with effectively filling and destaging the flash cache on the drives which requires more thought from the array manufacturers. Either way I'm speculating but it seems there's an opportunity there for the taking.

Remember, Enterprise Hybrid Drives, heard here first!!

DMX Configuration Options

I've been looking at DMX configuration options this week. Essentially the question is how best to lay out a DMX-3 or DMX-4 array with a tiered configuration. For me there are two options and it's pretty clear which I prefer. First a little background. The following diagram shows the way DMX drives are deployed within a fully configured array. The array is divided into "quadrants", splitting each drive bay into two. Back-end directors (BED) provide connectivity to the drives as represented by the colour scheme. There are up to 4 BED pairs available for a full configuration.

Option 1 - Dedicated Quadrants

One option is to dedicate quadrants to a workload or tier. For example tier 1 storage gets given quadrant 1. Theoretically this should provide this tier with uncontended back-end bandwidth as all the tier 1 storage will reside in the same location. What it doesn't do is let tier 1 storage utilise unused bandwidth on the other BEDs, which as the array scales, may prove to be a problem.

There's also the question of physical space. As the drives are loaded into the array, if only a small number of them are tier 1, then potentially cabinet space is wasted. Either that or the configuration has to be build in an unbalanced fashion, perhaps placing more lower tier storage to the right of the array, using the expansion BEDs.

The second diagram shows how an unbalanced array could look - tier 2 devices on the left and right are loaded at different quantities and so lead to an unbalanced layout.

Option 2 - Mixed Workload

In this option, disks are spread across the whole array, perhaps placing tier 1 disks first followed by tier 2 devices. In this way, the I/O load is spread across the whole configuration. As new disks are added, they are distributed throughout the array, keeping performance even. The risk with this configuration lies in whether tier 2 storage will affect tier 1, as the array becomes busy. This can be mitigated with Cache partitioning and LUN prioritisation options.

I prefer the second option when designing arrays, unless there is a very good reason to segment workload. Distributing disks gives a better overall performance balance, reducing the risk of fragmenting (and consequently wasting) resources. I would also use the same methodology for other enterprise arrays too.

Bear in mind if you choose to use Enterprise Flash Drives (EFDs) that they can only be placed in the first storage bays either side of the controller bay and with a limit of 32 per quadrant. Mind you, if you can afford more than 32 drives then you've probably paid for your onsite EMC support already!!

Does XIV spell the end for the IBM/Netapp relationship?

There's been a lot of talk recently about the position in the market for Nextra, IBMs disk storage product, acquired with the purchase of XIV at the beginning of the year. I've been mulling it over and I have to say it creates a dichotomy for me.

On the one hand I see some of the technology benefits like the ability to write across all disks in a storage array so that no one disk is the hot spot for write I/O [didn't StorageTek effectively do this with the Iceberg system and in a RAID-5 configuration to boot?]. There's no doubting the performance benefits of distributing writes across as many spindles as possible, but at the expense of only providing RAID-1? I can only assume that either a new RAID paradigm from IBM is imminent or there's going to be some other magic which tells us why we don't need to write two full blocks of data.

That leads to the second part of my cogitation; what exactly is the business benefit of the Nextra? How will IBM pitch it against EMC/HDS/3par/Pillar and most relevant, Netapp?

Tony Pearson's blog entry from 2nd January, he states:

"However, this box was designed for unstructured content, like medical images, music, videos, Web pages, and other discrete files"

Does that mean Nextra's market positioning is in direct competition to Netapp? Surely not! If so, how will IBM determine which product (N-series or Nextra) should be pitched at a customer looking to provide infrastructure to support unstructured data? Netapp would win every time as it holds all the feature cards in its hand.

I'm sure I'm missing something that IBM have spotted with XIV. A little birdie tells me that there will be official product announcements imminently. Perhaps then there will be some light at the end of the tunnel.

Tiered Storage/Broadband Model

Comments on my last post reminded me of a conversation I had recently regarding the setting of specific performance criteria for storage tiers. Some companies attempt to set specific performance guarantees; for example setting tier 1 storage to a response time of 10ms or less, tier 2 to 20ms or less.

I think there are two problems with this approach. Firstly, within a shared infrastructure it is *impossible* to guarantee that all I/O operations will complete within a pre-determined performance range (like the 10ms example). It may be possible to say 99.9% of I/Os measured over a 5 minute period are guaranteed, but there will always be the odd spike which will make 100% an unrealistic target.

Secondly, I think most users have no idea what a response time of 10ms or 20ms has on their application when they are planning a new project and consequently they will take the best tier possible - unless there is a big difference in the cost.

Take the following analogy. Imagine you are looking for a new broadband Internet service. Your ISPs now offer from 512Kb/s up to 24Mb/s. Which service do you take? I think most people will take the fastest service possible - until they see the cost. Imagine the price structure was as follows;

• 24Mb/s - £100
• 8Mb/s - £20
• 2Mb/s - £10
• 512Kb/s - £2

In this scenario, I'd be happy with the 8Mb/s service, because in reality I have no idea whether I *need* 24Mb/s (but rather I just *want* it).

The same logic applies for tiering. Most users think they will need tier 1 and unless they have a price/cost disincentive to not use it, then tiering won't work.

How Many IOPS?

A question I get asked occasionally is; "How many IOPS can my RAID group sustain?" in relation to Enterprise class arrays.

Obviously the first question is to determine what the data profile is, however if it isn't known, then assume the I/O will be 100% random. If all the I/O is random, then each I/O request will require a seek (move the head to the right cylinder on the disk) and the disk to rotate to the start of the area to read (latency) which for 15K drives is 2ms. Taking the latest Seagate Cheetah 15K fibre channel drives, each drive has an identical seek time of 3.4ms for reads. This is a total time of 5.4ms, or 185 IOPS (1000/5.4). The same calculation for a Seagate SATA drive gives a worst case throughput of 104 IOPS, approximately half the capacity of the fibre channel drive.

For a RAID group of RAID-5 3+1 fibre channel drives, data will be spread across all 4 drives, so this RAID group has a potential worst case I/O throughput of 740 IOPS.

Clearly this is a "rule of thumb" as in practical terms, not every I/O will be completely random and incur the seek/latency penalties. Also, enterprise arrays have cache (the drives themselves have cache) and plenty of clever algorithms to mask the issues of the moving technology.

There are also plenty of other points of contention within the host->array stack which makes this whole subject more complicated, however, when comparing different drive speeds, calculating a worst case scenario gives a good indication of how differing drives will perform.

Incidentally, as I just mentioned, the latest Seagate 15K drives (146GB, 300GB and 460GB) all have the same performance characteristics, so tiering based on drive size isn't that useful. The only exception to this is when a high I/O throughput is required. With smaller drives, data has to be spread across more spindles, increasing the available bandwidth. That's why I think tiering should be done on drive speed not size...

The Right Way to do Vendor Comparisons

Good old Chuck Hollis has stirred up the vendor vitriol over the last week with two posts comparing the capacity efficiency of EMC's CX4, Netapp's FAS and HP's EVA products. See "Your Usable Capacity May Vary" and "Updates to Capacity Post".

Unsurprisingly, Chuck's conclusion is that EMC comes out on top (if it hadn't, would he still have posted - I think not). Now, not content with the statements Chuck made, HP have responded in kind. Check out the blog here (HP). There have also been plenty of comments on Chuck's posts, many of them non-too positive.

Whilst the opposing sides continue to score points off each other by highlighting the merits of their own technology, my mind drifts to the subject of exactly how vendor comparisons can be made. In some of my previous roles, I've had to help bring quotes for storage and SAN switches into line to make them as equivalent as possible in terms of their capacity. The trouble is, it isn't that simple.

Think of the difference between the switch vendors. Until recently, some vendors had full line speed blades in their hardware, however others followed the over-subscription model, sharing bandwidth between physical ports. If you're being charged by the port, then there's a clear difference in what you are getting for your money with these two technologies. My view was to work out the bandwidth per port as another comparison and to break down the cost of individual components, creating a more detailed cost model.

The same thing applies to arrays, whether enterprise or modular. Inevitably, most users don't follow the vendor best practices, choosing to use their own design (whether tested in their labs) or using a customised best practice model. There are also those who don't follow any model at all. :-)

Why do users do this? Easy - they all have different requirements and customise their hardware to match this.

Back to testing. We need some real world independent testing. So rather than vendors submitting their hardware to SPC tests in which they set the configuration, the independent testers need to set the hardware specification based on common sense configurations. Now you may say common sense isn't that common and this would allow "interpretations" of configurations but I disagree. I think common sense configurations would more likely get user approval and any vendor who believes their hardware is best would have no reason not to take part.

So, vendors out there - got any hardware you want to loan out?