Enterprise Computing: Using USP for Migrations

Thanks to Hu Yoshida for the reference to a previous post of mine which mentioned using virtualisation (USP, SVC, take your pick) for performing data migrations. As Hu rightly points out, the USP, USP-V, NSC55 and USP-VM can all be used to virtualise other arrays and migrate data into the USP as part of a new deployment. However nothing is ever as straightforward as it seems. This post will discuss the considerations in using a USP to virtualise and migrate data into a USP array from external sources.

Background

The Universal Volume Manager (UVM) feature on the USP enables LUN virtualisation. To access external storage, storage ports on the USP are configured as "External" and connected either directly or through a fabric to the external storage. See the first diagram as an example of how this works.

As far as the external storage is concerned, the USP is a Windows host and the settings on the array should match this. Within Storage Navigator, each externally presented LUN appears as a RAID group. This RAID group can then be presented as a single LUN or if required, carved up into multiple individual LUNs.

The ability to subdivide external storage isn't often mentioned by HDS; it's usually assumed that external storage will be passed through the USP on a 1:1 basis and if the external storage is to be detached in the future then this is essential. However if a configuration is being built from scratch then external storage could be presented as larger LUNs and subdivided within the USP. This is highlighted in the second diagram.

At this point, external storage is being passed through the USP but the data still resides on the external array. The next step is to move the data onto LUNs within the USP itself. Here's the tricky part. The target LUNs in the USP need to be exactly the same size as the source LUNs on the external array. What's more, they need to be the same size as the way the USP views them - which is *not* necessarily the same as the size on the external storage itself. This LUN size issue occurs because of the way the USP represents storage in units of tracks. From experience, the best way to solve this problem was to actually present the LUN to the USP and see what size the LUN appears as. When I first used UVM, HDS were unable to provide a definitive method to calculate the size a LUN would appear within Storage Navigator.

The benefits of virtualisation for migration can fall down at this point. If the source array is particularly badly laid out, the target array will retain the multiple LUN sizes. In addition, a lot of planning needs to be performed to ensure the migration of the LUNs into the USP doesn't suffer from performance issues.

Data is migrated into the USP using Volume Migration, ShadowImage (or TSM). This clones the source LUN within the USP to a LUN on an internal RAID group. At this point, depending on the migration tool it may be necessary to stop the host to remap to the new LUNs. This completes the migration process. See the additional diagrams, which conceptualise migration with TSM.

Now, this example is simple; imagine the complexities if the source array is replicated. Replication has to be broken, potentially requiring an outage for the host. Replication needs to be re-established within the USP but data has to be fully replicated to the remote location before the host data can be confirmed as consistent for recovery. This process could take some time.

Summary

In summary, here are the points that must be considered when using USP virtualisation for migration:

1. Configuring the external array to the USP requires licensing Universal Volume Manager.
2. UVM is not free!
3. Storage ports on the USP have to be reserved for connecting to the external storage.
4. LUN sizes from the source array have to be retained.
5. LUN sizes aren't guaranteed to be exactly the same as the source array.
6. Once "externalised" LUNs are replicated into the USP using ShadowImage/TSM/VM.
7. A host outage may be required to re-zone and present the new LUNs to the host.
8. If the source array is replicated, this adds additional complication.

I'll be writing this blog up as a white paper on my consulting company's website at www.brookend.com. Once it's up, I'll post a link on the blog. If anyone needs help with this kind of migration, then please let me know!

Storage Predictions for 2009

It's the end of another year and of course time to the obligatory posts on predictions in the industry for the next 12 months. True to form, I've spent some time thinking and here are my top 5 ruminations for the coming year.

• EMC join SPC. EMC will finally have the epiphany we've all been dreaming of and embrace the world of generalised benchmarking. DMX will prove to be underpowered (due to the lack of SATA drives and the discovery that SSD is in fact the slowest disk technology) and be outperformed by Tony Asaro and his new Drobo appliance.
• HDS Win At Web Awards. HDS embrace new media in a game changing way and Hu wins first prize at the Weblog Awards. Barry Burke is so impressed, he defects to HDS from EMC, becoming www.thestoragedefeatist.com.
• XIV Conquers the World. IBM releases XIV-1000, by sending Moshe Yanai back in time from 2032, the time when Atmos became self-aware and took over storage for the entire world. EMC counter, sending StorageZilla (now aged 32) back to defeat Moshe in a monumental battle pitting SATA against SSD.
• JWT joins Star Trek. Jon Toigo bows out of storage to follow a career in acting. His first role as the father of Willam Riker in Star Trek XI is critically acclaimed as a work of genius.
• Convoy II: SWCSA is Released. The life of Marc Farley is brought to the big screen as he attempts to outwit the authorities and drive from the west to east coast using nothing but his blogging skills. Farley is portrayed on screen by Kris Kristofferson, reprising his role from the original cult movie.

Check back next year to see how many of these predictions did in fact come true.

Betamax

In case you don't always go back and look at comments (and let's face it, it's not easy to track them), then have a look at Tony's comment from my post yesterday. It's good to see a bit of banter going on and that's what HDS could do with.

HDS have hardly developed a good blogging prowess, it's more a case of "oh well, better do that" than taking a lead in new media.

Look at EMC.

There's geeky leaky Storagezilla with his uber-technical posts and sneaky advance notice of EMC technology.

Next The Storage Anarchist with his ascerbic character and product assassinations of the competition.

And who can forget Chuck, EMC's futurologist with his head in the cloud.

There's others of course, filling in the technical detail. Apologies if I haven't mentioned you by name. EMC have certainly grabbed Web 2.0 and given it a good shake.

Sadly HDS don't seem to have the same enthusiasm for marketing to their customers. Blog posts are few and far between from the small slew of bloggers they have to date. Content is shallow and that's a big problem.

We *all* know USP is faster than DMX. Anyone who's had the products in their lab know exactly what I'm talking about. Unfortunately unless HDS make a song and dance about it, they're going to be the Betamax of the Enterprise storage world.

Tony, keep the posts coming! Give is some real substance to beat up the competition with!!

2 Days, 2 Bod Posts

For the second time in two days I find myself drawn to comment on a Storagebod related post.

The subject today is Tony Asaro's rant on one of StorageBod's recent posts denegrating virtualisation.

Now let's get things clear - I like HDS's implementation of virtualisation. I've deployed it, I'd recommend it again, but some of Tony's comments are way off base.

"The cost per GB for your DMX FATA and SATA drives is much higher than using other tiered storage solutions." - yes, but UVM ain't free - there's a licence charge. When you virtualise an array, you're not paying for just JBOD, you're paying for extra stuff like the controllers. Also on the USP array you have to reserve out ports for virtualisation; if you connect the storage together through a fabric then you'll be taking up fabric ports too. The point is, the cost of HDS virtualisation means there's a break even point in the TBs of storage - from my experience, that was a big number.

"Storagebod does not want to have applications span multiple storage systems but other IT professionals are open to doing this. And storage virtualization is a powerful technology to enable this. That is the point of any virtualization technology - to overcome the physical limitations of IT infrastructure." - there are very good reasons for not spanning arrays with applications, like ensuring replicated storage is consistent, for instance. Whilst virtualisation allows a virtual array to grow in size to almost limitless amounts (247PB in USP V) it also means there's a concentration of risk; multiple components to fail, multiple places where data can be pinned in cache when things go wrong. In fact, placing data on lower availability arrays will increase risk.

"That may be true for Storagebod but that is not my experience in most data centers. We are shifting from transactional data being the “big space-hogs” to unstructured data consuming the lion’s share." - this may be true, but USP LUN-based virtualisation isn't going to help here. Overlaying file-level granularity data migration onto LUN-based arrays would require a particularly complicated scheme for ensuring data for migration was concentrated onto exactly the right LUNs so they could be moved to another tier. Anyway, why put unstructured data on expensive enterprise arrays?

I think we all expected Tony would have something better to talk about than technology HDS brought to the market 4+ years ago. We need to hear something new, something game-changing (oh and not me-too stuff like HDS putting SSDs into their arrays).

Tomorrow I *promise* I'll talk about something else.

Just Delete It Claus, Just Delete It

Claus Mikkelsen has woken up recently and started posting after a large break. Perhaps he's preparing for all those impending Christmas deliveries. Anyway, the crux of his post it to explain how he's moved from 2-4TB of home storage rather than take the time to sort out the mess of his home data. He then goes on to detail lots of clever technology which allows more data to be stored with less.

As I've posted many times before, we're just storing ourselves up a heap of trouble by not addressing the underlying issue here - delete the unwanted data.

We're creating storage landfills which will still need to be sorted out in the future. Like toxic waste in a rubbish dump, losing that critical file will eventually cost dearly.

Think of Claus' problem. Moving from 2-4TB doubles the amount of data that needs to be backed up (how do you back up 2TB of storage at home?), means any restores take longer, means spending more time searching for that file you know you had once, but can't remember what you called it - and if you use an online service for backup means you are paying unnecessarily for each month.

Take my advice, spend the time in developing (a) a good naming standard for your home files (b) a good standard for directories for storing your home files (c) delete the stuff you don't need. Immediately. Period.

SMI-S Is Dead

Take it from me, SMI-S is a thing of the past. If there's one thing the last few months have taught me it's how different each vendor's products really are. I've been working on a tool called SRA (see the link here) which will report on storage in a consistent manner. Let me tell you that isn't easy...

• EMC Symmetrix/DMX - Physical disks are carved into smaller segments called hypers. These are then recombined into LUNs which then might be recombined into composite devices (metas) and replicated, cloned or snapped. The hypers that make up a LUN can come from anywhere within an array and can be moved around at will by a tool designed to improve performance, completely ruining your original well-planned configuration. Combining hypers give you RAID, which wasn't RAID before and was something called mirrors but is now, and is even RAID-6! Devices have personalities which survive their presentation or removal from a port. A device can have multiple personalities at the same time. LUNs use a nice numbering system based on hex - but don't expect them to number nicely if you destroy and create devices. Bit settings (flags) are used to ensure host SCSI commands work correctly.
• HDS USP/HP XP - Physical disks are grouped into RAID groups from which LUNs are carved. Until recently you couldn't span RAID groups easily (unless you were combining some free space in each RAID group). Devices don't have a personality until they're presented to a host on a port, but they can have multiple personalities. HDS use a form of punishment known as CCI for anyone foolish enough to think they had made their arrays easy to manage. LUNs are numbered using a relic of the mainframe and yes, you can move things around to balance performance, but don't think you can do it unless there are spare LUNs (sorry LDEVs) around. Different host types are supported by a setting on a host group which lets you confuse the hell out of every one by telling them their LUN numbers are all the same but unique. Oh, and the storage the user sees doesn't actually have to be in the array itself.
• HP EVA - Phew! Physical disks are managed in groups (which it's recommended to only have one of, but you can have more if you really must) but they don't use RAID at the group level because that would be too easy. Instead disks are grouped into Redundancy Storage Sets, which reduce the *risk* of disk failures but don't protect directly against them. LUNs are created only when they need to be presented to a host and they don't have simple LUN numbers, but rather 32 digit UUIDs. RAID protection is done at the LUN level, making it more difficult to conceptualise than either of the previous two examples.
• Pillar Axiom - now we're getting really abstract. With Axiom, you can tier data on different levels of performance, but wait for it - they will be on the same drive, but utilising different parts of the same spindle! Argh! Enough!

Clearly every vendor wants to differentiate their product so you'll buy from them and not the competition. In some respects they *have* to differentiate otherwise all the vendors would spend their time in litigation with each other over patent copyright! (wait a minute, they already are). So SMI-S or any other standard is going to have a near impossible time creating a single reference point. Add to the mix the need to retain some competitive advantage (a bit like Microsoft holding back the really useful API calls in Windows) and to sell their own management tools and you can see why SMI-S will be at best a watered down generic interface.

So why bother. There's no benefit. Every vendor will give lip service to the standard and implement just what they can get away with.

The question is, what would replace it? There's no doubt something is needed. Most SRM tools are either overbloated, poorly implemented, expensive, or plainly don't work so some light touch software is a must.

I think the interim solution is to get vendors to conform to a standard API format, for example XML via an IP connection to the array. Then leave it to the vendor how to code up commands for querying or modifying the array. At least the access method would be consistent. We don't even see that today. All we need now is an acronym. How about Common Resource Access Protocol?

It's the little things that annoy

It annoys me when vendors release new products and don't update their global website correctly.

Like this link for HDS; http://www.hds.com/products/storage-systems/adaptable-modular-storage-2000-family/index.html?WT.ac=prodssams2000

Which should follow through to a page (link at the bottom) to AMS2000 Specifications. Except it doesn't - it only displays the older models. Doesn't a product release of the level of the AMS2000 warrant someone checking the website (i.e. the shop window) to ensure all the data is there.

Come on guys!!

Replacing the Virtualisation Component

There's no doubting that storage virtualisation will prove to be a key component of IT architecture in the future. The overall benefit is to abstract the physical storage layer from servers either in the fabric, or through the use of a dedicated appliance or even in the array itself.

Over time, storage resources can be upgraded and replaced, potentially without any impact to the host. In fact, products such as USP from HDS are sold on the virtues of their migration features.

However at some stage the virtualisation platform itself needs to be replaced. So how do we do that?

The essential concept of virtual storage is the presentation of a virtual world wide name (WWN). Each WWN then provides virtual LUNs to the host. The virtualisation appliance manages the redirection of I/O to the physical device, which also includes responding to SCSI LUN information queries (like the size of the LUN).

Ultimately, the host believes the virtual WWN is the physical device and any change to the underlying storage is achieved without affecting this configuration. If the virtualisation appliance must be replaced, then the virtual WWN could change and this means host changes, negating the benefit of deploying a virtual infrastructure.

As an example, HDS and HP allow USP/XP arrays to re-present externally connected storage as if it is part of the array itself. LUNs can be moved between either physical storage medium (internal or external) without impact to the host. However, the WWN used by the host to access the storage is a WWN directly associated with the USP/XP array (and in fact decoding the WWN shows it is based on the WWN serial number). If the USP is to be replaced, then some method of moving the data to another physical array is needed. At the same time, the host->WWN relationship has to change. This is not easy to achieve without (a) an outage (b) host reconfiguration (c) using the host as the data mover.

There isn't an easy solution to the issue of replacing the virtualisation tool. Stealing an idea from networking, it could be possible to provide a DNS style reference for the WWN with a "name server" to look up the actual physical WWN from the "DNS WWN". Unfortunately whilst this would be relatively easy to implement (a name server already exists in Fibre Channel) the major problem would be maintaining data integrity as a DNS WWN entry is changed and reads/writes start occurring from a new device. What we'd need is a universal synchronous replicator to ensure all I/Os written to an array are also written to any other planned target WWN, so as the WWN DNS entry is changed, it can't become live until a guaranteed synchronous mirror exists. I can't see many vendors agreeing to open up their replication technology to enable this; perhaps they could offer an API for "replication lite" which was used solely for migration purposes while the main replication product does the big replication features.

In the short term, we're going to have to accept that replacing the virtualisation engine is going to have some impact and just learn to work around it.

The Conserve IT Con

There's been a lot of blogosphere talk recently about the Wikibon "Conserve_IT" initiative and how California's Pacific Gas and Electric Company are taking the initiative and offering organisations rebates for demonstrable IT efficiencies. You can find comments here, here and here.

Unfortunately as with so many of these kinds of initiatives, the devil is in the detail. It is worth spending time reading the Conserve IT wiki page and Wikipedia's page on the California Energy Crisis of 2000/2001. To quote from Conserve IT: "Power in California is severely constrained". Why is that? The answer is pretty simple; California botched their deregulation of the energy markets and were screwed over by many companies, including Enron. In addition, significant underinvestment has placed restrictions on power distribution within the state (some of which is starting to be remedied).

It is pretty obvious that companies such as PG&E need other methods of controlling energy growth (Wikibon almost claim a virtue of the way per capita energy growth has remained flat in California compared to the rest of the US and Europe) and rebates for efficient energy use is their approach. Don't think of this as an altruisic green initiative - PG&E and others can't provide the electricity required.

So what about the Conserve IT initiative?

"As part of the qualification process, Wikibon has launched the Wikibon Energy Labs, an independent verification service that validates energy measurements of vendor products. "

Do we have so little faith in our vendors? When companies deploy equipment into their datacentres do they not measure the increased load on their PDUs? HDS and EMC provide tools for calculating the power demands of their products. These are detailed methods of analysis, down to the component level. Cisco provide significant information on power draw for their equipment (I've talked about this before). I'm sure other companies do to - feel free to mail me links or tools for calculating power demands for other manufacturers (I've asked HP about this for EVA but had no response so far).

The truth is, most eletricity generating companies are out to make money and giving organisations rebates for using less of their products makes no business sense. Where there are constraints on offering a service, then perhaps rebates or other incentives can do good, but this is not the norm and won't be replicated as a model across the rest of the US or the world.

As an organisation, you can do all the power calculations yourself. It isn't hard and it isn't hard to validate expected power draw against reality by getting one of your electricians to double check power consumption of a particular piece of hardware.

I'd urge all the vendors to make their power tools transparent and available. I would also urge them to incorporate KVA/cooling figures into their management tools.

In the meantime, read the links above and see how pointless Conserve IT is.

Off The Grid

I've been on holiday for the last week (sunning myself and the family in Cyprus). I had no Internet access - not even TV! Although I had no laptop (or Blackberry this time) I did take my iPod Touch, now configured with the mobile version of NewsGator. As I've mentioned previously, I have a 100+ RSS feeds (which I'll publish once I get around to it) on storage and others. My backlog was about 2500 entries, so I decided to challenge myself to get up to date and read as much as possible. Clearly I didn't read them all (there were plenty that could be skipped) but I read most and it provides for an interesting cross section...

EMC - blogs are run like a military machine; co-ordinating the news relating to new product releases and mercilessly hammering the competition. EMC have more storage bloggers than any other storage company and there are some good ones out there - one of my favourites is Information Playground by Steve Todd, where he discusses the design of Clariion.

Netapp - follows a close second to EMC with lots of bloggers and lots of competitor bashing. I particularly like Alex McDonald's postings.

IBM - doing a great job running the "resistance", fighting back against the continual onslaught of Barry (A Burke). Check out Barry (Whyte) and Tony Pearson. I'd like to see more from IBM though, especially their product developers working on DS arrays and XIV.

HDS - A jolly good bloke, but not really a player in the blogosphere. Only Hu contributes regularly, but doesn't engage in any serious debate.

Sun - quite literally on another planet with their storage strategy!

Dell - bought some toys, but doesn't know how to play with them. Unfortunately the older boy who could help them play with them has left...

Now there are more companies out there and I don't think I have any blog links from Brocade, 3Par, Compellent, Emulex, Qlogic, Pillar and others although I may be wrong (it is getting late). Any RSS link offerings gladly welcome - although I might not get around to reading them before my next holiday!

FCoE

Fibre Channel over Ethernet has been back on my radar recently, especially as it was touted again at Storage Networking World in Orlando last week. Unfortunately I wasn’t there and didn’t see for myself, although I was in Orlando the week before on vacation. I can imagine if I’d extended or moved the holiday to include SNW that I’d be none too popular with Mrs E and my sons.

Any hoo, I looked back over my blog and I first briefly mentioned FCoE back in April 2007, a whole 12 months ago. Now, we know 12 months is a long time in the storage world (in which time iSCSI will have claimed another 3000% market share, EMC will have purchased another 50,000 storage companies of various and dubious value, HDS will have released nothing and IBM will have developed 2 or 3 new technologies which won’t see the light of day until I’m dead and buried). I expect then that FCoE should have moved on somewhat and it appears it almost has. Products are being touted, for example, Emulex with the LP21000 CNA card (not an HBA card, please note the new acronym) and Cisco with their Nexus 5000 switch (plus others).

At this stage I don’t believe the FCoE protocol has been fully ratified as a standard. I have been spending some time wading reading through the FC-BB-5 project documentation on the T11 website, covering FCoE to understand exactly how the protocol works in more detail and how it can be compared to native fibre channel, iSCSI, iFCP and FCIP. In the words of Cilla, here’s a quick reminder on storage protocols in case you’d forgotten.

Fibre channel and the Fibre Channel Protocol (FCP) provide a lossless, packet based data transmission protocol for moving data between a host (initiator) and a storage device (target). FCP implements SCSI over fibre channel. To date, fibre channel has been implemented on dedicated hardware from vendors including Cisco and McDATA/Brocade. iSCSI uses TCP/IP to exchange data between a host and storage device using the SCSI protocol. It therefore includes the overhead of TCP/IP but provides for lossy and long distance connectivity. iFCP and FCIP are two implementations which encapsulate FCP in TCP/IP packets. FCIP extends an existing fibre channel SAN, whereas iFCP allows data to be routed between fibre channel SANs.

FCoE will sit alongside fibre channel and allow the transmission of FCP packets at the Ethernet layer, removing the need for TCP/IP (and effectively allowing TCP/IP and FCP packets to exist on the same Ethernet network).

So hurrah, we have another storage protocol available in our armoury and the storage vendors are telling us that this is good because we can converge our IP and storage networks into one and save a few hundred dollars per server on HBA cards and SAN ports. But is it all good? Years back, I looked at using IP over fibre channel as a way to remove network interface cards from servers. The aim was to remove the NICs used for backup and put that traffic across the SAN using IPFC. I never did it. Not because I couldn’t; I’m sure technically it would have worked, but rather because the idea scared the willies out of “the management” for two reasons (a) we had no idea of the impact of two traffic types going over the same physical network and (b) the Network Team would have “sent the boys round” to sort us out.

Will this be any different with FCoE? Will anyone really be 100% happy mixing traffic? Will the politics allow the Networks teams to own SAN traffic entirely? Let’s face it, in large environments I currently advocate the separation of host, tape and replication traffic to separate fibre channel fabrics. I can’t imagine reversing my position and going back to single consolidated networks.

So then, is FCoE going to be better in smaller environments where the consolidation is more practical? Well, if that’s the case, then surely that makes FCoE just another niche player to FC, just like iSCSI.

It’s early days yet. There are a million-and-one questions which need to be answered, not least of which will be how FCoE will interoperate with standard FC, how drivers will interact with the existing storage protocol stack on a server and how performance/throughput will be managed. Some of these issues have been answered, however this blog entry is already far too long and rambling to include a discussion on these points this time and I will save them for another time.

Multi-vendor Storage

Reading Chuck’s blog during my vacation, I stopped on his comment that multi-vendor environments are on the decline due to their complexity and the hassle of dealing with multiple vendors.

I have to say that firstly, I don’t believe this and secondly and companies with large storage environments would be mad not to consider a multi-vendor setup.

The reasons people have problems with multi-vendor environments are because they don’t spend time turning their storage into a commodity. EMC and HDS both recommend their own LUN sizes; each will sell you their management solutions; each will have their own support matrices.

But these things can and should be standardised. It is a simple task to define and migrate to consistent LUN sizes, regardless of vendor hardware. Software tools can be simplified; most people choose use command line or the basic configuration tools rather than the bloated EMC tools, so no problem there. In addition, scripting can be developed for failover and PIT/Snapshot management, making their use generic across vendors. Finally, driver/firmware/HBA/fabric standards can all be established to converge on a common set across all storage vendors.

Once storage has been established as a commodity, any new purchases can come from any of the vendors in your multi-vendor strategy.

Oh and one last thought; do you really believe HDS, IBM and EMC would give you the absolute best price if they know you can only use their product on most of your server farm environment? Competition within the storage market is a false premise; moving to another hardware platform to replace an existing one takes months (in some cases years). Vendors know that customers’ threats to move everything to another platform are only hollow unless you have a true multi-vendor strategy.

HDS and Hardware

I’m away on holiday at the moment (which is a blessed relief, looking at the Easter weather in the UK, no pun intended) however catching up with some emails I noticed the supposed post on Storagezilla’s site that IBM are to purchase Diligent.

I nearly worked for the UK arm of Diligent many years ago when they were getting started, but that’s another story. What struck me about the comments ‘zilla made with reference to IBM, Moshe and IBM’s acquisitive streak was the predicament HDS finds itself in.

HDS traditionally partners and doesn’t purchase companies however they don’t own their underlying hardware – the product is produced by Hitachi and resold by HDS and HP. The recent IBM acquisitions and EMC’s relentless purchase of anyone with a storage product mean HDS is falling behind the game. They currently have 2 product lines; enterprise hardware and modular hardware, neither of which they own. With IBM and EMC making inroads into acquiring more hardware and software tools to build entire solutions, how could HDS hope to compete going forward?

I’m not sure if HDS have a relationship with Hitachi which guarantees them access to storage products and a say in product design, however if they don’t then I can’t see how they will compete with the big boys. HDS need to look to acquire technology and start producing integrated solutions pretty quickly or they are going to get squeezed out of what looks to be an increasingly competitive and solutions’ based market.

Oh, one other little footnote; my 10-year old son bought a iPod Touch at the airport and fortunately for him I’d brought my laptop with me. Unfortunately, he didn’t understand why I couldn’t get online and download iTunes to configure his new toy while flying along at 35,000 feet…

The Case of Decimal v. Binary

We all know that disk drive manufacturers have been conning us for years with their definition of what constitutes a gigabyte. There are two schools of thought; the binary GB, which is 1024x1024x1024 or 1,073,741,824 bytes and the decimal GB which is quoted as 1000x1000x1000 or 1 billion bytes. The difference is significant (7.4%) and something that has annoyed me for years, mainly for having to explain to "management" why the 100TB of storage they bought stores less than 100TB of data (significantly less with some manufacturers).

Seagate have just lost a class action law suit (see here) which means they are forced to give a 5% rebate to customers plus some free backup software.

Unfortunately, those of us who have purchased storage subsystems from vendors will not be able to claim as the case refers to customers who bought stand-alone units:

"You are a member of the settlement class if, between March 22, 2001 and September 26, 2007, you purchased in the United States a new Seagate brand hard disc drive from an authorized Seagate retailer or distributor, separately as a Seagate product that was not pre-installed into and bundled with a personal computer or other electronic device."

So I guess EMC/HDS/IBM can breathe easy at this stage. It does beg the question though, when are our large array vendors going to quote the real figures for the usable storage available on an array? I have various calculators I use to help me work out the usable storage (before and after LUN creation and with varying RAID types) as I need to present real rather than throretical figures when sizing arrays. I think we need some more transparency here.

CU Free

Does anyone use CU Free? Here's the reason for my question.

I've just implemented a migration from a pair of 9980V and 9970Vs to a single USP in one site and a 9970V and 9980V remaining in the other site. All of the MCU->RCU relationships (4 of them) are being used between the USP and the 99XXV boxes.

If I implement another USP in the site with the 9900's I want to replicate from the existing USP. Will CU Free let me exceed the MCU->RCU restriction or is it just a helpful way of saving me having to enter all the paths for all the CU relationships I want to use? i.e. is the restriction of 4 still there regardless?

Storage Standards - Arrays

Storage Standards - Arrays

After a recent posting by Stephen I thought it would be good to discuss standards and the sort of standards I'd look to implement when designing an infrastructure from scratch. Obviously it isn't always possible to start from a clean slate and consequently many sites have what I describe as "partial implementations of multiple standards" where different people have put their stamp on an environment but not necessarily been able or wanted to go back and change the previous configuration. Still, let's assume we have a little flexibility and have a blank canvas, so to speak.

Objectives

I think it's worth having an idea of what you are trying to achieve by laying down standards in the first place. I think it boils down to a number of issues:

• Risk: A messy configuration poses more risk to data loss. If Storage Admins aren't sure whether disks are in use and to which servers then devices can easily be re-used inadvertently or perhaps be neglected from replication and cause issues in a DR scenario. For me, reducing risk is my main reason for adherence to a set of rigourous standards (sounds like an airline, where they always tell you as you board that their main priority is your safety, I always thought it was to make money from me).


• Efficiency: Storage is easy to overallocate and allocations today are very inefficient. We keep getting told this on the surveys that come out on a weekly basis. In almost no time at all we'll have more wasted data than atoms in the known universe or something like that. I think the figures get quoted as anywhere between 30-70% wastage. Either way, that's a lot of money being wasted on spinny stuff which doesn't need to be. With sensible standards, at least the storage can be accounted for and attributed to an owner (who can then be charged lots for the privilege), even if that owner isn't then efficient themselves. Efficiency at the host level is a whole separate argument to be discussed later.


• Manageability: When the storage environment is easy to understand, it is easier to administer. I once worked in a site which had four EMC 8830 arrays (two local, two in a remote site) all connected to each other. The local arrays had SRDF'd gatekeepers! (Warning: That was an EMC-based joke; apologies to all the hardended HDS fans out there who don't get that one). Needless to say, locating storage which could be reused and determining hosts which had or did not have replication was a time consuming job. Each allocation took 3-4 times more than necessary and half my time was spent attempting to clean the environment up.
  

So now we know what the reasons are, perhaps we can look at some of the standard to apply. Unfortunately most standards will tend to be very generic, as local restrictions will constrain exactly how arrays are laid out.

• Use RAID protection. This may seem a little obvious, however what I mean by this statement is that you should be reviewing your use of RAID and the RAID format used by tier. Lower tiered storage may be more suited to a higher RAID level but high performance data may need RAID 1/10. Either way, you need it and you should have a standard per tier.


• Distribute data across as many physical disks as possible. This also may seem like common sense but achieving it isn't always that simple. As disks have become larger, then more LUNs can be carved from each RAID group. This can have a negative impact on performance if only a small number of array groups are in use. There are also issues for upgrading arrays; a single 4-drive array group can provide nearly a TB of storage using 300GB drives (much more with new 500 and 750 drives as they become the norm), so the physical restrictions of the hardware become more apparent. It can be a real challenge to balance cost versus performance if your organisation insists on only adding small increments of storage as they are needed.


• Keep LUN sizes consistent. I don't like having a large number of differing LUN sizes. In fact I prefer to have just one if I can get away with it, however it isn't always that simple. LUN sizes should be defined in blocks (as most arrays use 512 byte blocks) and be the same even across storage from different vendors. This makes any type of migration easier to achieve. One tricky problem is choosing an appropriate LUN size. I think choosing an appropriate size or sizes borrows heavily from historical numbers but you should consider the size of your array groups (or physical disks) when planning LUN sizes. The more LUNs in a RAID group then the higher risk of contention at the physical level. Consider keeping any LUNs sizes as multiples of each other; as systems grow, LUNs can then be consolidated down to larger sizes.


• Use a dispersed method for numbering LUNs. Both EMC and HDS (can't speak for IBM) will recommend an inital configuration in which consecutively numbered LUNs are taken from different array groups one at a time and then repeated until all data is allocated. This means whichever group of sequential LUNs numbers you choose they will automatically be balanced across array groups. I have worked in sites that have used both methods and I can say that sequential numbering adds a significant amount of extra workto the allocation process.


• Don't dedicate array groups to specific LUN sizes. It may be nice to use one array group to create smaller LUNs for say a log device. This is *not* a good idea as you will end up creating an I/O bottleneck on those volumes. If you must have differing LUN sizes, create an even number from each array group.


• Distribute physical disks across all back-end directors. This may seem obvious but it is possible to unevenly balance disks across back-end directors, especially if storage is purchased in increments and different drive sizes are used. Keep things consistent, distribute the disks of each size evenly across the available directors.


• Distribute hosts across all front-end directors. There are two possible ways to distribute hosts, by capacity and by performance. You should decide which is more important for you and load balance accordingly. Continue to monitor both performance and allocations to ensure you are gettingt the best out of your FEPs.


• Dedicate replication links from each front-end director card. I like to ensure I have plenty of ports for replication (I've seen one issue recently, vendor name withheld, where lack of processor power in terms of FEPs for replication caused a performance problem, which was resolved by having more replication links (but not necessarily more cross site bandwidth), so I reserve at least one port/port pair on each FED card for this purpose.


• Dedicate specific FE ports for replicated and non-replicated storage. I prefer if possible to dedicate FEPs to production/replicated and non-replicated hosts in order to ensure that the same performance is available on both the primary and DR sites. If a DR host is also used as a UAT or test box, then place those disks on a separate FEP; that's what SAN is for!
  
• Dedicate array groups for replicated and non-replicated storage. This seems to contradict some of the other statements however, from a logistical point of view and if enough storage is available, it can be attractive to reserve out certain array groups for replicated and non-replicated storage, ensuring that the same arrays and LDEV/LUN numbers are used on replicated boxes.
  
• Allocate host storage from as many array groups as possible. When performing an allocation, try and spread the load across as many RAID groups as possible.
  
• Make use of your LDEV/LUN ranges. On HDS systems as an example, I like to reseve out CU ranges for tiers, so 00-0F for tier 1, 10-1F for tier 2 and 30-3F for command devices, external devices etc.

This is by no means an exhaustive list and I'm sure I will remember more. If anyone has any suggestions on what they do, I'd be interested to hear!

PSSST....Green Storage

HDS announced today a few amendments to the AMS/WMS range. The most interesting is the apparent ability to power down drives which are not in use a-la-Copan.

According to the press release above, the drives can be powered down by the user as necessary, which presents some interesting questions. Firstly, I guess this is going to be handled through a command device (which presumably is not powered down!) which will allow a specific RAID group to be chosen. Imagine choosing to power down a RAID group someone else is using! Presumably all RAID types can be supported with the power down mode.

One of the cardinal rules about hardware I learned years ago was never to power it off unless absolutely necessary; the power down/up sequence produces power fluctuations which can kill equipment. I'm always nervous about powering down hard drives. I've seen the Copan blurb on all the additional features they have in their product which ensures the minimum risk of drive loss. I'd like to see what HDS are adding to AMS/WMS to ensure power down doesn't cause data loss.

Finally, what happens on the host when an I/O request is issued for a powered down drive? Is the I/O simply failed? It would be good to see this explained as I would like to see how consistency is handled, especially in a RAID configuration.

However, any step forward which makes equipment run cooler is always good.

The announcement also indicated that 750GB SATA drives will be supported. More capacity, less cooling....

Problems Problems

This week I've been working on two interesting (ish) problems. Well, one more interesting than the other, one a case of the vendor needing to think about requirements more.

Firstly, Tuning Manager (my old software nemesis) strikes again. Within Tuning Manager it is possible to track performance for all LUNs in an array. The gotcha I found this week is that the list of monitored LUNs represents only those allocated to hosts and is a static list which must be refreshed each time an allocation is performed!

This is just lack of thought on behalf of the developers not to provide a "track everything" option so it isn't necessary to keep going into the product, selecting the agent, refreshing the LUN list and tagging them all over again. No wonder allocations can take so long and be fraught with mistakes when Storage Admins have to include in their process the requirement to manually update the tuning product. I'm still waiting for confirmation that there isn't a way to automatically report on all LUNs. If there isn't then a product enhancement will be required to meet what I want. In the meantime, I'll have to ensure things are updated manually. So if you configured Tuning Manager and the LUN list when you first installed an array, have a quick look to see if you're monitoring everything or not.

I'm sure some of you out there will point out, with good reason, why HTnM doesn't automatically scan all LUNs, but from my perspective, I'm never asked by senior management to monitor a performance issue *before* it has occurred, so I always prefer to have monitoring enabled for all devices and all subsystems if it doesn't have an adverse affect on performance.

Second was an issue with the way NTFS works. A number of filesystems on our SQL Server machines show high levels of fragmentation, despite there being plenty of freespace on the volumes in question. This fragmentation issue seems to occur even when a volume is cleared and files are reallocated from scratch.

A quick trawl around the web found me various assertions that NTFS deliberately leaves file clusters between files in order to provide an initial bit of expansion. I'm not sure this is true as I can't find a trusted source to indicate this is standard behaviour. In addition I wonder if it the way in which some products allocate files; for instance if a SQL backup starts to create a backup file it has no real idea how big the file will become. NTFS (I assume) will choose the largest block of freespace available and allocate the file there. If another process allocates a file almost immediately, then it will get allocated just after the first file (which may only be a few clusters in size at this stage). Then the first file gets extended and "leapfrogs" the second file, and so on, producing fragmentation in both files.

I'm not sure if this is what is happening, but if this is the way NTFS is working then it would explain the levels of fragmentation we see (some files have 200,000+ fragments in a 24GB file). In addition, I don't know for definite that the fragmentation is having a detremental impact on performance (these are SAN connected LUNs). Everything is still speculation. I guess I need to do more investigation...

USP-VM

Hitachi has announced (10th September) the availability of a new storage array, the USP-VM. At first glance this appears to be the USP-V equivalent of the NSC55 as it has very similar characteristics in terms of cache cards, FEPs etc. Unfortunately HDS have provided links to specification pages not all of which include the USP-VM references. Bit sloppy that.

I've previously deployed a number of NSC55's and within 6 months wondered whether they were the right decision. They weren't as scalable as I needed and there were a few features (such as BED ports and FED ports sharing the same interface card) which were a bit of a concern (imagine losing a FEP and having to take 1/2 of all your BE paths offline to fix the problem). I'm always reminded of the DMX1000/2000/3000 range when I think of the NSC model as these EMC arrays weren't expandable and of course a DMX1000 quickly filled up....

Hu describes the USP-VM as "Enterprise Modular" in his blog entry. This may be a bit generous as (a) I doubt the USP-VM will be priced as low as modular storage and (b) I don't think it will support the whole range of disks available in a modular array. I say "think" as the link to the capacity page for the USP products doesn't yet include the USP-VM.....

DMX-4 Green or not?

After the recent EMC announcements on DMX-4, I promised I would look at the question of whether the new DMX-4 is really as green as it claims to be. I did some research and the results are quite interesting.



Firstly we need to set the boundaries. One of the hardest part of comparing hardware from different manufacturers is that they are intrinsically different (if they were too similar, the lawyers would be involved) and that makes it difficult to come up with a fair comparison. So, I've divided the comparisons into controller and disk array cabinets. Even this is difficult. The DMX has a central controller cabinet which contains only batteries, power supplies, interface boards and so on. The USP however uses half of the central controller for disks. The DMX has 240 drives per cabinet, however the USP has 256 disks per cabinet. This all needs to be taken into consideration when performing calculations.



Second, I want to explain my sources. I've tried to avoid the marketing figures for two reasons; firstly they usually refer to a fully configured system and secondly they don't provide enough detail in order to break down power usage by cabinet and by component. This level of detail is necessary for a more exact comparison. So, for the USP and USP-V, I'm using HDS's own power calculation spreadsheet. This is quite detailed, and allows each component in a configuration to be specified in the power calculation. For EMC, I'm using the DMX-3 Physical Planning Guide. I can't find a DMX-4 Planning Guide yet, however the figures on the EMC website for DMX-4 are almost identical to those for DMX-3 and it's as close as I can get.



DMX-3/4



The DMX figures are quite simple; the controller cabinet (fully loaded) takes 6.4KVA and a disk cabinet 6.1KVA. A fully configured controller cabinet has 24 controller slots, up to 8 global memory directors and 16 back and front-end director (FED) cards. A typical configuration would have eight 8-port FED cards and 8 BED cards connecting to all 4 disk quadrants. EMC quote the disk cabinet figures based on 10K drives. Looking at Seagate's website and standard 10K 300GB FC drives, each requires 18W of power in "normal" operation, so 240 drives requires 4.32KVA. The difference between this figure and the EMC value will cover when drives are being driven harder and the power supplies and other components which need powering within a disk cabinet. We can therefore work on an assumption of 25.4W per drive on average.

Now the figures for the controller cabinet are interesting. Remember EMC have no drives in the controller cabinet so all the power is for controllers, charging batteries and cache. So all that 6.4KVA is dedicated to keeping the box running.

USP

The HDS power calculator spreadsheet is quite detailed. It allows specific details of cache, BEDS, FEDs and a mix of 73/144/300GB array groups. A full USP1100 configuration has 1152 drives, 6 FEDs, 4 BEDs and 256GB of cache. This full configuration draws 38.93KVA (slightly more than the quoted figure on the HDS website. Dropping off 64 array groups (an array cabinet) reduces the power requirement to 31.50 KVA or 7.43KVA for the whole cabinet. This means the controller cabinet draws 9.21KVA and in fact the spreadsheet shows that a full configuration minus disks draws 5.4KVA. The controller cabinet has up to 128 drives in it, which should translate to about 3.7KVA; this is consistent with the 9.21KVA drawn by a full controller cabinet. The 7.43KVA in a cabinet translates to 29W per drive, making the HDS "per drive" cost more expensive.

This is a lot of data, probably not well presented but it shows a number of things;

1. There's an inescapable power draw per drive which can't be avoided; this equates to about 20W per drive.
2. The controller frame needs about 6KVA and this varies only slightly depending on the number of controllers and cache.
3. The HDS controller is slightly more efficient than the EMC.
4. The HDS disk array is slightly less efficient than the EMC.

Neither vendor can really claim their product to be "green". EMC are playing the green card by using their higher density drives. There's no doubting that this does compute to a better capacity to power ratio, however these green power savings come at a cost; SATA drives are not fibre channel and not designed for 24/7 workloads. Whilst these drives provide increased capacity, they don't provide the same level of performance and DMX systems are priced at a premium so you want to get the best bang for your buck. However, if EMC were to price a SATA-based DMX competitively, then the model is compelling, but surely that would take business away from Clariion. What's more likely to happen is customers choosing to put some SATA drives into an array and therefore see only modest incremental power savings.

So what's the future? Well, 2.5" drives currently offer up to 146GB capacity at 10K and only half the power demands, which also translates into cooling savings. Is anyone using these in building arrays? Hybrid drives with more cache should allow drives to be spun down periodically, also saving power. Either way, these sorts of features shoudn't come at the cost of the levels of performance and availability we see today.

One final note of interest...HDS are quoting figures for the USP-V. These show a 10% saving over the standard USP, despite the performance improvements...