DMX Configuration Options

I've been looking at DMX configuration options this week. Essentially the question is how best to lay out a DMX-3 or DMX-4 array with a tiered configuration. For me there are two options and it's pretty clear which I prefer. First a little background. The following diagram shows the way DMX drives are deployed within a fully configured array. The array is divided into "quadrants", splitting each drive bay into two. Back-end directors (BED) provide connectivity to the drives as represented by the colour scheme. There are up to 4 BED pairs available for a full configuration.

Option 1 - Dedicated Quadrants

One option is to dedicate quadrants to a workload or tier. For example tier 1 storage gets given quadrant 1. Theoretically this should provide this tier with uncontended back-end bandwidth as all the tier 1 storage will reside in the same location. What it doesn't do is let tier 1 storage utilise unused bandwidth on the other BEDs, which as the array scales, may prove to be a problem.

There's also the question of physical space. As the drives are loaded into the array, if only a small number of them are tier 1, then potentially cabinet space is wasted. Either that or the configuration has to be build in an unbalanced fashion, perhaps placing more lower tier storage to the right of the array, using the expansion BEDs.

The second diagram shows how an unbalanced array could look - tier 2 devices on the left and right are loaded at different quantities and so lead to an unbalanced layout.

Option 2 - Mixed Workload

In this option, disks are spread across the whole array, perhaps placing tier 1 disks first followed by tier 2 devices. In this way, the I/O load is spread across the whole configuration. As new disks are added, they are distributed throughout the array, keeping performance even. The risk with this configuration lies in whether tier 2 storage will affect tier 1, as the array becomes busy. This can be mitigated with Cache partitioning and LUN prioritisation options.

I prefer the second option when designing arrays, unless there is a very good reason to segment workload. Distributing disks gives a better overall performance balance, reducing the risk of fragmenting (and consequently wasting) resources. I would also use the same methodology for other enterprise arrays too.

Bear in mind if you choose to use Enterprise Flash Drives (EFDs) that they can only be placed in the first storage bays either side of the controller bay and with a limit of 32 per quadrant. Mind you, if you can afford more than 32 drives then you've probably paid for your onsite EMC support already!!

EMC Stretches DMX Both Ways

EMC made two DMX-4 announcements today. At the top end, the first details solid state drives which replace standard spinning HDDs. At the bottom end the second announces new 1TB SATA-II drives (and other miscellaneous stuff).

The solid state hard drive (or as Chuck Hollis describes it, the enterprise flash drive), isn't just a small 1-2GB device; this is a real replacement for 73GB or 146GB drives and will apparently slot right into a standard HDD bay. Assuming drive support is identical to standard HDDs, then it should be possible to create hypers from the drive(s) and present them to many separate hosts. I see this as significantly better than products such as Tera-RamSan. Firstly, the EMC device is more compact, second and most important, it integrates with the existing array, meaning better compatibility, support for existing functionality such as SRDF and TimeFinder and therefore more consistency.

Obviously a real issue will be cost versus standard HDDs, however I think there will be plenty of takers. Every large organisation has one or more application which needs more storage horsepower and the tradeoff between rewriting the application, purchasing more hardware or using SSD HDDs in DMX will make the latter option very appealing. Hopefully as time goes on, the cost of SSD drives will reduce significantly (think of how expensive SD cards used to be) and SSD technology will be more attractive to a wider audience.

Naturally the blog spin masters at EMC have been active; here's just a selection of the links I found:

• Chuck's Blog
• Confessions of an eBiz Junkie
• StorageZilla

Nothing yet from The Storage Anarchist though. Give it time...

The second announcement covered availability of 1TB for DMX-4. Fair play to EMC, they are offering storage technology across the spectrum of speeds and capacities, all in a single box. I still need to think through the implications of having even more choice in one array, not least of which is the impact on my design methodologies!

Oh and the second announcement re-announced thin provisioning for DMX-4. But everyone else is already doing that already. :-)

USP-V does SATA

So the rumours (here and here) are true. Here's the announcement to prove it. HDS are going to support 750GB SATA II drives in the USP.

This is an interesting position HDS are taking as thin provisioning will be able to take use of the enhanced drive capacity making the USP-V even more efficient than the DMX-4 with SATA drives.

However I do wonder whether HDS have been pushed into supporting SATA on the USP-V. I was under the impression that thin provisioning on external drives (the standard HDS line - use External SATA storage rather than configuring it within the USP itself) wasn't going to be available in the initial release. Perhaps HDS had to support SATA in order to get best usage out of the thin provisioning option and to answer customer complaints about using thin provisioning with expensive storage.

What I'd like to see next is how HDS will now position internal versus external storage. At what point do externally connected SATA drives become more cost effective than internal ones? This announcement seems to muddy the waters from that perspective.

I imagine we will get an announcement from Hu explaining how logical it is and how it is all part of the ongoing strategy....

Invista

There's been a few references to Invista over the last couple of weeks, notably from Barry discussing the "stealth announcement".



I commented on Barry's blog that I felt Invista had been a failure, due to the number of sales. I'm not quite sure why this is so, as I think that virtualisation in the fabric is utimately the right place for the technology. Virtualisation can be implemented at each point in the I/O path - the host, fabric and array (I'll exclude application virtualisation as most storage managers don't manage the application stack). We already see this today; hosts use LVMs to virtualise the LUNs they are presented; Invista virtualises in the fabric; SVC from IBM sits in that middle ground between the fabric and the array and HDS and others enable virtualisation at the array level.



But why do I think fabric is best? Well, host-based virtualisation is dependent on the O/S and LVM version. Issues of support will exist as the HBAs and host software will have supported levels to match the connected arrays. It becomes complex to match multiple O/S, vendor, driver, firmware and fabric levels across many hosts and even more complex when multiple arrays are presented to the same host. For this reason and for issues of manageability, host-based virtualisation is not a scalable option. As an example, migration from an existing to a new array would require work to be completed on every server to add, lay out and migrate data.



Array-based replication provides a convenient stop-gap in the marketplace today. Using HDS's USP as an example, storage can be virtualised through the USP, appearing just as internal storage within the array would. This provides a number of benefits. Firstly driver levels for the external storage are now irrelevant (only requiring USP support, regardless of the connected host), the USP can be used to improve/smooth performance to the external storage, the USP can be used for migration tasks from older hardware and external storage can be used to store lower tiers of data, such as backups or PIT copies.



Array-based replication does have drawbacks; all externalised storage becomes dependent on the virtualising array. This makes replacement potentially complex. To date, HDS have not provided tools to seamlessly migrate away from one USP to another (as far as I am aware). In addition, there's the problem of "all your eggs in one basket"; any issue with the array (e.g. physical intervention like fire, loss of power, microcode bug etc) could result in loss of access to all of your data. Consider the upgrade scenario of moving to a higher level of code; if all data was virtualised through one array, you would want to be darn sure that both the upgrade process and the new code are going to work seamlessly...



The final option is to use fabric-based virtualisation and at the moment this means Invista and SVC. SVC is an interesting one as it isn't an array and it isn't a fabric switch, but it does effectively provide switching capabilities. Although I think SVC is a good product, there are inevitably going to be some drawbacks, most notably those similar issues to array-based virtualisation (Barry/Tony, feel free to correct me if SVC has a non-disruptive replacement path).



Invista uses a "split path" architecture to implement virtualisation. This means SCSI read/write requests are handled directly by the fabric switch, which performs the necessary changes to the fibre channel headers in order to redirect I/O to the underlying target physical device. This is achieved by the switch creating virtual initiators (for the storage to connect to) and virtual targets for the host to be mapped to. Because the virtual devices are implemented within the fabric, if should be possible to make the virtual devices accessible from any other fabric connected switch. This poses the possibility of placing the virtualised storage anywhere within a storage environment and then using the fabric to replicate data (presumably removing the need for SRDF/TrueCopy).

Other SCSI commands which inquire on the status of LUNs are handled by the Invista controller "out of band" by an IP connection from the switch to the controller. Obviously this is a slower access path but not as important in performance terms as the actual read/write activity.

I found a copy of the release notes for Invista 2.0 on Powerlink. Probably about the most significant improvement was that of clustered controllers. Other than that, the 1.0->2.0 upgrade was disappointing.

So why isn't Invista selling? Well, I've never seen any EMC salespeople mention the product never mind pushing it. Perhaps customers just don't get the benefits or expect the technology to be too complex, causing issues of support and making DR an absolute minefield.

If EMC are serious about the product you'd have expected them to be shoving it at us all the time. Maybe Barry could do for Invista what he's been doing in his recent posts for DMX-4?

SATA in Enterprise Arrays

In a previous post on DMX-4 I discussed the use of SATA drives in enterprise arrays. A comment from our Storage Anarchist challenged my view of the resilience of SATA drives compared to FC.

Now unless I've been sleeping under a rock, the storage industry has over the last 5 years pummelled us with the warning that SATA are not enterprise arrays, the technology having derived from PC hardware. They were good for 8/5 rather than 24/7 work and not really suitable for large volumes of random access.

Has that message now changed? Were we fooled all along or is this a change of tack to suit the marketers?

What do you think? Are SATA drives (and I mean an *entire* array) in an enterprise array acceptable now?

EMC Power - link?

Thanks to Barry/Mark for their posts/comments on power usage. Call me stupid (and people do) but I can't find the EMC Power Calculator for download on Powerlink as 'zilla suggests (although I did find a single reference to it).

Can you post the link guys? Is it EMC internal only? If so, any chance on sharing it with us? If I can get a copy I'll do more detailed calculations on the components too.

DMX-4 Green or not?

After the recent EMC announcements on DMX-4, I promised I would look at the question of whether the new DMX-4 is really as green as it claims to be. I did some research and the results are quite interesting.



Firstly we need to set the boundaries. One of the hardest part of comparing hardware from different manufacturers is that they are intrinsically different (if they were too similar, the lawyers would be involved) and that makes it difficult to come up with a fair comparison. So, I've divided the comparisons into controller and disk array cabinets. Even this is difficult. The DMX has a central controller cabinet which contains only batteries, power supplies, interface boards and so on. The USP however uses half of the central controller for disks. The DMX has 240 drives per cabinet, however the USP has 256 disks per cabinet. This all needs to be taken into consideration when performing calculations.



Second, I want to explain my sources. I've tried to avoid the marketing figures for two reasons; firstly they usually refer to a fully configured system and secondly they don't provide enough detail in order to break down power usage by cabinet and by component. This level of detail is necessary for a more exact comparison. So, for the USP and USP-V, I'm using HDS's own power calculation spreadsheet. This is quite detailed, and allows each component in a configuration to be specified in the power calculation. For EMC, I'm using the DMX-3 Physical Planning Guide. I can't find a DMX-4 Planning Guide yet, however the figures on the EMC website for DMX-4 are almost identical to those for DMX-3 and it's as close as I can get.



DMX-3/4



The DMX figures are quite simple; the controller cabinet (fully loaded) takes 6.4KVA and a disk cabinet 6.1KVA. A fully configured controller cabinet has 24 controller slots, up to 8 global memory directors and 16 back and front-end director (FED) cards. A typical configuration would have eight 8-port FED cards and 8 BED cards connecting to all 4 disk quadrants. EMC quote the disk cabinet figures based on 10K drives. Looking at Seagate's website and standard 10K 300GB FC drives, each requires 18W of power in "normal" operation, so 240 drives requires 4.32KVA. The difference between this figure and the EMC value will cover when drives are being driven harder and the power supplies and other components which need powering within a disk cabinet. We can therefore work on an assumption of 25.4W per drive on average.

Now the figures for the controller cabinet are interesting. Remember EMC have no drives in the controller cabinet so all the power is for controllers, charging batteries and cache. So all that 6.4KVA is dedicated to keeping the box running.

USP

The HDS power calculator spreadsheet is quite detailed. It allows specific details of cache, BEDS, FEDs and a mix of 73/144/300GB array groups. A full USP1100 configuration has 1152 drives, 6 FEDs, 4 BEDs and 256GB of cache. This full configuration draws 38.93KVA (slightly more than the quoted figure on the HDS website. Dropping off 64 array groups (an array cabinet) reduces the power requirement to 31.50 KVA or 7.43KVA for the whole cabinet. This means the controller cabinet draws 9.21KVA and in fact the spreadsheet shows that a full configuration minus disks draws 5.4KVA. The controller cabinet has up to 128 drives in it, which should translate to about 3.7KVA; this is consistent with the 9.21KVA drawn by a full controller cabinet. The 7.43KVA in a cabinet translates to 29W per drive, making the HDS "per drive" cost more expensive.

This is a lot of data, probably not well presented but it shows a number of things;

1. There's an inescapable power draw per drive which can't be avoided; this equates to about 20W per drive.
2. The controller frame needs about 6KVA and this varies only slightly depending on the number of controllers and cache.
3. The HDS controller is slightly more efficient than the EMC.
4. The HDS disk array is slightly less efficient than the EMC.

Neither vendor can really claim their product to be "green". EMC are playing the green card by using their higher density drives. There's no doubting that this does compute to a better capacity to power ratio, however these green power savings come at a cost; SATA drives are not fibre channel and not designed for 24/7 workloads. Whilst these drives provide increased capacity, they don't provide the same level of performance and DMX systems are priced at a premium so you want to get the best bang for your buck. However, if EMC were to price a SATA-based DMX competitively, then the model is compelling, but surely that would take business away from Clariion. What's more likely to happen is customers choosing to put some SATA drives into an array and therefore see only modest incremental power savings.

So what's the future? Well, 2.5" drives currently offer up to 146GB capacity at 10K and only half the power demands, which also translates into cooling savings. Is anyone using these in building arrays? Hybrid drives with more cache should allow drives to be spun down periodically, also saving power. Either way, these sorts of features shoudn't come at the cost of the levels of performance and availability we see today.

One final note of interest...HDS are quoting figures for the USP-V. These show a 10% saving over the standard USP, despite the performance improvements...

DMX-4

I've had a quick look over the specifications of the new DMX-4 compared to the DMX-3. There aren't really a lot of changes. The backend director connectivity has been upped to 4Gb/s and presumably that's where the 30% throughput improvement comes from (with some Enginuity code changes too I guess).

There are a number of references to energy efficiency, however the "old" and "new" DMX cooling figures are the same and power figures are almost identical. The improved energy efficiency I think is being touted due to the availability of 750GB SATA drives for DMX (not now but later) but in reality that's not going to be a significant saving unless you're filling your entire array with SATA drives. One statement I want to validate is the following:

"Symmetrix DMX-4 is the most energy efficient enterprise storage array in the world, using up to 70 percent less power than competitive offerings."

There are some security enhancements - but there would have to be in order to justify the RSA purchase....

On the positive side, having the option of SATA drives is a good thing - I'd use them for Timefinder copies or dump areas. I wouldn't fill an array with them though.

Perhaps the most surprising announcement is (in green for extra emphasis):

In addition, EMC plans to introduce thin provisioning capabilities for Symmetrix DMX in the first quarter of 2008, enabling customers to further improve storage utilization and simplify storage allocation while continuing to improve energy efficiency.

Whoa there, I thought from all the recent posts (especially this) that Virtualisation/Thin Provisioning was something to be used with care. It will be interesting to see how EMC blogkets this one...

Bring it on!!

I'm glad to see EMC have upped the ante with their series of announcements today. I've not had time to digest them all but I did read Storagezilla's refreshing post summarising a lot of the announcements. Once I've read things in a bit more detail then I'll comment however, I like the fact EMC have moved the DMX on in performance and eventually capacity. I'm not sure though whether this should be DMX-4 or DMX-3.5 as I don't see a lot of new features. Hopefully that's just me not reading things fully.

I think it's about time for a 3-way product comparison. I just need to find the time...!