Performance Part V

Here's the last of the performance measurements for now.

Logical Disk Performance - monitoring of LDEVs. There are three main groups Tuning Manager can monitor; IOPS, throughput (transfer) and response time. The first two are specific to particular environments and the levels for those should be set to local array performance based on historical measurement over a couple of weeks. Normal "acceptable" throughput could be anything from 1-20MB/s or 100-1000 IOPS. It will be necessary to record average responses over time and use these to set preliminary alert figures. What will be more important is response time. I would expect reads and writes to 15K drives in a USP to perform at 5-10ms maximum (on average) and for 10K drives to perform up to 15ms maximum. Obviously synchronous write response will have a dependency on the latency of writing to the remote array and that overhead should be added to the above figures. Write responses will also be skewed by block size and number of IOPS

Reporting every bad LDEV I/O response could generate a serious number of alerts, especially if tens of thousands of IOPS are going through a busy array. It is sensible to set reporting alerts high and reduce them over time until alerts are generated. These can then be investigated (resolved as required) and the thresholds reduced further. LDEV monitoring can also benefit from using Damping. This option on an Alert Definition allows an alert to be generated only if a specific number of occurrences of an alert are received within a number of monitoring intervals. So, for instance, an LDEV alert could be created when 2 alert occurrences are received within 5 intervals. Personally I like the idea of Damping as I've seen plenty of host IOSTAT collections where a single bad I/O (or handful of bad I/Os) are highlighted as a problem when 1000s of good fast IOPS are going through the same host.

This is the last performance post for now. I'm going to do some work looking at the agent commands for Tuning Manager, which as has been pointed out here previously, can provide more granular data and alerting (actually I don't think I should have to run commands on an agent host, I think it should all be part of the server product itself, but that's another story).

Bring it on!!

I'm glad to see EMC have upped the ante with their series of announcements today. I've not had time to digest them all but I did read Storagezilla's refreshing post summarising a lot of the announcements. Once I've read things in a bit more detail then I'll comment however, I like the fact EMC have moved the DMX on in performance and eventually capacity. I'm not sure though whether this should be DMX-4 or DMX-3.5 as I don't see a lot of new features. Hopefully that's just me not reading things fully.

I think it's about time for a 3-way product comparison. I just need to find the time...!

Performance Part IV

Next on the performance hitlist is port tracking. This one is slightly more tricky to collect in Tuning Manager as HTnM uses absolute values for port throughput (Port IOPS and Port Transfer) in alerting, rather than relative values like % busy. This is a problem because the figures of both IOPS and throughput (KB/MB/s) will vary wildly depending on the traffic profile.

For instance, a high number of very small blocksize I/O will overwhelm the processor managing the port (resulting an seemingly low throughput) however a large blocksize could max out the storage port in terms of throughput (i.e. pushing a 2Gb/s FC link to its limit). As processor busy isn't available for alerting either, then I'd set a threshold limit on Port Transfer based on the capacity of the fibre channel interface. For example, with a 2Gb/s port, throughput should never be over 50% (if you want 100% redundancy in case of path loss) so I'd set figures of 40% of capacity for warning and 50% for critical in HTnM alerting. That's 40MB/s and 100MB/s respectively. These are high figures and sustained throughput at this level may require some load balancing or attention. If these levels generate no alerting then it may be possible to reduce them to start highlighting peaks.

Choosing a figure for IOPS is more problematic. I'd suggest picking an arbitrary level based on a few week's data from all ports. Set a limit based on historical data that's likely to trigger the occasional altert but will not create critical errors continually. Alerts can then be monitored and if there are alert trends, action can be taken. Hopefully it should be then possible to start to reduce alerting thresholds as more issues are solved.

Although TrueCopy initiators can't be monitored, RCU Targets can, so the same logic can be applied to throughput values.

On a totally unrelated subject, I have a recommendation to never de-seed chillis without gloves. I've just removed the seeds from my first crop of the year (including the wonderful looking purple tiger) and my entire face is glowing where I've wiped it. I have no idea how I'm now going to get my contact lenses out...

A couple of interesting comments

A couple of comments have been posted which look quite interesting so I think they're worthy of being re-mentioned in case you missed them.

First of all, at www.hds.com/webtech is a list of up coming webinars. The list looks a little "lite" however the first should be of interest (getting better alerting from Tuning Manager) bearing in mind my recent postings. Thanks to Rick for that one.

Second, a new site at www.lunmonkey.com. I haven't tried it, but it looks to be a site which will help autogenerate symconfigure commands based on a map of your storage configuration. I'd be interested to see if anyone's tried this.

While I'm on that topic, I think we need more Open Source type storage products. Clearly the vendors aren't going to produce products which provide real cross-system support (for obvious reasons). What we need is a Linux for storage. Despite all the talk, I don't think anyone is really doing it yet.

Performance Part III

Next under discussion for performance is array groups.

First the background (and as usual, apologies to those who already know all this). HDS enterprise arrays lay their disks out in array groups, either RAID-1/0, RAID-5 and RAID-6. Variable size LUNs are then carved out of the array groups for presentation to hosts. Obviously it makes sense to ensure that every host has their LUNs selected from as many array groups as possible. This means that large volumes of read and write data can be serviced quickly and on writes, the data written into cache can be destaged to disk quickly.

Pick any hour in the production day and use something like Tuning Manager and you'll likely see (especially on unbalanced systems) the standard exponential curve for IOPS or MB/s written. The 80/20 rule applies; around 20% of the array groups will be doing 80% of the I/O. Ideally it would be best to have workload balanced evenly across all array groups, however the effort of rebalancing the data probably doesn't justify the returns, unless of course you have some very busy array groups. Personally I'd look to ensure no single array group exceeds 50% active and I'd want 25-50% read hits (remember that you need to make sure you have some spare capacity to cater for disk sparing). Any array groups exceeding these metrics are candidates for data movement. I've recently used Cruise Control and I found it a disappointment - EMC's Optimiser swaps two LUNs using a third temporary LUN to manage the exchange. Cruise Control expects you to provide a free LUN as a target of migration. This may be difficult if a very quiet array group has been fully allocated. Therefore I tend to recommend manual exchanges, especially if hosts have a Logical Volume Manager product installed.

Balancing workload will mean checking array groups and moving any "hot" LUNs away from each other. This is where knowing your data becomes important and if possible knowing how the data is mapped on the host to make sure LUNs aren't just busy with transient data (for example, multiple LUNs that comprise a single concatentate volume on a host may be busy over time as more data is allocated to the volume). It is also equally possible that a single host may be able to overload one array group, so in that instance, moving the LUN will provide no benefit and the data layout on the host will need to be addressed.

It's possible to spend hours looking at array group balancing. The key is to make sure the effort is worth the result.

Performance - Part II

Next on the performance list - Sidefile. Sidefile is only relevant if you are using asynchronous replication. Cache is used to store write I/O requests (which have been committed locally) until they have been confirmed by a remote array in a TrueCopy pair. Both the local and remote arrays use sidefile cache to store replication recordsets which must be processed in seqence order in order to maintain consistency. The benefit of having Sidefile cache is that it minimises the effect of replication latency on TrueCopy write I/O to a local array. Sidefile usage rises and falls as write activity rises and falls, however if (for instance) replication is being managed across a shared IP network then other IP traffic could increase latency and affect the amount of sidefile cache used.

HDS recommends not letting Sidefile cache rise to higher than 10%, however there are a number of parameters which can be set to control sidefile usage. Probably most serious is Pending Update Data Rate (defaulting to 50%), which if breached causes primary array I/O delay and eventually TrueCopy pair suspension. There are also two other parameters, I/O Delay Start and I/O Delay Increase. Breaching these thresholds causes I/O delay, however it isn't clear how much of an impact this has on a host.

Now, I don't know where the 10% threshold comes from when sidefile controls by default start at 30%. Doing a simple calculation on an array with 20 active ports doing 5MB/s write IOPS each on average produces 100MB/s. With 48GB of cache, it takes less than 1 minute to reach 10% of sidefile cache, easily possible if replication goes across a congested network. I imagine that HDS are recommending that sidefile problems should be alerted as early as possible.

Hardware Replacement Lifecycle Update

Marc makes a good comment here about the use of virtualisation within the replacement lifecycle and indeed products such as USP, SVC and Invista can help in this regard. However at some stage even the virtualisation tools need replacing and the problem remains, although in a different place. I was hoping as part of the USP-V announcement HDS would indicate how they intend to help customers migrate from an existing USP which is virtualising storage, but alas it didn't happen.

Perhaps we need virtual WWNs which work in a similar way to DNS, where a small, easily replaceable and duplicatable device holds an alias of logical to physical WWNs. This appliance would be vendor independent and multiple devices would exist in the fabric as do today's DNS servers in IP. All allocations would be made once and for all via virtual WWN (vWWN) and if the storage is then virtualised that would be up to the vendor.

Of course, the difficulty with this model is how to determine which DNS entry points to the current valid copy of data... Oh well, it was only a 30 second idea....

HiCommand CLIEX

I've been using the HiCommand CLIEX (the extended CLI) today. It differs from the normal CLI in that it talks directly to the array via a command device and bypasses both Storage Navigator and Device Manager. HDS have needed this way of communicating with an array for a long time.

On the one hand, CLIEX is a good thing; the functions are more akin to those offered by EMC's Solutions Enabler in that I can perform direct configuration on the array without an intermediate product. Some of the things I've tried include extracting the actual configuration of an array (which can be obtained in XML format) and creating Host Storage Domains and assigning LDEVs. Although the commands aren't lightning quick (no pun intended), they are certainly quicker than the corresponding commands through Device Manager or Storage Navigator and obviously remove the need to install Device Manager in its entirety. As I'm running tests against a new array, I want to create lots of HSDs (one per FEP) and assign a separate LDEV to each. CLIEX saved me a lot of time (especially as I haven't got a Device Manager up yet).

On the negative side, the commands are powerful and "delete" functions give no warning or final check; so at your peril use the delete functions! I checked removing an LDEV while running (a lot) of I/O to a LUN and the command was bounced, however when the I/O was reduced (but the LUN still technically in use), I was able to pull the LDEV away with the obvious consequences. Also, these commands talk directly to the array and appear to bypass the configuration on the SVP. If you make a CLIEX change followed up by a Storage Navigator change without first having refreshed SN, then you simply overwrite the CLIEX changes (I know, I tried it).

One little bugette I found... The CLIEX command hdvmcliex is actually a batch file, so if you want to call the command from within a batch script you need to prefix with the "CALL" command.

I can see the CLIEX interface could be extremely useful. I intend to use it to mass configure a USP from scratch. In addition, I can also see how it could be useful to create dynamic failover scripts (more on this in a future post) without the need for Device Manager to be running. However HDS need to beef up the security around the product to prevent inadvertent allocation and deallocation gotchas. They also need to consider moving the locking mechanism (which allows a reserve to be taken on the SVP and prevent SN/CLIEX/Device Manager clashing configuration changes) to being a specific CLIEX command rather than relying on the Device Manager locking function.

One final thought... I'm not sure of any additional security to prevent any user with a command device from installing and running CLIEX and trashing an entire array. Unless you know otherwise?

Using Tuning Manager for something useful - Part I

Stephen commented about Tuning Manager and doing something useful with the data. I thought I would use this as an opportunity to highlight some of the things I look at on a regular basis (almost daily, including today in fact). Part I - Write Pending.

First a little bit of background; Write Pending is a measure of unwritten write I/Os which are in cache and haven't been destaged to disk. A USP/9980V (and most other arrays) store write I/O s in cache and acknowledge successful completion to the host immediately. The writes are then destaged to disk asynchronously some time later.

Caching writes (which I could be wrong, could have been invented by IBM and called Cache Fast Write or DASD Fast Write, I can't remember which was which) provides the host with a consistent response and helps to remove the mechanical delay of writing to disk. On average, most I/O will be a mixture of reads and writes and therefore an array can even out the write I/O to provide a more consistent I/O response. I see consistent I/O response as one of the major benefits or features of enterprise arrays.

Tuning Manager collects Write Pending (expressed as a percentage of cache being used for write I/O) in two ways; you can use Performance Reporter and collect real-time data from the agent/command device talking to the array. This can be plotted on a nice graph down to 10 second intervals. You can also collect historical WP data, which is retrieved from the agent on an hourly basis, providing an average and maximum figure. Hourly stats are collected for a maximum of 99 periods (just over 4 days) and can be aggregated into daily, weekly and monthly figures.

A rising and falling WP figure is not a problem and this is to be expected in the normal operation of an array, however WP can reach critical levels at which a USP will perform more aggressive destaging to disk; these figures are 30% and 40%. Once WP reaches 70%, inflow control takes over and starts to elongate response times to host write requests to limit the amount of data coming into the array. This obviously has a major impact on performance. I would look to put monitoring in place within Tuning Manager to alert when WP reaches these figures. Why would an array see high WP levels? Clearly, all the issues are write I/O related but they could include high levels of backup, database loads, file transfers and so on. It could also be an issue with data layout.

If an array can't cope with WP reaching 30%+ then some tuning may be required; consider distributing workload over a wider time period (all backups may have been set up to start at the same time, for instance) and investigate for write activity how that data is laid out; distributing the writes over more LUNs and array groups could be a remedy. It also may be possible that you have to choose faster array group disks for certain workloads and/or purchase additional cache.

I think Tuning Manager could do with a few improvements in terms of how Write Pending data is collected. Firstly, unless I'm running Performance Reporter, I have no idea within an hourly period when the max WP was reached and for how long it stayed at this level. Some kind of indicator would be useful - an hour is a long time to trawl through host data.

I'd also like to know, when the Max WP occurs, which LUNs had the most write I/O activity. LDEV activity is also summarised on the hour so it's not easy to see which LUNs within an hourly interval were those causing the WP issue; as an example, an LDEV could have been extremely active for 10 minute and caused the problem but, on average could have done much less I/O than another LDEV which was consistently busy (with a low I/O rate) for the whole 1 hour period. The averaging over 1 hour has a tendency to mask out peaks.

Finally, I'd like to be able to have triggers which can start more granular collection. So when a critical WP issue occurs, I want to collect 1 minute I/O stats for the next hour to an external file. I think it is possible to trigger a script from an alert, but I'm not clear on whether that trigger occurs at the point the max WP value is reached or whether it is triggered when the interval collection occurs on the hour (by which time I could be 59 minutes too late).

Next time, I'll cover Side-file usage for those who run TrueCopy async.

Responding to Comments

I'm not quite sure what the right way is to respond to posts; If I comment after them, then there's a chance that the responses might be missed. Anyway I will attempt to go back and check all those comments I've not responded to. First, here's a response to Cedric's question, on VMware;

What I was referring to was the way in which (a) VMware lays storage out over LUNs, (b) the way storage systems do failover at a LUN level.

For standard arrays which present LUNs, the lowest unit of granularity for TrueCopy/SRDF etc is the LUN. That's fine for systems that have lots of LUNs and then recombine them at the host level to create volumes. VMware works on a smaller number of larger LUNs (in my experience 50-200GB metas or LUSEs) and then divides this storage across multiple guests. So failover of a LUN could mean failover of a number of hosts. The tradeoff is therefore on how to lay out the storage in order to allow failover to work without affecting multiple hosts and also when additional storage is needed for a host how to present that additional storage.

The issue occurs because SAN storage is presented through the VMware hypervisor. iSCSI LUNs are presented directly to the host and so the original granularity can be retained and iSCSI LUNs can still be replicated. Therefore DR on iSCSI presented LUNs can easily be achieved.

Under the current versions of VMware (and please correct me if I'm wrong) it is possible to boot a guest from an iSCSI LUN and therefore theoretically possible to fail this over to another array. Personally, I'd not do that as I think it would present significant complications to achieve and I'd prefer to just present data LUNs for replication and keep O/S data local.

I hope this clarifies my thinking, feel free to post if you want me to clarify more.

Hardware Replacement Lifecycle

How do you replace your hardware? I've always thought a sound technology replacement lifecycle is essential. Brand new arrays are great. You can't wait to get them in and start using the hardware and before you know it, they are starting to fill up. If you didn't order a fully populated array (and if we are buying DMX-3, then that's almost a certainty) then you might extend it a few times with more disks, cache and ports.

But then the free maintenance period starts to loom. Nobody wants to think about the hassle of getting out that old array. Maintenance kicks in and before you know it, you're paying charges on hardware you want to replace.

In some respects it's a bit like using a credit card. You tell yourself you absolutely won't use it for credit. You'll pay the balance off each month, but then something comes along that you just must have (like that shiny new plasma screen) and you think; "I can pay that off over a couple of months, no sweat". But then you're hooked and paying interest which you never wanted to pay.

Although it's hard, it is essential to establish a hardware replacement strategy. There are lots of good reasons for shipping old hardware out:

• you will have to pay maintenance
• there may be compatibility issues
• parts may become difficult to locate (e.g. small disk sizes)
• the vendor may end support
• power/cooling/density pressures may increase
• no new features will be developed for non-strategic hardware

So here's my plan. Agree a replacement schedule and stick to it. Firstly you keep your risk exposure to a minimum, but also you'll have the vendors knocking at your door as they will know you are in the market for new equipment. Users of the array can also be notified of the lifecycle and be made aware ahead of time of the refresh cycle. Imagine a 4-year hardware lifetime. An example timeline could be;

1. Year 0 to the end of Year 3- GREEN: the array is a target for new allocations. The array is a target for maintenance upgrades on a regular basis (say 3-6 months, depending on release cycles)
2. Year 4, first half - AMBER: the array is still a target for adding storage to existing hosts on the array. No new storage/ports/cache will be added to the array. No new hosts will be added to the array. Maintenance on the standard cycle still continues.
3. Year 4 second half - RED: the array is a target for active migrations. Servers requiring new storage are forced to move off the array in order to receive their storage. Servers are scheduled to be moved to a new array within the 6 months. No new maintenance other than emergency fixes are to be applied.
4. Year 4 end - BLUE: the array is empty. Maintenance is terminated. The array is decommissioned and shipped out.

There's an obvious flaw in this policy - new hardware has to be on the floor during the last 6-12 months of the term to make it work. But, in any event, that has to happen anyway; either you get a new array in early, or you pay maintenance on the old hardware as you deploy a new array and migrate to it.

It may be that vendor hardware releases dictate that the 4 year term is shortened or increased to ensure the latest technology can be put on the floor. That could be an issue depending on the terms of maintenance for existing equipment, however getting that sorted is just a case of getting the right commercial arrangement.

What do you do?

Device Manager Update

More constructive Device Manager issues are posted here: Device Manager Issues.

HiCommand Export

Has anyone tried to use the HiCommand database transfer command (hcmdsdbtrans)? You can use it to export the contents of a database for later re-importation.

I've been looking at it as a way to get a Device Manager test environment, or to get a configuration between sites where there's no IP connectivity. The command produces a configuration file which as luck would have it, is a zip file. So just run the command and change the extension to ".zip" and you can look at the contents.

There are lots of files in CSV format within the export. These contain all of the expected details, such as the list of hosts, arrays, replication details and so on. There are a few interesting things of note, which I can't find anywhere else in Device Manager, either in the web interface or the CLI. For example, there's a file called licencekey.csv which lists all of the licences installed on the arrays in the database. Another file, host.csv (not surprisingly) lists all of the hosts defined in Device Manager. This seems to contain data I've not been able to locate in other parts of the product, specifically the O/S version for those hosts which have reported back via an agent. It's a shame this information is being collected and not being made available.

One file contains the SQL commands to rebuild the database. This is useful as I can see what data is being stored and how it interrelates.

I believe there is an official API for Device Manager but I've never been able to get a hold of the documentation. Does anyone know if it is a published document?

New Poll - VTL

As VTL has pictured recently, I thought I'd make that the subject of the next poll. It should be up on the right-hand side bar now.

USP Poll

I've closed the USP poll and the results are as follows;

• It rocks! What a great DMX basher! 30%
• It's OK, some of the new features seem useful. 15%
• It looks like just like the old USP but slightly faster 40%
• I don't see what all the fuss is about... 10%
• What's a USP-V? 5%

I think this sums up the comments that were flying at the time.

ECC 6.0 vs HiCommand 6.0?

EMC recently announced the release of ECC (ControlCenter) version 6.0. This version finally meets customer demands for VMware support (which despite the fact EMC own VMware is a long time coming) plus a whole host of other enhanced reporting.

I did a lot of work recently on virtualisation of storage tools using VMware and this version of ECC was planned not only to be the one which supported virtual guests (i.e. could understand how the VMware ESX Server had taken real storage and presented the guest O/S some of that) but also would be the version to run in a virtualised environment. Previous versions of ECC would work virtualised, however EMC were understandably concerned with performance and any negative impact bad performance of ECC on VMware would create. New new versions of ECC also support new features and functionality like other vendor's arrays (there were issues with the 5.x versions supporting NSC55 correctly). I think the enhanced reporting will prove extremely useful and overall ECC is now a great product.

Compare this to HDS' suite of products, HiCommand, currently at version 5.5 (check here for the compatibility matrix for Device Manager dated November 2006, not including the USP-V, check here for the same for Tuning Manager, quoting version 5.1 and other versions in the text, woefully out of date). A new version of this software will be needed in order to support USP-V and the new functionality. Will it be called 6.0?

Device Manager is still a second rate product at the current version. There are so many fundamental problems with the product. Here are just a few:

• Poor agent handling. Multiple products still require multiple agents. No O/S information retrieved from host, no indication whether the agent is running. Single fixed host information "push" from the client, no central scheduling for intra-day host collection. No agent push deployment from the server.
• No integrated (e.g. Active Directory) security.
• Only 2 storage reports, one of which can't understand dual pathed devices.
• Poor integration with other software, including CCI.
• Quirky, inflexible allocation method for assigning storage and for assigning hosts into groups.
• Lack of consistent feel for the products - the GUI has been made more consistent across the product range but agent installations are to differing locations (and change depending on things such as the version of HDLM installed)
• Poor documentation
• Unclear feature definitions (e.g. deletion of a host removes the "host security" - does that actually delete the storage, the HSD, or what?)

This is only a brief list as putting it all here in the blog would be more dull to read than my postings normally are. Also bear in mind Device Manager doesn't have to do that much - it doesn't even support other vendor's hardware. However I made this list to provide a comparison with how far ECC is ahead of HiCommand, bearing in mind they are at similar development levels.

I still think HDS don't get the whole software proposition. I know we all laugh at EMC calling themselves a software company, however at least they are heading in the right direction and excluding the integration issues from the 3 million other software companies they've purchased over the last few years they are turning out good SRM products (caveat, even ECC has issues).

As the storage offerings from HDS get more complicated, we need the tools to be trustworthy, usable, intuitive and unless they start getting there, no-one will want to use thin provisioning and virtualisation because the tools will just not deliver on enabling the productivity savings that those features can achieve.

HDS, pleeeeease tell me version 6.0 will be better? At the same time, you might want to update your website to ensure the compatibility matrix references the latest products.

CCI Update

A while back, I commented on how CCI was a pain in the proverbial **** because specifying the replicated LUNs required them to be presented onto a port/HSD. Snig kindly pointed out that although the LDEV needs to be presented, you can now just use the LDEV number rather than the complicated TID/LUN representation previously required.

I've been checking this out today as I've been doing some replication work and want to put together a central replication server which can build disk groups for all possible hosts on an array. Any hoo, I found that you can now specify the LUN number of the device on a port in the HORCM file, if you specify the port and the HSD number - like CL1-C-6 or similar, where 6 is the HSD number. Unfortunately you still need to specify the TID which has to be obtained from RAIDSCAN.

I gleaned this information from "Command Control Interface User and Reference Guide" version MK90RD011-18 (June 2006). In here I still can't see a reference to using LDEVs rather than LUN numbers, which to me, still makes the CCI specification a pile of unmentionables. In fact, I republish here this remarkable paragraph from section 4.20.1 of the said manual:

"The way what CCI has addition of argument for the host group to the raidscan command and
the configuration file will not be able to maintain the compatibility with conventional CLI.
Therefore, CCI adopts a way that supports in the form which specifies a host group in the
port strings as follows."

Now this has me confused - in fact those clever Japanese have already pre-empted me because the previous paragraph tells me how I was about to be confused:

"The USP/NSC and 9900V have the defined host group in the port and are able to allocate
Host LU every this host group. CCI does not use this host LU, and specifies by using absolute
LUN in the port. Therefore, a user can become confused because LUN of the CCI notation
does not correspond to LUN on the host view and Remote Console. Thus, CCI supports a way
of specifying a host group and LUN on the host view."

It would be useful if someone from HDS could confirm if I've read the manual correctly (or if there's a later release) as I still don't think replication based on LDEV is possible. The sooner it is brought in the better; the current method is just way too risky for my liking.

RTFM

Quite a while back, I posted on the Green Datacentre. Actually, the post was over a year ago and datacentre power and cooling issues have become a really hot (sic) issue.

However, I have to castigate myself severely for not Reading the Flaming Manual. In particular the power demands of the Cisco 9513 switch. Really the problem comes from working in an environment where power/cooling isn't wasn't an issue - just whack it in and get the electricians to provision whatever power you need. No need to actually validate the power requirements - just overprovision and everything will be fine. But recently, I wanted to power a 9513 up outside the datacentre to do pre-installation checks. The question was, could I power it up from a normal 13A socket? With 6000W of power required, surely not! Well, this is where RTFM comes in.

Thumbing through the 95xx Installation Guide I found that the chassis and fan trays take 318 watts, supervisor cards take 126 watts each and the biggest module, 48 port line-cards, 195 watts each. Therefore a fully populated 9513 would require 2715 watts. Based on UK power supplies of 240V, that's just over 11 amps. So, technically (although I wouldn't advise it and probably wouldn't try it) a 13A power supply would suffice.

So what's with the 6000W power supply? In fact, the power supplies in the 9513 are 6000W *capable* but don't have to demand or supply that level of power. Each takes two power feeds (from separate PDUs hopefully) and if only one feed is available, the power supply can only provide 2900W. If you think about it, what that means is that overall, 2 power supplies can provide ample power to run the switch without being under stress (and therefore provide more consistent, smooth power). Should a PDU fail, then each PSU can supply 2900W, again easily enough to run the whole switch. If *at the same time* one PSU should also fail, the remaining power supply can still run the whole switch. It may not be a desirable situation but it is possible. In terms of what power/cooling requirement should be catered for, it is actually a maximum of 2715 watts.

It just shows what reading the manual can do for you.

Certification Nightmare

One of the things I've always found a real issue in heterogeneous environments is that of certification. By certification I mean confirming that an installation is vendor supported. This is no mean feat in today's storage world. There are multiple layers - storage array, fabric, HBA, O/S, logical volume manager and multipathing software. There are also issues of virtualisation, product co-existence (e.g mixing switches from the same vendor but different OEM vendors), management tools, replication software and applications.



The stack of products and levels that must be considered is huge, especially if each layer has multiple products. It is also true that certain vendors don't help this situation (a) when new software is required to support a new array (for example) and it affects all other layers of the stack (b) competing vendors don't fully support their competitor's technology until well after the release date.



I've not yet seen a suitable solution for tracking certification levels. Firstly, even the vendors can't document their own products in a consistent fashion. Consider Emulex - querying the driver level on a Windows host returns a string which *contains* the version, but isn't in a consistent format. The returned string is different for Emulex on Solaris.



I think we've reached the point where we need a Certification Authority. Someone needs to take control and get all vendors to map their products to a consistent standard. Actually, I'd like to do it and I'm working on a markup language to help. It wouldn't be difficult. Probably the most troublesome part would be assigning an index code to each version/product/level that vendors produce - and getting them to use them....

By the way, in case anyone suggests SAN Advisor or Onaro SANScreen, I don't think either of these products are up to the mark. I've seen SAN Advisor demonstrated a number of times and it looked like a generation 1 product. SANScreen was interesting but not a certification tool as such - more like a semantic rather than syntactic storage product.

Using VTLs

The discussion on EMC's VTL has got StorageZilla and other EMC'ers a little excited. It stems from Barry's post on the DL6000, a Godzilla of a VTL as I've previously discussed before.

I think it is worthwhile reviewing the exact point of having a VTL in the first place.

Tape has issues and we all know it. They break, wear out, develop errors and most importantly get lost, compromising their content and potentially leading to embarrassment and large fines for the companies involved. Tape has the advantage that it is cheap, portable and (other than keeping it at the right temperature) costs a minimal amount to store.

Point 2; why do we do backups? They're done to recover from inadvertent data loss - logical corruption, user error, hardware failure and so on. Depending on company size, it is possible that tape may be used in the DR process but for large companies, they will probably either run a second site or use a third party DR company. It is likely that these companies will *not* rely on restoring from tape in the event of a complete site outage or disaster. BC/DR processes for these companies will be ultimately more complex than calling back the tapes from Iron Mountain.

Backups have also become a method of archiving data in the absence of a proper tool for archiving. Restoring data for archive purposes from backups depends on staff who have specific knowledge in order to know what to restore. The format of backup data precludes the ability to easily search and index content and mine it for competitive purposes. It might be *possible* but is it *practical* and can the effort justify the rewards gained from the data? Unlikely.

VTLs have developed to answer some of the issues relating to tape, namely the failure rates of backups and the time to access to get data back on restores. VTL's do not answer the issues of tape loss - let's face it, if you don't send the tapes offsite, you can't lose them. VTLs *have* to be located offsite from the main data otherwise you are at risk so why not just put the tape library offsite instead?

There are plenty of other techniques available to ensure you don't have to go running to backups for data recovery. Point-in-time copies, CDP, remote replication, snapshots, VSS all provide the ability to get data back and get it back quickly. Tape never did that - why should we expect a VTL to do so?

Enterprise-scale customers will have multiple sites and already write data to a remote location, perhaps across IP or DWDM/dark fibre with fibre channel connections. They will have tape libraries with automation in place and not need to ship tapes offsite. Their issues with tape will revolve around getting backup successes to 100%, eliminating those failures which occur from faulty media and ensuring that when restores take place, then tape drives are available to enable as many restores to occur as possible. VTLs enable that. But I don't believe VTLs should enable that at any price.

80-90% or more of data on tape is at rest. Tape data expires and the tapes are reused in cycles. Tape is effective because the 90% of inactive media can be put on a shelf and left alone. If you are realistic, then you'd say even automated tape libraries are not effective and they should only contain perhaps only the last 6 months or so of backup data. As tapes are cheap, multiple copies can be kept. If one tape is damaged or fails, it doesn't affect the content on the remaining tapes.

VTLs need to offer the same level of functionality:

• You want power consumption to be as low as possible.
• You want TCO to be as low as possible.
• You don't want component failure to affect the whole archive.
• You want granular access to your data to enable you to restore what you want when you want.

My point in the previous post was that the Copan product has been designed to address these requirements. It only powers up 25% of the disks because you never read and write from all your tapes at any one time - you couldn't do it because you never had the drives available to mount the tapes! Also, tape data is usually multiple copies of the same thing going back over months or years, so you would only restore the *last* copy in a DR situation, which is likely to be much less than 25% of the data on tape (or even a VTL). A point worthy of note here; the Copan system doesn't block access to data on drives that are powered down. It simply powers down another drive and powers up the one needed to provide access to the data so all the user sees is a delay in access.

The Copan system writes data on shelves which are treated as individual virtual libraries. Any one can be powered down individually to replace failed drives, so the whole system doesn't have to be taken down. That also helps to meet the issue of a hardware failure not affecting all the content. Data is not spread across the whole system so is not all at risk if a shelf did fail. The system also performs periodic data validation/drive scrubbing to ensure drives which are going to fail can be easily identified.

I'd like to end on one thought. If I was implementing ILM, I would implement a strategy which puts the most valuable data on Enterprise arrays. "Valuable" would mean both risk of losing and also risk of time to access; I'd want to access the data 24/7 and not lose it. As data value reduces then it goes on to less expensive technology where the tradeoff of cost versus availability is met. For instance, development data on modular storage. Finally, backup data would sit on the least expensive hardware platform. EMC are suggesting I keep that data on their most *expensive* platform!

Think of it this way - if you have a DMX3 already, why bother buying a DMX3 VTL solution? Just shove some 500GB drives into your existing DMX3 and backup straight to disk - it will be exactly the same in terms of availability and reliability but without the VTL licence cost!

Mine's Bigger than yours - Part Deux

Old Barry's at it again with the DMX pitch, this time for VTL.

Here's a quick comparison with Copan's Revolution 220TX system and the specification of the DL6100 from EMC's website.

Copan Revolution 220TX

• Single Cabinet
• 896 drives (500GB SATA)
• Power (max): 6368 watts
• Throughput: 5.2TB/hour
• Capacity: 448TB max
• Emulation: 56 libraries, 56 drives, 8192 virtual cartridges

EMC DL6000

• 7 cabinets (max configuration)
• 1440 drives (500GB LCFC)
• Power (max): 49100 watts
• Throughput: 6.4TB/hour
• Capacity: 615TB max
• Emulation: 256 libraries, 2048 drives, 128000 virtual cartridges

So, you'd need 1.5 Copan devices to match the EMC kit and yes, it doesn't scale as well in terms of virtual components, but there are some big issues here. For instance, EMC's device isn't green - the power demands are huge and not surprising, as the drives are all spinning all the time (Copan have only 25% max if theirs in use at any time). Floor density is not good in the DMX - why? Because the drives are in use all the time and it is an Enterprise array, so timely replacement of disk failures is important - but less so for a virtual tape system which can tolerate downtime.

So what would you go for? My money would be on a DMX for Enterprise work, which it is great at - and not as a "one size fits all" system, which it quite plainly is not.

Trusting TrueCopy

For those people who use TrueCopy on a daily basis, you'll know that the assignment of a TrueCopy pair is based on a source and target storage port, host storage domain and LUN. This means a LUN has to be assigned out to replicate it.

The part that has always worried me is the fact that the target LUN does not need to be in a read-only status and can be read-write.

Please, HDS to save my sweaty palms, change the requirement to make the target volume read-only before it can be a TrueCopy target....

Thin Provisioning - Cisco Style

There have been so many discussions on thin provisioning since the Hitachi USP-V announcement. When a major player takes on a specific technology, all of a sudden we realise that everyone else has already been doing it. Tony Asaro's post probably provides the best summary of those who do it today.

One vendor not appearing on the list is Cisco. Not surprising as they don't produce storage systems, however they do produce fibre channel switches which also implement thin provisioning.

I talked about the issue not that long ago here. Now I'm laying out ports for real and its not as clear as it seems. The low down is if you put ports in "dedicated" mode, you get the port speed reserved (or 4Gbps if you set to auto, regardless of the speed negotiated) and in "shared" mode you have a minimum requirement, 4Gbps ports need 0.8Gbps reserved for instance, and the figure reduces in proportion for 2 and 1Gbps ports. More details can be found here. This means not all combinations are possible and you get "Bandwidth Not Available" messages when you don't expect it. As this was confusing me, I've put together a port speed calculator, you can pick it up at Cisco Rate Calculator.

New Product Announcement,,,

There's another new product out this week from HP - the XP24000... It seems to have some similarities to another product launch, 224 ports, thin provisioning, partitioning... :-)

Poll Results

Thanks to everyone who voted in the poll on Optimiser/Cruise Control; seems like most people like the recommendations but don't trust it to make automatic changes.

I've got a new poll running for the next month - tell me what you think about USP-V. Now I'm *sure* there will be lots of opinion on this....

Port Indexes on Cisco Switches

I spent today building some Cisco MDS9513 switches. It's good to get into the nuts and bolts of technology once in a while. They are big beasts - 2nd generation technology and with 11 usable slots, the biggest available line cards (48 ports), 528 ports in a single chassis. However the build had one fly in the ointment. As part of the (inherited) design, the chassis included a "generation 1" eight ethernet port IP line card for implementing FCIP.

The issue revolves around a feature called port indexes, which are used to track the ports installed in a MDS chassis. Generation 1 chassis have a maximum of 252 port indexes, generation 2 technology supports up to 1020 port indexes. However, a generation 1 line card inserted into a generation 2 chassis dumbs down the whole switch to generation 1 port indexes. So, with 252 indexes, 32 taken up by the FCIP card, only 240 port indexes are left which directly translates to 240 ports - less than half the switch capacity! If line cards are installed that take the port count above 252 then these additional cards won't come online - they will initially power up then power down.

In this instance the solution will be to move IP services to another (smaller) switch and hopefully Cisco will bring out a generation 2 version of the IP blade soon. There is a bigger problem though, and that is for any customers looking to take 9513s and use them with the SSM module, which supports products such as EMC Invista and Kashya (now EMC RecoverPoint). As far as I am aware, there's no plans for a generation 2 SSM module any time soon, so using the SSM module in the 9513 chassis will create the same port restriction issues.

I don't see why Cisco couldn't simply produce a gen 2 version of the old gen 1 blades which did nothing more than re-jig the port indexing. Come to think of it, surely they could patch the firmware on the gen1 line-cards to fix the problem. Obviously it is not that simple, or perhaps not that many people are using Invista and RecoverPoint to make it worthwhile.

USP-V...and another thing

I hate it when I write a post then think of other things afterwards. One more USP-V thought (I'm sure there will be more). One of the drawbacks of the current hardware with virtualisation is the effort to remove/upgrade the USP. Was there anything in the announcements on Monday to cater for this? I was hoping HDS would announce USP "clustering". Although they don't think it is necessary from a resiliency perspective, it certainly is if you want to upgrade and haven't done a 1 for 1 passthrough on LUNs (i.e. presented "big" LUNs to the array and then carved them up in the USP).

So HDS, did you do it?

Oh, and another one...for customers who've just purchased a standard USP, will there be a field upgrade to USP-V?

USP-V - bit of a let down?

It seems from the posts seen so far on the blogosphere that the USP release is causing a bit of a stir (10 points for stating the obvious I think). So, here’s my take on the announcements so far.

First of all, it’s called USP-V – presumably because of the “Massive 500-Percent Increase in Virtualized Storage Port Performance for External Storage”. I'm not sure what that means - possibly more on that later.

As previously pointed out, the USP-V doesn’t increase the number of disks it supports. It stays at 1152 and disappointingly the largest drive size is still 300GB and only 146GB for 15K drives. I assume HDS intends to suggest that customers should be using virtualisation to connect to lower cost, higher capacity storage. That’s a laudible suggestion and only works if the Universal Volume Manager licence is attractive to make it work. In my experience this is an expensive feature and unless you’re virtualising a shed-load of storage, then it probably isn’t cost effective.

There have been some capacity increases; the number of logical LUNs increases 4-fold. I think this has been needed for some time, especially if using virtualisation. 332TB with 16384 virtual LUNs meant an average of 20GB per LUN, obviously now it is only 4GB. Incidentally, the HDS website originally showed the wrong internal capacity here: http://www.hds.com/products/storage-systems/capacity.html, showing the USP-V figures the same as the base USP100. It’s now been corrected!

Front-end ports and back-end directors have been increased. For fibre-channel the increase is from 192 to 224 ports (presumably 12 to 14 boards) and back-end directors increase from a maximum of 4 to 8. I’m not sure why this is if the number of supportable drives hasn’t been increased (do HDS think 4 was insufficient or will be see a USP-V MKII with support for more drives?). Although these are theoretical maxima, the figures need to be taken with a pinch of salt. For example, the front-end ports are 16-port cards in the USP and there are 6 slots. This provides 96 ports, the next 96 are provided by stealing back-end directors (this is similar to DMX-3 – 64 ports maximum which can be increased to 80 by removing disk director cards). Surprisingly, throughput hasn’t been increased. Control bandwidth has, but not cache bandwidth. Does the control bandwidth increase provide the 500% increase in virtualisation throughput for external storage?

What about the “good stuff”? So, far, all I can see is Dynamic (thin) Provisioning and some enhancements to virtual partitions. The thin provisioning claims to create virtual LUNs and spread data across a wide number of array groups. I suspect this is simply an extension of the existing copy-on-write technology, which if it is, makes it hardly revolutionary.

I’d say the USP-V is an incremental change to the existing USP and not quite worthy of the fanfare and secrecy of the last 6 months. I’d like to see some more technical detail (HDS feel free to forward me whatever you like to contradict my opinion).

One other thought. I don’t think the DMX-3 was any less or more radical when it was released....

100 Up

I've reached 100! No, not 100 years old, but my 100th post. I've surprised myself that I've managed to keep going this long.

So a bit of humour - I'll save the USP-V discussions until later - who remembers Storage Navigator for the 9900 series?

Well, the left-hand toolbar has various icons on it. One of them I've reproduced here;

it looks like a fried egg and a spanner. But what is it meant to represent (btw, it's the icon for LUSE/VLL).

Answers on a postcard please....

Mine's bigger than yours - do we care?

Our resident storage anarchist has been vigorously defending DMX - here. It's all in response to discussions with Hu regarding whether USP is better than DMX. Or, should I say DMX-3 and Broadway (whoops, I mentioned the unmentionable, you'll have to shoot me now).

I have to say I enjoyed the technical detail of the exchanges and I hope there will be a lot more to come. Any insight into how to make what are very expensive storage subsystems work more effectively has to be a good thing.

But here's the rub. Do we care about how much faster DMX-3 is over USP? I doubt the differences are more than incremental and as I've both installed, configured and provisioned storage on 9980V/USP/8730/8830/DMX/DMX2/DMX3, I think I've enough practical experience to qualify it. (By the way, I loved StorArch's comment about how flexible BIN file changes are now. Well, they may be, but in reality I've found EMC cumbersome to release configuration changes).

Finally I'll get round to the point of this post; most large enterprise subsystems are of the same order of magnitude of performance. However I've yet to see any deployment where performance management is executed to such a degree that, hand on heart, the storage admins there can claim they sequeeze 100% efficient throughput. I'd make an estimate that things probably run 80% efficient, with the major bottlenecks being port layout, backend array layout and host configuration.

So the theoretical bantering on who is more performant than the other is moot; now, EMC, HDS or IBM, come up with a *self tuning* array then you've got a winner...

Simulator Update

I managed to get a copy of the Celerra Simulator last week and I've just managed to get it installed. Although it is simple, it is quite specific on requirements - it runs under VMware ACE as a Linux program. It needs an Intel processor and can't run on a machine with VMware already installed. Fortunately my test server fits the bill (once I uninstalled VMware Server). Once up, you administer through a browser.

At this stage that's as far as I've got - however it looks good. More soon.

Port Oversubscription

Following on from snig’s post, I promised a blog on FC switch oversubscription. It’s been on my list for some time and I have discussed it before, however it has also a subject I’ve discussed with clients from a financial perspective and here’s why; most people look at the cost of a fibre channel switch on a per port basis, regardless of the underlying feature/functionality of that port.

Not that long ago, switches from companies such as McDATA (remember them? :-) ) provided full non-blocking architecture. That is, they allowed full 2Gb/s for any and all ports, point to point. As we moved to 4Gb/s, it was clear that Cisco, McDATA and Brocade couldn’t manage (or didn’t want) to deliver full port speed as the port density of blades increased. I suspect there were be issues with ASIC cost and fitting the hardware onto blades and cooling it (although Brocade have just about managed it).

For example, on Generation 2 Cisco 9513 switches, the bandwidth per port module is an aggregate 48Gb/s. This is regardless of the port count (12, 24 or 48), so, although a 48 port blade can (theoretically) have all ports set to 4Gb/s, the ports on average only have 1Gb/s of bandwidth.

However the configuration is more complex; ports are grouped into port groups, 4 groups per blade of 12Gb/s each, putting even more restriction on the ability to use available bandwidth across all ports. Ports can be dedicated or use shared bandwidth within a port group. In a port group of 12 ports, set three to dedicated bandwidth of 4Gb/s and the rest are (literally) unusable. Whilst I worked at a recent client, we challenged this option with Cisco. As a consequence, 3.1(1) of SAN-OS allows the disabling of all restrictions so you can take the risk and set all ports to 4Gb/s and then you’re on your own.

How much should you pay for these ports? What are they actually worth? Should a 48-port line card port cost the same as a 24-port line card port? Or, should they be rated on bandwidth? Some customers choose to use 24-port line cards for storage connections or even the 4-port 10Gb/s cards for ISLs. I think they are pointless. Cisco 9513’s are big beasts; they eat a lot of power and need a lot of cooling. Why wouldn’t you want to cram as many ports into a chassis as possible?

The answer is to look at a new model for port allocation. Move away from the concepts of core-edge and edge-core-edge and mix storage ports and host ports on the same switch and where possible within the same port group. This would minimise the impact of moving off-blade, off-switch or even out of port group.

How much should you pay for these ports? I’d prefer to work out a price per Gb/s. From the prices I’ve seen, that makes Brocade way cheaper than Cisco.

Tuning Manager CLI

I've been working with the HiCommand Tuning Manager CLI over the last few days in order to get more performance information on 9900 arrays. Tuning Manager (5.1 in my case) just doesn't let me present data in a format I find useful, and I suppose that's not really surprising as, unless you're going to add a complete reporting engine into the product, then you'll be wanting to get the data out of the HTnM database and build your own bespoke reports.

So I had high hopes for the HTnM CLI, but I was unfortunately disappointed. Yes, I can drag out port, LDEV, subsystem (cache etc) and array group details, however I can only extract one time period of records at a time. I can display all the LDEVs for a specific hour, or a day (if I've been aggregating the data) but I can't specify a date or time range. This means I've had to script extracting and merging the data and the result is, it is sloooow. Really slow. One other really annoying feature - fields that report byte throughput sometimes report as "4.2 KB" sometimes as "5 MB" - which programmer thought a comma delimited output would want a unit suffix?

I'm expecting delivery of HTnM 5.5 (I think 5.5.3 to be specific) this week and here's what I'm hoping to find; (a) the ability to report over date/time range (b) the database schema to be exposed for me to extract data directly. I'm not asking much - nothing much more than other products offer. Oh, and hopefully something considerably faster than now.

What's your favorite fruit? EMC versus HDS

Nigel has posted the age old question, which is best EMC or HDS? For those who watch Harry Hill - there's only one way to sort it out - fiiiiight!

But seriously, I have been working with both EMC and HDS for the last 6 years on large scale deployments and you can bet I have my opinion - Nigel, more opinion than I can put on a comment on your site, so forgive me for hijacking your post.

Firstly, the USP and DMX have fundamentally the same architecture. Front end adaptor ports and processors, centralised and replicated cache and disks on back-end directors. All components are connected to each other providing a "shared everything" configuration. Both arrays use hard disk drives from the same manufacturers which have similar performance characteristics. Both offer multiple drive types, the DMX3 including 500GB drives.

From a scalability perspective the (current) USP scales to more front-end ports but can't scale to the capacity of the DMX3. Personally, I think the DMX3 scaling is irrelevant. Who in their right mind would put 2400 drives into a single array (especially with only 64 FC ports)? The USP offers 4GB FC ports, I'm not sure if DMX3 offers that too. The USP scales to 192 ports, the DMX3 only 64 (or 80 if you lose some back-end directors).

The way DMX3 and USP disks are laid out is different. The USP groups disks into array groups depending on the RAID type - for instance a 6+2 RAID group has 8 drives. It's then up to you how you carve out the LUNs - they're completely customisable to your choice of size. Although a configuration file can be loaded (like an EMC binfile) its usually never used and LUNs are user-created through a web interface to the USP SVP called Storage Navigator. LUN numbering is also user configured, so it's possible to carve all LUNs consecutively from the same RAID group - not desirable if you assign LUNs sequentially and put them on the same host. EMC split physical drives into hypers. Hypers are then recombined to create LUNs - two hypers for a RAID1 LUN, 4 hypers for a RAID5 LUN. The hypers are selected from different (and usually opposing) back-end FC loops to provide resiliency and performance. It is possible for users to create LUNs on EMC arrays (using Solutions Enabler), but usually not done. Customers tend to get EMC to create new LUNs via a binfile change which replaces the mapping of LUNs with a new configuration. This can be a pain as it has to go though EMC validation and the configuration has to be locked for new configurations until EMC implement the binfile.

For me, the main difference is how features such as synchronous replication are managed. With EMC, each LUN has a personality even before it is assigned to a host or storage port. This may be a source LUN for SRDF (an R1) or a target LUN (an R2). Replication is defined from LUN to LUN, irrespective of how the LUNs are then assigned out. HDS on the other hand, only allow replication for LUNs to be established once they are presented on a storage port and the pairing is based on the position of the LUN on the port. This isn't easy to manage and I think prone to error.

Now we come to software. EMC wipe the floor with HDS at this point. Solutions Enabler, the tool used to interact with the DMX is slick, simple to operate and (usually) works with a consistent syntax. The logic to ensure replication and point-in-time commands don't conflict or lose data is very good and it takes a certain amount of effort to screw data up. Solutions Enabler is a CLI and so quick to install and a "lite" application. There's a GUI version (SMC) and then the full blown ECC.

HDS's software still leaves a lot to be desired. Tools such as Tuning Manager and Device Manager are still cumbersome. There is CLIEX, which provides some functionality via the command line, but none of it is as slick as EMC. Anyone who uses CCI (especially earlier versions) will know how fraught with danger using CCI commands can be.

For reliability, I can only comment on my experiences. I've found HDS marginally more reliable than EMC, but that's not to say DMX isn't reliable.

Overall, I'd choose HDS for hardware. I can configure it more easily, it scales better, and - as Hu mentions almost weekly, it supports virtualisation (more on that in a moment). If I was dependent on a complex replication configuration, then I'd choose EMC.

One feature I've not mentioned earlier is virtualisation. HDS USP and NSC55 offer the ability to present externally connected arrays and present them as HDS storage. There are lots of benefits for this - migration, cost saving etc. I don't need to list them all. It's true that virtualisation is a great feature but it is *not* free and you have to look at the cost benefit of using it - or beat your HDS salesman up to give you it for free. Another useful HDS feature is partitioning. An array can be partitioned to look like up to 32 separate arrays. Great if you want to segment cache, ports and array groups to isolate for performance or security.

There are lots of other things I could talk about but I think if I go on much further I will start rambling...

Optimisation tools

Large disk arrays can suffer from an imbalance of data across their RAID/parity groups. This is inevitable even if you plan your LUN allocation as data profiles change over time and storage is allocated and de-allocated.

So, tools are available. Think of EMC Optimizer, HDS Cruise Control and Volume Migrator.

I've put a poll up on the blog to see what people think - I have my own views and I'll save them until after the vote closes next week.

Goodbye ASNP

It's all over. ASNP is no more. Not really a surprise as it stood for nothing useful. With 2500 members, it could have been so much more, however I think it won't be missed.

Kryptonite Discovered!

Totally off post but fantastic non the less!! Kryptonite Discovered

Hurrah for EMC

Hurrah! EMC has implemented SMI-S v1.2 in ControlCenter and DMX/Clariion (although the reference on the SNIA website seems to relate to SMI-S v1.1). Actually it seems that you need ECC v6.0 (not out yet and likely to be a mother of an upgrade from the current version) and I'd imagine the array support has been achieved using Solutions Enabler.

So quick poll, how many of you out there are using ECC to manage IBM DSxxx or HDS USP arrays? How many of you are using HSSM to manage ECC arrays? How many of you are using IBM TPC to manage anything other than DSxxx arrays??

Simulator Update

Following a few comments on the previous simulator post, it doesn't look like there are any more simulators out there for general use.

If anyone does know - feel free to comment!

Simulator Update

Following a few comments on the previous simulator post, it doesn't look like there are any more simulators out there for general use.

If anyone does know - feel free to comment!

The Power Question

I've seen a lot of discussion (and I think a bit of a theme at SNW) on power consumption in datacentres. Obviously the subjects of global warming and increased energy prices have put the subject at the centre of focus. But I think when datacentres are being built, there isn't an issue. The problem comes along as the datacentre fills up with equipment. Invariably, new equipment (especially in the storage world) is coming in denser and requiring more power per rack or square metre. So, as equipment is swapped out and replaced, the original calculations done on how much power per square metre is needed are no longer accurate and the balance tips from one of "have we got the space" for new equipment to one of "can we power the new kit up".

I don't see how this problem will be solved as datacentre planners will always cater for the power/cooling of today's products, not the mythical power demands of future products. Datacentres will therefore have a finite life, after which you may as well start again.

Here's a practical example; There is a manufacturer of highly dense storage arrays (that don't need to be powered up all the time) who can't deploy into a number of London datacentres I know because product density would cause the array to fall to the floor. The datacentres were never designed to take products of that weight...

Another Great Idea

I've another great idea for a software product (I have these ideas from time to time, but converting them into reality always proves difficult).

So, museum environments for backups. They're going to be a major headache going forward, even more than they are today as there are more demands on the timely keeping and retrieval of backups. What's needed is a product which can understand legacy backup products and do two things; (a) extract a copy of the backup catalog into a single database, based on a standard schema for backup data and (b) read the content of backup media directly without the need to use the old backup product.

This may seem like backup software companies giving away their IP but I don't think it is. I was in a discussion recently where EMC would not give support on (admittedly very) old versions of Legato, especially with respect to merging catalogs from multiple platforms. This leads to costly and risky options requiring the retention of legacy hardware (subject to failure), legacy software (no longer supported) and legacy media (prone to failure). The lack of a single catalog precludes the ability to easily identify and manage backups and backup images when multiple backup systems exist.

I wonder if any of the vendors out there would be happy to let me have copies and information on the defunct versions of their backup products?

Footnote; I am aware of the backup management products out there like Bocada; to my knowledge none of them actually merge the catalogs into a view at the file level or offer direct media restore.

Storage as a commodity

I just read a comment over at Zerowait regarding Netapp and proprietary hardware. It reminded me of something I was thinking about recently on the commoditisation of storage.

There's nothing worse to my mind than a storage vendor who has no competition. Inevitably in some organisations that situation can exist when a single supplier is chosen to supply (for example) switches, SAN or NAS. The difficulty though is how to avoid that situation. Most vendors would love to lock you into their proprietary tools and relating back to the above article link, Netapp is one I see who try that more than anyone. They have a bewildering array of interlinked product options; once your hooked (especially where you use a feature to retain long term backups via snapshot/vaults) then you're sucked into a dependency on their products which just isn't healthy.

What's the solution? Well, for me I like to commoditise storage functionality. Pick out those features which all vendors support and only use the proprietary features where absolutely necessary. At least then you can maintain multiple vendors all on the hook for your next piece of business.

Of course implementing commoditised storage is more difficult than just picking a few common product features. However as far as your users are concerned, a LUN is a LUN and NAS storage is NAS storage, with a few caveats on things like driver levels for HBAs and so on.

I've previously posted a modular storage comparison sheet. As an example, here are some of the features that almost all support:

• RAID 5 protection
• Consistent LUN size
• dual pathing
• Active/Passive failover
• remote replication
• Fibre Channel presentation
• SNMP alerting
• online code upgrades
• hot swappable components

Before I get lots of comments saying "hold on, not all modular products are the same"; remember I'm not saying that. What I am saying is having a consistent set of requirements allows you to maintain a shortlist of vendors who can all enjoy the healthy competition of bidding for business. So, time to draw up a NAS spreadsheet....

AoE/FCoE/iSCSI

Robin harris discusses AoE from Coraid. I looked at this last year (reminder) as I saw it as a great way to get a FC/iSCSI solution at a low cost. However, before everyone rips out their FC SANs and runs to put an Ethernet solution in place, take one step back and consider the issues. Fibre Channel is successful because it works; because it is reliable. FC switches have features such as non-blocking architecture, QOS, preferred path and so on which help to remove or eliminate throughput or performance issues. Would ATA over Ethernet (or for that matter as it seems to be a topic of the moment) FC over Ethernet provide for that level of switch point to point bandwidth guarantee?

Consider also your monitoring tools. Both Brocade and Cisco offer features to do traffic redirecting (e.g. SPAN ports) to easily analyse SAN traffic without putting TAPs in place. Will AoE and FCoE offer that?

Consider security. Will AoE provide the same level of security as FC?

Without a doubt, you get what you pay for; however you should only pay for what you *need*. If you are running a mission critical application FC is still the best option - interoperability is more widely tested; diagnostic tools are mature; the technology is reliable. I do think there's a place for AoE, iSCSI and FCoE, but use it in the wrong place and what you save in cost, you may pay for later in downtime.

Where are all the simulators

I love the Netapp simulator (well, apart from the annoying issues with creating and deleting disks) and I use it all the time. It is great for testing ideas, testing scripting and generally refreshing knowledge on commands before having to touch real equipment. I use it with VMware (as I have probably mentioned before) and I can knock up a new environment in a few minutes by cloning an existing machine. Netapp have got a huge advantage in offering the tool as it enables customers who can't or won't put in test equipment to do work and protect their production environments.

So, where are all the other simulators? Is it just that I don't know they exist or do most vendors not provide them? For the same reasons as I mentioned above, if there were simulators for EMC DMX, HDS USP, Cisco and Brocade/McDATA switches, then there would be a huge opportunity for people to test and develop scripts, test upgrades and other useful work.

Would anyone else like a simulator? Can the vendors tell me why they don't produce them?

Improving efficiency

Hu posted an interesting view of storage utilisation here. His view is that virtualisation on the USP would improve on the average 30% utilisation. I have to say that I disagree with this. There are lots of reasons why storage remains unused in the enterprise:

1. inefficient administration by Storage Admins, SAs and DBAs (lost/orphan disks etc)
2. deliberate overallocation by SAs and DBAs (in an attempt to manage change control versus growth)
3. in process tasks (frame/host migrations, delayed decommissions)
4. data distribution for performance management
5. Storage Admin "buffers" to manage growth.

Many of these issues are process driven rather than due to the inadequacies of technology. Now, "true" virtualisation may address some of the problems listed above, especially those technologies which provide for thin provisioning or other overallocation methods. There are plenty of technologies on the market already offering this style of allocation however there are obviously shortcomings with the technology that could hold it back; most notably are performance and reporting.

Performance is seen as a problem due to the overhead of having to work out where blocks have been virtualised to. In addition, I/O must be written to the virtualisation device and rewritten to the physical storage creating a "dual" write scenario. Data distributed across multiple devices may not provide the same performance profile and lead to uneven response times. In fact this scenario already exists in existing enterprise storage. Data is written to cache and destaged at a later time; arrays may contain more than one type of disk size and type; data is mapped across physical disks in a RAID configuration.

Reporting is an issue as most people like to know where their data resides. There are good reasons for this; if hardware fails or if any disaster scenario is invoked then it is important to know what data is pinned where; is it still left in cache? If a RAID group fails, what data did it contain and from which volumes? In addition, overallocation creates its own problems. Predicting how virtual volumes will increase their storage usage is tricky and you most certainly don't want to get caught out refusing I/O write requests for production disks. This issue was very obvious with the early implementation of thin provisioning on Iceberg (a StorageTek storage subsystem) where reporting failed to cope with volumes that had been copied with snapshot.

Now, if HDS were to add thin provisioning to the USP, how good would that be....

Distributed Backup

Following on from a previous post on RAID and backup, I've been doing some more thinking on how to back up consumer data from a PC workstation. I reckon I've got about over 200GB of data on my server, which previously was on my main workstation. I dabbled with the Linksys NSLU2 however I hated it; I was really nervous (a) about the fact I would lose access to the device (b) it couldn't cope with the volume of files and seemed to lose track of what I had allocated and (c) how I would recover the data from the USB drives I used if the device eventually packed up. In fact, I got rid of the NSLU2 when it did lose track of my data. I was lucky to find a Windows read-only driver capable of reading the NSLU2 format and I got my data back.

Getting back to the question in hand, how would I back up 200GB? I guess I could fork out a few grand for an LTO drive and tapes, but that's not cost effective. I could do disk to disk copy (which I do) but D2D isn't as portable as tape and much more expensive if I intend to maintain multiple copies. I should mention that I've automatically discounted DVD and HD-DVD/Blu-Ray due to lack of capacity and cost (the same applies to the latest optical drives too).

I could use one of the many network backup services on offer. About 10 years ago, I looked at the feasibility of setting up one of these services for the storage company I worked for at the time. It was almost feasible; Freeserve was doing "free" dial-up internet (you paid for just the cost of the calls) and companies such as Energis were selling virtual dial-up modems on very good terms. However the backup model failed as the cost to the customer just didn't stack up due to the length of time to copy files out to the backup service.

I think network backup services *could* be the best answer to safeguarding your PC/workstation data. However the existing services have issues for me; basically I don't trust someone else with my data, which could include bank details, confidential letters and files. Even if I can encrypt my data as it is transmitted to the network backup service, they still have *all* of my data and with enough compute power could crack my encryption key.

If anyone has examples of services which could provide 100% security, I'd be interested to know.

So, here's my idea. a distributed backup service. We all have plenty of free space on those 500GB drives we've installed. Why not distribute your backups amongst other users in a peer to peer fashion? There are two main drawbacks to my mind. First, how can I guarantee my data will always be available for me to access (PCs may be powered off) and secondly, how can I guarantee security?

Existing P2P services work by finding as many servers/PCs as possible which hold the data you want to download. Many may not be online; many may be online and running slow. By locating multiple copies of the required data, then hopefully one or more will be online and available for download.

The same can be applied to backups; split files up and distribute the fragments to P2P servers and index where they are. The fragments would need to be encrypted in some way to guarantee anonymity but make them common to files on both your machine and others. You then maintain an index to rebuild the backup data; all that then needs to be backed up is the index which could easily fit onto something like a CD-ROM. All data could then be recovered using just a small CD index, which could be recreated from anywhere.

There are a lot of unanswered issues; how would data be encrypted; how would the fragments be created and "de-duplicated", how would fragments be distributed across the P2P members to ensure availability? How would the fragments be created to prevent the actual original files from being discovered?

Still, it's only a concept at this stage. But using the internet and all that unused disk space out there could prove a winner.

Giving RAID the thumbs up

Just read Robin Harris' post at his new blog location; http://blogs.zdnet.com/storage/?p=116 and his comment on another blog discussing RAID. He quotes a VAR who has tracked disk failures and thinks RAID is an expensive luxury for desktops.

It's interesting to see the failure rates quoted, anywhere from 1-3%, which on the face of it, seems low. However when its *your* disk that has failed and the data is irretrievable, there's cold comfort to be had in failure rate statistics. I run RAID1 on my server; I have two 500GB SATA drives. Backing up that volume of data on a regular basis is a nightware without investing in a very expensive backup solution like LTO and it is a real disappointment to see tape hasn't kept pace with disk in terms of the capacity/cost ratio.

So, I'm sticking with RAID. I augment it with disk-to-disk backups because, yes, you do have to cater for the d'oh factor of user errors or even dodgy software which corrupts files too, but RAID works for me and that's all I need to worry about.

Uh Oh Domino

It seems that the world is moving to Exchange for email messaging. Unfortunately there are some of us still using Lotus Notes/Domino.

As a messaging product, it seems to me to be reasonably efficient; our Domino servers can support upwards of a thousand users, perhaps 1-2TB of Notes mailboxes. Domino stores the mailboxes as individual files with the .nsf extension. Each of these is opened and held by the nserver.exe task. When using Netbackup with the Notes/Domino agent, the Netbackup client backs up all nsf files on a full backup and the transaction logs and changed nsf files (i.e. those with a new DBID) for an incremental backup. This creates a significant amount of hassle when it comes to performing restores.

A restore is either the full nsf file from the last full backup, or the nsf file plus transaction logs, which are then applied to the nsf file to bring the mailbox up to date. This process is incredibly inefficient because (a) transaction logs contain data for all users and must be scanned for the records relating to the restoring mailbox (b) the transaction logs need to be restored to a temporary file area, which could be considerable (c) the restored logs are disregarded after the restore has completed and so have to be restored again for the next mailbox restore.

So, I’ve been looking at ways to bin Netbackup and improve the backup/restore process. As servers are being rebuilt on Windows 2003 Server, I’ve been looking at VSS (Volume Shadowcopy Services). This is a Windows feature which permits snapshots of file systems to be taken in co-operation with applications and underlying storage. In this instance there isn’t a Lotus Domino provider, so any snapshots taken are dirty (however I did find the dbcache flush command which flushes updates and releases all nsf files). Netapp used to have a product called SnapManager for Lotus Domino which enabled Netapp snapshots of mailboxes using the Domino Backup API. The product has been phased out, as tests performed by Netapp show that dirty snapshots with the security of logs can be used to restore mailboxes successfully. IBM provide trial versions of Domino, so, I’ve downloaded and installed Domino onto one of my test servers under VMware and run the load simulator while taking snapshots with VSS. I’ve also successfully restored a mailbox from a snapshot so there’s no doubting the process works. However my simple task isn’t one of scale. Typical mailboxes are up to 1GB in size and there could be hundreds of active users on a system at any one time. My concern is whether VSS can manage to take snapshots with this level of activity (and not impact the O/S) but also whether the snapshots will be clean or what level of corruption we can expect.

The only way to test this is to implement on a full scale Domino environment and probably with live users. That’s where things could get interesting!

WWN Decoder

As pointed out by Richard, my WWN decoder stopped working when I redid my website. Here's a new link; http://www.brookend.com/html/resources/wwndecoder.asp.

Developing a Tiering Strategy

Implementing a storage tiering strategy is a big thing these days. Everyone should do it. If you don't then you're not a "proper" storage administrator. Being serious and moving away from the hype for a second, there is a lot of sense in implementing tiering. It comes down to 1 thing - cost. If disk and tape storage was free, we'd place all our data on the fastest media. Unfortunately storage isn't free and therefore matching data value to storage tiers is an effective way of saving money.

Choosing the Metrics

In order to create tiers it's necessary to set the metrics that define different tiers of storage. There are many to choose from:

• Response time
• Throughput
• Availability (e.g. 5 9's)
• Disk Geometry (73/146/300/500GB)
• Disk interconnection (SATA/FC/SCSI)
• Usage profile (Serial/Random)
• Access Profile (24x7, infrequent)
• Data value
• Array Type (modular/enteprise)
• Protection (RAID levels)

There are easily more, but these give you a flavour of what could be selected. In reality, to determine the metrics to use, you really need to look at what would act as a differentiator in your environment. For example, would it be really necessary to use 15K speed drives rather than 10K? Is availability important - should RAID6 be considered over RAID5? Is there data in the organisation that would exist happily on SATA drives rather than fibre channel? Choosing the metrics is a difficult call to make as it relies on knowing your environment to a high degree.

There are also a number of other options to consider. Tiers may be used to differentiate functionality, for example tiers could be used to specify whether remote replication or point-in-time copies are permitted.

Is It Worth It?

Once you've outlined the tiers to implement, you have to ask a simple question - will people use the storage tiers you've chosen? Tiering only works if you can retain a high usage percentage of the storage you deploy - it's no use deploying 20TB of one tier of storage and only using 10% of it. This is a key factor. There will be a minimum footprint and capacity which must be purchased for each tier and unless you can guarantee that storage will be used, any saving from tiering may be negated by unused resources. Narrow your tiering choices down to those you think are actually practical to implement.

Making the Move

So, the tiers are set, storage has been evaluated and migration targets have been identified. How do you make it worthwhile for your customers to migrate? Again, things come back to cost. Tiers of storage will attract differing costs for the customer and calculating and identifying the cost savings will provide a justification for investing in the migration. In addition, tiers can be introduced as part of a standard technology refresh - a process that regularly happens anyway.

Gotcha!

There are always going to be pitfalls with implementing tiering:

1. Don't get left with unusable resources. It may be appealing to identify lots of storage which can be pushed to a lower tier. However, if the existing tier of storage is not end-of-life or unless you have customers for it, you could end up with a lot of high tier unused storage which reflects badly on your efficiency targets. Make sure new storage brought in for tiering doesn't impact your overall storage usage efficiency.
2. Avoid implementing technology specific tiers which may change over time. One example; it is popular to choose to tier by drive size on the assumption that higher capacity drives offer a lower performance and therefore are matched to a lower tier. But what happens when the predominant drive type changes or you buy a new array in which the larger drives perform equally well compared to an older array? How should those tiers be classified?
3. Be careful when choosing absolute parameters for tiers. For example, it is tempting to quote response time characteristics in tiers. However, no subsystem can guarantee consistent response times. It may be more appropriate to set confidence limits, such as offering "<>

Iterative Process

Developing a tiering strategy is an iterative process which will constantly be refined over time. There's no doubt, that implemented correctly, it will save money. Just don't implement it and forget about it.

Standards for Measuring Performance

I had a bit of banter following a post I made on ITToolbox earlier in the week. The question was posed as to why major disk array vendors (other than Netapp) don't produce performance statistics. There are benchmarks; the Standard Performance Evaluation Corporation and Storage Performance Council for instance. However SPEC only covers NAS and neither HDS and EMC sign up for SPC.

Producing a consistent testing standard is extremely difficult. Each storage array vendor has designed their hardware to have unique selling points and more importantly, each vendor can't make their systems too close to their rivals otherwise there'd be more money made by lawyers than the vendors themselves.

Let's pick a few examples. In terms of architecture, DMX (EMC) and USP (HDS) have a similar design. They both have front-end channel adaptor cards, central cache and back-end loops on which (probably identical) hard disk drives are connected. However the way in which the physical storage is carved up is totally different.

HDS uses the concept of an array or parity group; a 6+2 RAID group has 8 disks of the same size which can be carved up into logical units (LUNs). The address at which these LUNs are mapped is up to the user, but typically LUNs will be dispersed across multiple array groups to ensure that consecutive LUNs are not mapped to the same physical disks. This process ensures that data is hitting as many spindles as possible.

EMC chooses another method. Each physical drive is divided into hypers, or logical slices. These are then recombined to make LUNs. RAID 1 LUNs have 2 hypers, RAID5 LUNs have 4 hypers. Each hyper is taken from a different back-end drive loop to improve performance and resiliency.

Now, the comments earlier in the week referred to Netapp. Their architecture comes from a NAS file serving design, which uses RAID 4 and has all the operations for RAID calculations handled in memory. LUNs are carved out of RAID "volumes" or aggregates.

So what is a fair configuration to use? Should it be the same number of spindles? How much cache? How many backend loops and how many front-end ports? Each vendor can make their equipment run at peak performance but choosing a "standard" configuration which sets a level playing field for all vendors is near impossible. Not only that, some manufacturers may claim their equipment scales better with more servers. How would that be included and tested for?

Perhaps rather than doing direct comparisons, vendors should submit standard style configurations, based on a GB capacity on preset LUN sizes against which testing is performed using common I/O profiles. This, with list price would let us make up our own minds.

Either way, any degree of testing won't and shouldn't stop healthy discussion!

Software Shortcomings

I was invited last week to speak at the first (of hopefully) many HDS User Forums in the UK. The subject was Storage Resource Management and I talked on the subject and my thoughts for about 30 minutes. One slide generated the most interest and I've included it here. It shows some of the issues current SRM products have and which are not really being addressed by software vendors.

A case in point is the installation of Device Manager I performed this week on a virtual domain. HiCommand Device Manager 5.1 is supported on VMware (2.5) however I couldn't get the software to install at all. I tried the previous version which worked fine, so I was confident the Windows 2003 build was OK. HDS pointed me at a new feature I'd not seen before, Data Execution Prevention which is intended to prevent certain types of virus attacks based on buffer overflows. Whilst this solved the problem, it didn't fill me with a great deal of confidence to think Windows judged HDS's software as a virus. With DEP enabled, the installation got further but still eventually failed. On HDS' advice, re-running the installation again worked.

At the forum, HDS presented their SRM roadmap. If it all comes to fruition then I'll be able to do my provisioning, monitoring and other storage management tasks using my Blackberry whilst sipping Pina Colada's on a Carribbean beach. Back in the real world, my concern is that if the existing tools don't even install cleanly, how am I expected to trust a tool which is moving my disks around dynamically in the background?

It's easy for me to target HDS in this instance but all vendors are equally culpable. I think there's a need for vendors to "walk before they can run". Personally, I'd have more trust in a software tool that was 100% reliable than one which offered me lots of new whizzy features. Vendors, get the basics sorted. Get confidence in your tools and build on that. That way I might get to do provisioning from the beach before I'm too old and grey to enjoy it.

Backing Up Branches

I've spent the last few weeks being heavily involved in a storage performance issue. Unfortunately I can't discuss the details as they're too sensitive (in fact, I can't even mention the TLA vendor in question), however it did make me think more about how we validate a storage architecture design will actually work.

As part of another piece of work I'm doing, I have been looking at getting branch or remote office data into core datacentre infrastructure, the premise being not to write tapes in remote locations and therefore be at risk of compromising personal information.

I've looked at a number of solutions, including the Symantec/Veritas PureDisk option (the flashy flash demo can be found here). What strikes me with this and other solutions is the lack of focus on data restores and meeting RTO and RPOs for applications that may run in branch locations. In fact, I think most products are aimed at the remote office which runs a small Windows implementation and therefore isn't worried too much about recovery time of their data.

The technologies such as PureDisk work by reducing the data flowing into the core datacentres, either by de-duplication, compression or both technologies. This is fine if (a) your data profile meets their view of compression (b) your applications/data produce a small percentage of changed data on a daily basis. Unfortunately for some applications this just doesn't occur. In addition, if you lose the site (or even easier, corrupt or lose the local metadata database) then you're looking at a total restore from core to recover an entire server. These solutions work on the basis that branch offices have low WAN bandwidth and the backup tools utilise this by making a small pipe appear as a much larger pipe through the aforementioned de-dupe and compression. This doesn't help if a full restore needs to be performed.

Some people may say "but you could manage the restore by sending a tape to the site containing the last backup" - however that defeats the original objective of keeping tapes out of the branch.
I'd like to see some real-world examples from Symantec/Veritas showing how long backup/restores would take with their product. Of course they're not going to provide that as it would immediately highlight the issues involved. Perhaps we need an independent backup testing lab to do that comparison for all backup products on the market.