ECC Lite

I've talked previously about management tools. I don't think we'll see a unified "one tool fits all" solution. However in the meantime I think there are some things we can do;

So, ECC, huge product, lots of features. What happens if you want to do a little configuration work but don't want to deploy an entire ECC infrastructure. I suggest ECC Lite, a simple GUI based version of SYMCLI. It lets you visualise your storage, do the same TimeFinder and SRDF functions and to build and kick off symconfigure. It runs under windows on your local machine (if you have Gatekeeper access to your Symmetrix) and doesn't require big servers. Lastly it is FREE. It doesn't exist; perhaps I'll write it.

In the interests of equality (certainly with HDS, I can't comment on IBM) if HDS provide command device access to their arrays, I'd write one for that too.

So, we get the basics; the terms that define LUNs, array groups, etc can then be made generic, then we can create a product that spans both vendors using generic storage terms. I'd like that one too.

Pushing the datacentre distances

I've been wrestling with a couple of architecture issues over the last few weeks. They are interlinked; but stand alone as separate issues.

First is three datacentre. So imagine a scenario where there is a normal datacentre pair, reasonably close to each other. New rules require another datacentre some distance away, which will be out of zone. This means a distance of perhaps 50 or 60 miles. The original datacentre pair were only 5 miles apart. The aim is to be able to replicate between the local pair with a third copy in the remote centre. Not a problem you might say; the local pair can be synchronously replicated, the remote copy can be async. Well, OK, that would work and the major array vendors can do that today. But....I really don't want async and I don't necessarily want to pay for three copies.

How can I do it? At the moment I don't think it is possible; certainly I need to still have three copies and I can't reduce the replication to the remote site to synchronous without incurring a penalty. There is a possible scenario using a USP to virtualise the disks at the remote site to one of the local sites however this may not provide suitable data integrity.

I'd like to see the vendors provide this solution; in fact I think the best place for this requirement to be implemented is within the SAN as part of a director. I don't think any vendors are working to develop this; if they are they'd be cutting into the storage array vendors patch, something that might not be too popular.

In case anyone decides to comment, I am aware of EMC InVista but I don't think it offers this degree of functionality and flexibility.

Device Manager TNG & Central Storage Management

Right, I've got Device Manager 5 installed pending installation of Tuning Manager 5. Agreed, the layout looks better (shame it is all still Java, my least programming tool). Now I need to generate some data. But yes it looks better.

Old JWT at DrunkenData seems to think www.storagerevolution.com is still alive. Here's his recent post discussing the site plans. Unfortunately I don't think there's any holy grail here (excuse the DaVinci Code reference - it wasn't intentional - I haven't even seen the film or read the book). Storage is evolving daily, let's face it, EMC is buying nearly a company a day at the moment! Keeping up with this technology is impossible; spending 12 months developing an infrastructure will provide you with a design that is 12 months out of date, a long time in this industry. Just look at how quickly new technology is being introduced; think of how technologies such as virtualisation will skew all the products in terms of accounting and managing data.

Creating a single consistent framework for all vendors and technologies is simply impossible, so we have to think more pragmatically. We need a baseline; that baseline needs to create generic storage terms against which we can map technologies. Let's face it, all the major storage vendors have split mirror technology and COW technology, they just call it a different name. The same applies to other functionality - synchronous/async replication; NAS, iSCSI and so on.

My suggestion; start with the basics. Create a framework that maps against hardware from an agnostic position. Revise and develop it; increase the complexity to encompass application and business requirements. OK, now that will take some time; but at least there will be a working solution in place.

NO Storage Vendor is going to hold back their product development lifecycle just to make sure they work with generic standards, they will continue to press for market share bring new features to the market and always look to maintain their USP.

We will have to run just to keep up.

The missing storage generation

My Tuning Manager issues go on; there's obviously a problem with the version I'm using - reports don't work correctly even after the re-install. However one, small side effect, I picked up the wrong subsystems to monitor after a reboot of the TM server. This was because I didn't follow the instructions and ensure that the LUN binding remained the same after reboot. That meant the physicaldevice assignments from Windows changed after reboot and TM kindly picked up the wrong systems. My fault. My suggestion of the day, RTFM.

I had a very nice man at HDS contact me and explain how much better Version 5 is. Well, I'd agree - nice interface, in fact the whole HiCommand suite has nice colours at the top. Tuning Manager is green I think, Device Manager is blue. From what I know, the aim was to improve the GUI first, hence the first 5.x release - then concentrate on the other features which needed improving. I'll have an evaluation of 5.0.xxx done soon and I'll let you know how it rocks.

I started out working on mainframes. For those who are too young to remember, these were huge machines with football fields of storage and stored about the same amount of data as you can get on an iPod (apologies to those older than me who remember 360, I started on 370 and ESA). Running mainframes has become a problem; see http://www.techweb.com/wire/hardware/170000208 which dicusses the shortage of mainframe trained people. Anyway, that wasn't my reason for bringing it up; here's the reason. 17 years ago, IBM released DFSMS - the Storage Management Subsystem (or System Managed Storage). This allowed files to be directed to the most appropriate storage tier based on the importance of the file. These policy settings could be specified using a basic like language which allowed complex decisions on file locations to be made. Data was backed up and moved between tiers using a hierarchical storage manager - ILM.

Best of all, we had SAN in the form of ESCON, Enterprise System Connection, only recently replaced by FICON, but a fibre interconnect solution. One advanced feature offered virtualisation; EMIF - Escon Multi-Image Facility. This allowed any interface (e.g. HBA) to be used by any domain on the mainframe to connect to the storage. This isn't even possible today - every domain in say a Sun F15 or E25 as an example, needs dedicated HBA cards.

So what happened to the development of storage features between the late 80's and today? Where did the storage generation go? Personally, I moved to Open Systems and watched all these features being re-invented. Best of all, I loved the way all the "new" features in the O/S world were discovered! Nostalgia is not what it used to be....

Tuning Manager Again; Virtualising Tape

Tuning Manager didn't work properly. I needed to re-install it. Unfortunately I got into an infinite loop. The uninstall process requested I start one of the Tuning Manager services, when I did and tried again, the uninstall failed asking me to stop all services...whoa.

So, strategically thinking, where should backup be virtualised? There are virtualising options to completely virtualise the tape drive and the tape cartridge; this can be on disk or eventually to tape, either in a one-to-one relationship or virtualised. These are good but have drawbacks; how do I get off the virtualised media? If the data was written to tape in a 1:1 relationship with the backup product, then I'm OK, if not I need a migration tool.

OK, so I could abandon the concept of tapes and write backups to a disk pool; great, but how do I retain those backups indefinitely? Not really practical on disk.

So, I could abandon the backup software altogether - if I use NAS products I can take snapshot type backups, there are even products which will snapshot SAN (SCSI) devices. However I'm tied to this product for my data backup going forward.

Hmm. It is a quandry. I'm settling on a number of requirements;

1. Retain data in the format of the backup product, wherever the data finally ends up.
2. Use tape as the final destination for data which must be maintained long term.
3. Use disk as an intermediary; depending on the backup product, choose integral disk backups or use a VTL solution.
4. Don't completely virtualise the entire tape subsystem; allow tapes to be written which can still be understood by the backup product.

Seems simple. Anyway, Tuning Manager, sorted it. Needed to uninstall from the console. Shame I lost all my data though.

Tuning Manager, storage on storage

I've done more work on Tuning Manager today and added a number of systems for collection. I'm not keen on the interface (currently using version 4.1) and so will be getting version 5.0 installed quickly. The proof of the pudding will be in the information I get out of the product. I'll give it a few days then start to see what we're collecting.

How much storage is now used to monitor, ahem, storage? With the creation of databases for collating performance data and configuration data (think of ECC for instance) a significant infrastructure and storage volume is needed simply to manage the storage itself. How much storage is needed before we reach Parkinson's Law and all the storage we have is for monitoring itself?

I shared a few beers with Dan Crain last night. It's an interesting time for the storage fabric vendors. Whilst I can't mention specifics, there's a lot of interesting stuff on the horizon and the storage fabric is moving well and truly away from a basic interconnect. Value add is going to become a big thing; a basic monolithic storage interconnect will no longer cut the mustard.

More on ECC and Black Boxes

So, I checked it out. The official position is not clear. There's no direct support under VMware ESX, but some components are supported (things like the fibre channel agent etc) but a lot of stuff won't be considered until ESX 3.0. Still waiting for clarification.

I've also been looking at USP configurations on two fronts; first, Virtual Partition Manager (VPM) allow storage resources on a USP to be placed into individual storage partitions (SLPRs). These can be administered separately. We're looking at this for separation of tier 1 and tier 2 storage. I like the idea that both storage ports and cache can be partitioned along with LDEVs to create a separately administerable storage partition. This not only simplifies the administration of multi-tiered storage in a single frame but also protects resources at each level of tiering.

The second USP configuration work involved LDEV layout. Although we ask our customers to consider the USP as a "black box", it clearly isn't and LUN distribution is paramount to obtain best performance, so I've been looking at where we allocate ShadowImage LDEVs. We plan to use higher capacity disks for SI to reduce cost, so disk layout is important. The conclusion is we should keep the primary disk and the SI copy within the same half of the frame, i.e. either on the left or the right as this will give the best performance when replicating data.

For those of you who have no idea what I'm on about, I'll do an entry on the TagmaStore one day, or you can look at the HDS website.

Oversubscribed and undersupported

Been thinking about oversubscription and port usage. I suspect most SANs are seriously underutilising their available bandwidth. Oversubscription is therefore a good thing - but it will be dependent on being able to use that oversubscription. Cisco switches (gen2) allow oversubscription at the port group level. On a 48-port card this is per group of 12 ports but there are restrictions - try putting more than two 4Gb/s ports at dedicated in a port group and you'll have to start turning other ports off! Restricting per port group is not enough. McDATA restrict by 32 port blade - fine, but the blades run at 2Gb/s unrestricted anyway. Brocade - not restricted - am I paying for bandwidth I don't need? Give me lots of ports and a reasonable amount of bandwidth for my money - and let me share it everywhere.

Why does EMC not support ECC on VMware? OK, I guess the Repository and Server components may cause a problem, but the stores - surely they can. What is going on with EMC and acquisitions? I can't believe it can integrate all of the software from the companies recently purchased - come on, at least ECC on VMware please - they're both your products.

Intellectual Property

A while back I talked about the size increase in hard disk drives. We see disks growing at Moore's Law rates, predicting a doubling of size every 18 months to 2 years. EMC proclaim a petabyte of storage within a single storage subsystem. But what does that actually mean?

Imagine an average page of text, around 600 words and 3000 characters. Multiply that up to a book of 300 pages, that's around 900KB to store natively, but say 5MB with formatting. So, 1 Petabyte could store 215 million books - more than the entire collection of the British Library!

OK, this is hardly a scientific example, however what it does show is the amount of intellectual capital that exists in a single disk subsystem and therefore the subsequent responsibility on storage managers and storage architects to protect and provide timely access to this data.

How can we do this?

Within the subsystem itself we can guard against hardware failure with resiliency - RAID5/6, redundant components, multiple power sources - and picking a trusted vendor with a track record in producing solid products.

Outside the subsystem, we can prevent loss due to more catastrophic reasons by remote replication of data to another datacentre which could be located near or a long distance away. It is possible to take multiple replicas to be completely sure.

To prevent against data corruption rather than equipment failure, we can take regular backups - these could be snapshots on disk or tape based backups secured offsite.

So there are lots of options. The starting point however should always be to evaluate the value of the data to be secured and to base investment in data protection on the value of that data. Don't bother synchronously replicating data which will be discarded and never reused or which can easily be recreated (for example User Test Data). Conversely, use replication and snapshot technologies on production databases used to deliver key company services.

Remember that data is any company's intellectual capital and it is our responsibility to ensure its safety.

Planning for SAN resilience

One aspect of storage design must consider the issues of resilience. All infrastructure components are subject to failure; even five 9's of reliability means an outage of just over 5 minutes per year. How do we plan for that?

Multipathing

This is a simple one; two or more entire fabrics connecting hosts to storage. If one fabric fails, then the other can take over. This design consideration is not just for recovery, it assists in maintenance, so one fabric can be upgraded whilst the other maintains operation. Multipathing is of course expensive; doubling up on all equipment. But it does reduce the risk of failure to an almost negligible number.

Director Class versus Switch

As mentioned, director class switches offer at least five 9's availability. Departmental switches on the other hand offer more like three 9s, which is a considerably less resilient piece of equipment. So, for a resilient SAN architecture, don't put deparmental switches into the infrastructure at points of criticality.

Component Failure

Director class five 9's refers to the failure of an entire switch. It doesn't refer to the resilience of an individual component. So, plan to spread risk across multiple components. That may mean separate switches, it may mean across separate blades on switches. Hardware capacity growth means blades have moved from 4-port (e.g. McDATA) to 32 and 48 port blades (Cisco), reconcentrating the risk back into a single blade. So, spread ISLs across blades, spread clustered servers across switches and so on.

In summary, look at the failure points of your hardware. Where they can't be remedied with redundant components, plan to spread the risk across multiple components instead. If you can afford it then duplicate the equipment with multiple fabrics, HBAs and storage ports.