Replacing the Virtualisation Component - II

Thanks to Chuck who pointed out to me SVC's ability to move virtual WWNs between nodes during replacement. At some stage in the future I may get to play with SVC but I haven't so far, so this feature eluded me. Question: is SVC the *only* block virtualisation appliance to offer this functionality and is it a seamless operation or does it require downtime?

How about InVista or Incipient (or any other vendor I may have missed off - we will assume USP doesn't have the facility)? Answers on a postcard comment please.

InVista - EMC's Problem Child

Remember when EMC offered a fabric-based storage virtualisation product called InVista? Apparently they still do, as a customer of mine mentioned recently. Unfortunately they had trouble getting any sort of decent reference site for the product and weren't that impressed.

I had a quick check on EMC's site and the last press release for the product I could find was dated December 8th 2007. Compare that to the news about DMX, Clariion and probably almost every other product put out by EMC and InVista doesn't get much press. In fact general opinion perceives InVista as EMC's problem child, finding trouble gaining acceptance with storage managers at large. Why?

To be honest, I'm not sure why. The *concept* of fabric-based virtualisation seems appealing and once the appliance has been integrated into a SAN, data migration should be simple. Perhaps there's a lack of trust in the scalability or performance, perhaps a lack of trust that the product actually works, perhaps a nervousness in placing yet another layer into the infrastructure which, if it fails, could be massively disruptive to normal storage operations.

That leads to the question - what should EMC do? I doubt they will drop the product, as that will give competitors (especially IBM & SVC) too much of a marketing opportunity. Maybe they will acquire a competitor (like Incipient) and slowly transform InVista into a new product. I don't have a crystal ball to answer that one.

I'll close with one thought - are there any other products with the word "Vista" out there that are also equally dismissed and derided?

SAN Virtual Appliances

LeftHand, FalconStor, Arkeia and Datacore all now offer VMware appliance versions of their products. I'm in the process of downloading them now and I'm hoping to install over the next few days and do some testing. I've previously mentioned some VM NAS products which I've installed but not reported back on. I'll try to summarise all my findings together.

It seems that the market for virtual appliances (certainly in storage) is getting bigger. I think this is a good thing but I'm not sure that the virtualisation technology today provides capabilities to allow all vendors to virtualise their products. I suspect that the iSCSI brigade will get best benefit out of this wave of technology but fibre channel will not, as (from my experience) VM products don't directly pass through fibre channel hardware to the VM guests (I'm aware of how RDM works in a VMware environment but I don't think pass-through of target devices is sufficient).

Will IBM produce an SVC Virtual Appliance? I doubt it, but products such as Invista should be perfect candidates for virtualising as they don't sit in the data path and the controller parts aren't critical to performance. So EMC, show us your commitment to Invista and make 3.0 the virtual version!

Using Virtualisation for DR

It's good to see virtualisation and the various products being discussed again at length. Here's an idea I had some time ago for implementing remote replication by using virtualisation. I'd be interested to know whether it is possible (so far no-one from HDS can answer the question on whether USP/UVM can do this, but read on).




The virtualisation products make a virtue out of allowing heterogenous environments to be presented as a unified storage infrastructure. This even means carving LUNs/LDEVs presented from an array into consituent parts to make logical virtual volumes at the virtualisation level. Whilst this can be done, it isn't a requirement and in fact HDS sell the USP virtualisation on the basis that you can virtualise an existing array through the USP without destroying the data, then use the USP to move the data to another physical LUN. Presumably the 1:1 mapping can be achieved on Invista and SVC (I see no reason why this wouldn't be the case). Now, as the virtualisation layer simply acts as a host (in USP's case a Windows one - not sure what the others emulate) then it is possible (but not usually desirable) to present storage which is being virtualised to both the virtual device and a local host, by using multiple paths from the external array.


If the virtualisation engine is placed in one site and the external storage in another, then the external storage could be configured to be accessed in the remote site by a DR server. See example 1.

Obviously this doesn't gain much over a standard solution using TC/SRDF other than perhaps the ability to asynchronously write to the remote storage, making use of the cache in the virtualisation engine to provide good response times. So, the second picture shows using a USP as an example, a 3 datacentre configuration where there are two local USP's providing replication between each other but the secondary LUNs in the "local DR site" are actually located on external storage in a remote datacentre. This configuration gives failover between the local site pair and also access to a third copy of data in a remote site (although technically, the third copy doesn't actually exist).

Why do this? Well, if you have two closely sited locations with existing USPs where you want to retain synchronous replication and don't want to pay for a 3rd copy of data then you get a poor man's 3DC solution without paying for that third data copy.

Clearly there are some drawbacks; you are dependent on comms links to access the secondary copy of data and in a DR scenario performance may be poor. In addition, as the DR USP has to cache writes, it may not be able to destage them to the external storage in a timely fashion to prevent cache overrun due to the latency on writing to the remote external copy.

I think there's one technical question which determines whether this solution is technically feasible and that is; how do virtualisation devices destage cached I/O to their external disks? There are two options I see; firstly they destage using an algorithm which minimises the amount of disk activity or they destage in order to ensure integrity of data on the external disk in case of a failure of the virtualisation hardware itself. I would hope the answer would be the latter rather than the former here, as if the virtualisation engine suffered some kind of hardware failure, I would want the data on disk to still have write order integrity. If this is the case then my designs presented here should mean that the remote copy of data would still be valid in case of loss of both local sites, albeit as an async copy slightly out of date.

Can IBM/HDS/EMC answer the question of integrity?

Invista

There's been a few references to Invista over the last couple of weeks, notably from Barry discussing the "stealth announcement".



I commented on Barry's blog that I felt Invista had been a failure, due to the number of sales. I'm not quite sure why this is so, as I think that virtualisation in the fabric is utimately the right place for the technology. Virtualisation can be implemented at each point in the I/O path - the host, fabric and array (I'll exclude application virtualisation as most storage managers don't manage the application stack). We already see this today; hosts use LVMs to virtualise the LUNs they are presented; Invista virtualises in the fabric; SVC from IBM sits in that middle ground between the fabric and the array and HDS and others enable virtualisation at the array level.



But why do I think fabric is best? Well, host-based virtualisation is dependent on the O/S and LVM version. Issues of support will exist as the HBAs and host software will have supported levels to match the connected arrays. It becomes complex to match multiple O/S, vendor, driver, firmware and fabric levels across many hosts and even more complex when multiple arrays are presented to the same host. For this reason and for issues of manageability, host-based virtualisation is not a scalable option. As an example, migration from an existing to a new array would require work to be completed on every server to add, lay out and migrate data.



Array-based replication provides a convenient stop-gap in the marketplace today. Using HDS's USP as an example, storage can be virtualised through the USP, appearing just as internal storage within the array would. This provides a number of benefits. Firstly driver levels for the external storage are now irrelevant (only requiring USP support, regardless of the connected host), the USP can be used to improve/smooth performance to the external storage, the USP can be used for migration tasks from older hardware and external storage can be used to store lower tiers of data, such as backups or PIT copies.



Array-based replication does have drawbacks; all externalised storage becomes dependent on the virtualising array. This makes replacement potentially complex. To date, HDS have not provided tools to seamlessly migrate away from one USP to another (as far as I am aware). In addition, there's the problem of "all your eggs in one basket"; any issue with the array (e.g. physical intervention like fire, loss of power, microcode bug etc) could result in loss of access to all of your data. Consider the upgrade scenario of moving to a higher level of code; if all data was virtualised through one array, you would want to be darn sure that both the upgrade process and the new code are going to work seamlessly...



The final option is to use fabric-based virtualisation and at the moment this means Invista and SVC. SVC is an interesting one as it isn't an array and it isn't a fabric switch, but it does effectively provide switching capabilities. Although I think SVC is a good product, there are inevitably going to be some drawbacks, most notably those similar issues to array-based virtualisation (Barry/Tony, feel free to correct me if SVC has a non-disruptive replacement path).



Invista uses a "split path" architecture to implement virtualisation. This means SCSI read/write requests are handled directly by the fabric switch, which performs the necessary changes to the fibre channel headers in order to redirect I/O to the underlying target physical device. This is achieved by the switch creating virtual initiators (for the storage to connect to) and virtual targets for the host to be mapped to. Because the virtual devices are implemented within the fabric, if should be possible to make the virtual devices accessible from any other fabric connected switch. This poses the possibility of placing the virtualised storage anywhere within a storage environment and then using the fabric to replicate data (presumably removing the need for SRDF/TrueCopy).

Other SCSI commands which inquire on the status of LUNs are handled by the Invista controller "out of band" by an IP connection from the switch to the controller. Obviously this is a slower access path but not as important in performance terms as the actual read/write activity.

I found a copy of the release notes for Invista 2.0 on Powerlink. Probably about the most significant improvement was that of clustered controllers. Other than that, the 1.0->2.0 upgrade was disappointing.

So why isn't Invista selling? Well, I've never seen any EMC salespeople mention the product never mind pushing it. Perhaps customers just don't get the benefits or expect the technology to be too complex, causing issues of support and making DR an absolute minefield.

If EMC are serious about the product you'd have expected them to be shoving it at us all the time. Maybe Barry could do for Invista what he's been doing in his recent posts for DMX-4?

Port Indexes on Cisco Switches

I spent today building some Cisco MDS9513 switches. It's good to get into the nuts and bolts of technology once in a while. They are big beasts - 2nd generation technology and with 11 usable slots, the biggest available line cards (48 ports), 528 ports in a single chassis. However the build had one fly in the ointment. As part of the (inherited) design, the chassis included a "generation 1" eight ethernet port IP line card for implementing FCIP.

The issue revolves around a feature called port indexes, which are used to track the ports installed in a MDS chassis. Generation 1 chassis have a maximum of 252 port indexes, generation 2 technology supports up to 1020 port indexes. However, a generation 1 line card inserted into a generation 2 chassis dumbs down the whole switch to generation 1 port indexes. So, with 252 indexes, 32 taken up by the FCIP card, only 240 port indexes are left which directly translates to 240 ports - less than half the switch capacity! If line cards are installed that take the port count above 252 then these additional cards won't come online - they will initially power up then power down.

In this instance the solution will be to move IP services to another (smaller) switch and hopefully Cisco will bring out a generation 2 version of the IP blade soon. There is a bigger problem though, and that is for any customers looking to take 9513s and use them with the SSM module, which supports products such as EMC Invista and Kashya (now EMC RecoverPoint). As far as I am aware, there's no plans for a generation 2 SSM module any time soon, so using the SSM module in the 9513 chassis will create the same port restriction issues.

I don't see why Cisco couldn't simply produce a gen 2 version of the old gen 1 blades which did nothing more than re-jig the port indexing. Come to think of it, surely they could patch the firmware on the gen1 line-cards to fix the problem. Obviously it is not that simple, or perhaps not that many people are using Invista and RecoverPoint to make it worthwhile.