Do You Really Need a SAN - Of Course You Do!

The wacky boys at Forrester have a great new article posted relating to the requirement to have a Storage Area Network. Here's a link to their post. Tony Asaro and Hu Yoshida have both posted on the subject already but I couldn't resist having my 2 cents.

SANs evolved for very good reasons; the need to consolidate storage, the need to provide additional connectivity to storage arrays and the need to remove the requirement to closely couple the storage to the server (remember 25m limits on SCSI cabling). SANs and most notably fibre channel, enable that (as does iSCSI for the record).

Some of the Forrester objections to deploying SAN include;

Low Capacity Utilization - I did some work a couple of weeks ago at a client with purely DAS . They had 30% utilisation on their disks and after excluding boot drives, still had 30% utilisation. I've never seen a SAN array only 30% full, unless it was a new array onto which data was being deployed.

Inability to Prioritize Application Performance - Hmm, this seems a bit odd. DMX has Dynamic Cache Partitioning, Optimiser, USP has I/O prioritisation, Compellent can dynamically move data between tiers of storage; 3PAR has similar features which allow performance to be tweaked dynamically. There's also the same options in the fabric, especially with Cisco equipment. DAS has no such benefit, in fact if you have performance issues on a single server then you're potentially in a world of pain to fix it.

Long Provisioning Times - this is not a technology issue but one of process. I can provision terabytes of storage in minutes, however I have to guarantee that within a shared environment I don't take down another production environment - that's the nature of shared systems. In addition, users think they can just demand more and more storage without any consequences. Storage resources are finite - even more so in a non-scalable DAS solution. With sensible process, SAN storage can be turned around in hours - not the case for DAS unless you intend keeping spare disks onsite (at a price).

Soaring Costs - again, another conundrum. If you focus on pure hardware then SANs are inevitably more expensive, however TCO for storage is very rarely done. Don't forget SAN also includes iSCSI, which can be implemented across any IP hardware - hardly expensive.

So, there are other benefits that SAN easily wins over DAS.

Disaster Recovery. SAN-based replication is essential in large environments where the requirement to manually recover each server would be totally impractical. Imagine trying to recover 100+ database servers where each server requires the DBA to log in and perform forward recovery of shipped logs - all in a 2 hour recovery window.

Storage Tiering. SANs allow easy access to multiple storage tiers, either within the same array or across multiple arrays. Without SAN, tiering would be wasteful, as most servers would not be able to utilise multiple tiers fully.

SANs also provide high scalability and availability, simply not achievable with DAS.

There was a reason we moved to SAN. SAN has delivered, despite what Forrester say. However like all technologies, they need to be managed correctly. With sensible planning, standards and process, SANs knock DAS into a cocked hat.

FCoE

Fibre Channel over Ethernet has been back on my radar recently, especially as it was touted again at Storage Networking World in Orlando last week. Unfortunately I wasn’t there and didn’t see for myself, although I was in Orlando the week before on vacation. I can imagine if I’d extended or moved the holiday to include SNW that I’d be none too popular with Mrs E and my sons.

Any hoo, I looked back over my blog and I first briefly mentioned FCoE back in April 2007, a whole 12 months ago. Now, we know 12 months is a long time in the storage world (in which time iSCSI will have claimed another 3000% market share, EMC will have purchased another 50,000 storage companies of various and dubious value, HDS will have released nothing and IBM will have developed 2 or 3 new technologies which won’t see the light of day until I’m dead and buried). I expect then that FCoE should have moved on somewhat and it appears it almost has. Products are being touted, for example, Emulex with the LP21000 CNA card (not an HBA card, please note the new acronym) and Cisco with their Nexus 5000 switch (plus others).

At this stage I don’t believe the FCoE protocol has been fully ratified as a standard. I have been spending some time wading reading through the FC-BB-5 project documentation on the T11 website, covering FCoE to understand exactly how the protocol works in more detail and how it can be compared to native fibre channel, iSCSI, iFCP and FCIP. In the words of Cilla, here’s a quick reminder on storage protocols in case you’d forgotten.

Fibre channel and the Fibre Channel Protocol (FCP) provide a lossless, packet based data transmission protocol for moving data between a host (initiator) and a storage device (target). FCP implements SCSI over fibre channel. To date, fibre channel has been implemented on dedicated hardware from vendors including Cisco and McDATA/Brocade. iSCSI uses TCP/IP to exchange data between a host and storage device using the SCSI protocol. It therefore includes the overhead of TCP/IP but provides for lossy and long distance connectivity. iFCP and FCIP are two implementations which encapsulate FCP in TCP/IP packets. FCIP extends an existing fibre channel SAN, whereas iFCP allows data to be routed between fibre channel SANs.

FCoE will sit alongside fibre channel and allow the transmission of FCP packets at the Ethernet layer, removing the need for TCP/IP (and effectively allowing TCP/IP and FCP packets to exist on the same Ethernet network).

So hurrah, we have another storage protocol available in our armoury and the storage vendors are telling us that this is good because we can converge our IP and storage networks into one and save a few hundred dollars per server on HBA cards and SAN ports. But is it all good? Years back, I looked at using IP over fibre channel as a way to remove network interface cards from servers. The aim was to remove the NICs used for backup and put that traffic across the SAN using IPFC. I never did it. Not because I couldn’t; I’m sure technically it would have worked, but rather because the idea scared the willies out of “the management” for two reasons (a) we had no idea of the impact of two traffic types going over the same physical network and (b) the Network Team would have “sent the boys round” to sort us out.

Will this be any different with FCoE? Will anyone really be 100% happy mixing traffic? Will the politics allow the Networks teams to own SAN traffic entirely? Let’s face it, in large environments I currently advocate the separation of host, tape and replication traffic to separate fibre channel fabrics. I can’t imagine reversing my position and going back to single consolidated networks.

So then, is FCoE going to be better in smaller environments where the consolidation is more practical? Well, if that’s the case, then surely that makes FCoE just another niche player to FC, just like iSCSI.

It’s early days yet. There are a million-and-one questions which need to be answered, not least of which will be how FCoE will interoperate with standard FC, how drivers will interact with the existing storage protocol stack on a server and how performance/throughput will be managed. Some of these issues have been answered, however this blog entry is already far too long and rambling to include a discussion on these points this time and I will save them for another time.

Using that 8Gbps

Symantec/Veritas announced this week the release of Netbackup 6.5. It seems to me like the company has been talking about this version of the product forever and as is mentioned on other blogs, there are lots of new features to play with.



The ones I've been looking forward to are called SAN Media Server and SAN Client. These allow the SAN to be used (via FC/SCSI) as the transport method for backup data between the client and the media server.



For Symantec, implementing this feature is more difficult than it sounds; A standard HBA operates as an initiator with a storage device (disk or tape) acting as the target. I/O operations are instigated by the initiator to the target which then replies with the results. With the SAN Media Server, Symantec have had to re-write the firmware on the HBAs in the media server to act as a SCSI target. This allows it to receive backup data from the client, from when it is then handled as normal by the media server.



This option has a huge number of potential benefits; If you have already provided a SAN connection to a host (and possibly a production IP connection), there's no need now to provide a backup LAN connection too; just use the SAN. On new installations, this could save a fortune in network ports and would use a lot of potentially wasted fibre channel bandwidth. Throughput could be vastly improved; most fabrics run at 2Gbps today compared to host 1Gbps (OK, the data has to be read from the disk too, but most hosts have two HBA cards, so 4Gbps of aggregate bandwidth). Fabrics tend to be more well structured and therefore suffer less with bottleneck issues.

The only downside I can see is the need to provide one honkin' great media server to suck up all that backup data!

Certification Nightmare

One of the things I've always found a real issue in heterogeneous environments is that of certification. By certification I mean confirming that an installation is vendor supported. This is no mean feat in today's storage world. There are multiple layers - storage array, fabric, HBA, O/S, logical volume manager and multipathing software. There are also issues of virtualisation, product co-existence (e.g mixing switches from the same vendor but different OEM vendors), management tools, replication software and applications.



The stack of products and levels that must be considered is huge, especially if each layer has multiple products. It is also true that certain vendors don't help this situation (a) when new software is required to support a new array (for example) and it affects all other layers of the stack (b) competing vendors don't fully support their competitor's technology until well after the release date.



I've not yet seen a suitable solution for tracking certification levels. Firstly, even the vendors can't document their own products in a consistent fashion. Consider Emulex - querying the driver level on a Windows host returns a string which *contains* the version, but isn't in a consistent format. The returned string is different for Emulex on Solaris.



I think we've reached the point where we need a Certification Authority. Someone needs to take control and get all vendors to map their products to a consistent standard. Actually, I'd like to do it and I'm working on a markup language to help. It wouldn't be difficult. Probably the most troublesome part would be assigning an index code to each version/product/level that vendors produce - and getting them to use them....

By the way, in case anyone suggests SAN Advisor or Onaro SANScreen, I don't think either of these products are up to the mark. I've seen SAN Advisor demonstrated a number of times and it looked like a generation 1 product. SANScreen was interesting but not a certification tool as such - more like a semantic rather than syntactic storage product.

Port Oversubscription

Following on from snig’s post, I promised a blog on FC switch oversubscription. It’s been on my list for some time and I have discussed it before, however it has also a subject I’ve discussed with clients from a financial perspective and here’s why; most people look at the cost of a fibre channel switch on a per port basis, regardless of the underlying feature/functionality of that port.

Not that long ago, switches from companies such as McDATA (remember them? :-) ) provided full non-blocking architecture. That is, they allowed full 2Gb/s for any and all ports, point to point. As we moved to 4Gb/s, it was clear that Cisco, McDATA and Brocade couldn’t manage (or didn’t want) to deliver full port speed as the port density of blades increased. I suspect there were be issues with ASIC cost and fitting the hardware onto blades and cooling it (although Brocade have just about managed it).

For example, on Generation 2 Cisco 9513 switches, the bandwidth per port module is an aggregate 48Gb/s. This is regardless of the port count (12, 24 or 48), so, although a 48 port blade can (theoretically) have all ports set to 4Gb/s, the ports on average only have 1Gb/s of bandwidth.

However the configuration is more complex; ports are grouped into port groups, 4 groups per blade of 12Gb/s each, putting even more restriction on the ability to use available bandwidth across all ports. Ports can be dedicated or use shared bandwidth within a port group. In a port group of 12 ports, set three to dedicated bandwidth of 4Gb/s and the rest are (literally) unusable. Whilst I worked at a recent client, we challenged this option with Cisco. As a consequence, 3.1(1) of SAN-OS allows the disabling of all restrictions so you can take the risk and set all ports to 4Gb/s and then you’re on your own.

How much should you pay for these ports? What are they actually worth? Should a 48-port line card port cost the same as a 24-port line card port? Or, should they be rated on bandwidth? Some customers choose to use 24-port line cards for storage connections or even the 4-port 10Gb/s cards for ISLs. I think they are pointless. Cisco 9513’s are big beasts; they eat a lot of power and need a lot of cooling. Why wouldn’t you want to cram as many ports into a chassis as possible?

The answer is to look at a new model for port allocation. Move away from the concepts of core-edge and edge-core-edge and mix storage ports and host ports on the same switch and where possible within the same port group. This would minimise the impact of moving off-blade, off-switch or even out of port group.

How much should you pay for these ports? I’d prefer to work out a price per Gb/s. From the prices I’ve seen, that makes Brocade way cheaper than Cisco.

Storage as a commodity

I just read a comment over at Zerowait regarding Netapp and proprietary hardware. It reminded me of something I was thinking about recently on the commoditisation of storage.

There's nothing worse to my mind than a storage vendor who has no competition. Inevitably in some organisations that situation can exist when a single supplier is chosen to supply (for example) switches, SAN or NAS. The difficulty though is how to avoid that situation. Most vendors would love to lock you into their proprietary tools and relating back to the above article link, Netapp is one I see who try that more than anyone. They have a bewildering array of interlinked product options; once your hooked (especially where you use a feature to retain long term backups via snapshot/vaults) then you're sucked into a dependency on their products which just isn't healthy.

What's the solution? Well, for me I like to commoditise storage functionality. Pick out those features which all vendors support and only use the proprietary features where absolutely necessary. At least then you can maintain multiple vendors all on the hook for your next piece of business.

Of course implementing commoditised storage is more difficult than just picking a few common product features. However as far as your users are concerned, a LUN is a LUN and NAS storage is NAS storage, with a few caveats on things like driver levels for HBAs and so on.

I've previously posted a modular storage comparison sheet. As an example, here are some of the features that almost all support:

• RAID 5 protection
• Consistent LUN size
• dual pathing
• Active/Passive failover
• remote replication
• Fibre Channel presentation
• SNMP alerting
• online code upgrades
• hot swappable components

Before I get lots of comments saying "hold on, not all modular products are the same"; remember I'm not saying that. What I am saying is having a consistent set of requirements allows you to maintain a shortlist of vendors who can all enjoy the healthy competition of bidding for business. So, time to draw up a NAS spreadsheet....

Brocade/McDATA Merge Approved

The Brocade purchase of McDATA has been approved by both shareholders. The expected completion of the merger is 29 January.

I'm interested to see the merged product lines and how BrocDATA intends to support both product sets (especially the director class devices). There will be a lot of customers out there looking to see what bridging technology the merged company will produce and how the roadmap will look.

Whatever happens, Mc-cade needs to come out with something quickly or Cisco will be in and mercilessly stealing their market share.

iSCSI Continued (2)

After my previous post on iSCSI testing I promised some more detail. So my test environment is based on the Netapp Simulator version 7.2.1, which you can download if you're a Netapp customer - not sure if it's available to non-customers (it should be as it is a great marketing tool) but I guess if you want to find out you could ask Dave.

Netapp filers export LUNs as iSCSI devices. As far as I can tell, the LUN implementation on ONTAP is effectively a qtree (i.e. a share) based on the way I created it. Anyway, once created, I associated the LUN with an initiator group and the initiator group has the access associated with it (hope you're all following this). The screenshot here shows the output from the LUN show command and igroup show command which list the LUNs I created and their association. You can see the igroup can be used to provide access to a number of servers based on the iqn, iSCSI Qualified Name, which is used to reference an iSCSI target or initiator device. The iqn seems to be a "gentlemens agreement" format, based on the reverse DNS of the server on which the iSCSI device resides, plus the date and month that domain was registered. In this case I registered the test server I have the iSCSI initator on and I got "iqn.1991-05.com.microsoft" followed by the specific server identifier of "vmware2.vmware.brookend.com", the name of my server itself.

My first inclination (being a hacker of old) was to spoof this, so I configured another server (XP this time) with the iSCSI initiator and changed its iqn. Voila I can access the same disks, albeit from another IP address. Being iSCSI and block data, the filer simulator didn't care about multiple access (which is fine) and I spent some time trying to break the shared LUN writing and reading data from both sources.

Needless to say this example highlighted how simple security does (or doesn't) work. I know CHAP authentication is available and I suspect there are many more security options that I need to investigate, so that's my next thing. Getting standards and security right I think will be more important than making sure the network performs.

More details to come.