Who needs FCoE anyway?

I've been working on getting my "home SAN" into a usable configuration over the last few weeks. One hassle has been VMware (and I won't even mention Hyper-V again) and the support for fibre channel.

I guess thinking logically about it, VMware can't support every type of HBA out there and the line has to be drawn somewhere, but that meant my Qlogic and JNI cards were no use to me. Hurrah for Ebay, as I was able to pick up Emulex Lp9002L HBA cards for £10 each! I remember when these cards retailed at £600 or more.

Now I have two VMware instances accessing Clariion storage through McDATA switches and it all works perfectly.

That leads me on to a couple of thoughts. How many thousands of HBA cards are out there that have been ditched as servers are upgraded to the latest models? Most of them are perfectly servicable devices that will continue to give years of useful service, but "progress" to 4/8Gb fibre channel and FCoE dictates we must ditch these old devices and move on.

Why? It's not as if they cause a "green" issue - they're not power hungry or take up lots of space. I would also challenge anyone who can claim that they need more than 2Gb/s bandwidth on all but a small subset of their servers. (Just for the record, I see the case for using 4/8Gbs for large virtual server farms and the like as you've concentrated the I/O nicely and optimised resources)

So we need two things; (a) a central repository for returning old and unwanted HBAs (b) vendors to "open source" the code for their older HBA models to allow enthusiast programmers to develop drivers for the latest O/S releases.

If we can reduce the amount of IT waste, then I think this is a key strategy to any company claiming to be green in the world moving forward.

FCoE

Fibre Channel over Ethernet has been back on my radar recently, especially as it was touted again at Storage Networking World in Orlando last week. Unfortunately I wasn’t there and didn’t see for myself, although I was in Orlando the week before on vacation. I can imagine if I’d extended or moved the holiday to include SNW that I’d be none too popular with Mrs E and my sons.

Any hoo, I looked back over my blog and I first briefly mentioned FCoE back in April 2007, a whole 12 months ago. Now, we know 12 months is a long time in the storage world (in which time iSCSI will have claimed another 3000% market share, EMC will have purchased another 50,000 storage companies of various and dubious value, HDS will have released nothing and IBM will have developed 2 or 3 new technologies which won’t see the light of day until I’m dead and buried). I expect then that FCoE should have moved on somewhat and it appears it almost has. Products are being touted, for example, Emulex with the LP21000 CNA card (not an HBA card, please note the new acronym) and Cisco with their Nexus 5000 switch (plus others).

At this stage I don’t believe the FCoE protocol has been fully ratified as a standard. I have been spending some time wading reading through the FC-BB-5 project documentation on the T11 website, covering FCoE to understand exactly how the protocol works in more detail and how it can be compared to native fibre channel, iSCSI, iFCP and FCIP. In the words of Cilla, here’s a quick reminder on storage protocols in case you’d forgotten.

Fibre channel and the Fibre Channel Protocol (FCP) provide a lossless, packet based data transmission protocol for moving data between a host (initiator) and a storage device (target). FCP implements SCSI over fibre channel. To date, fibre channel has been implemented on dedicated hardware from vendors including Cisco and McDATA/Brocade. iSCSI uses TCP/IP to exchange data between a host and storage device using the SCSI protocol. It therefore includes the overhead of TCP/IP but provides for lossy and long distance connectivity. iFCP and FCIP are two implementations which encapsulate FCP in TCP/IP packets. FCIP extends an existing fibre channel SAN, whereas iFCP allows data to be routed between fibre channel SANs.

FCoE will sit alongside fibre channel and allow the transmission of FCP packets at the Ethernet layer, removing the need for TCP/IP (and effectively allowing TCP/IP and FCP packets to exist on the same Ethernet network).

So hurrah, we have another storage protocol available in our armoury and the storage vendors are telling us that this is good because we can converge our IP and storage networks into one and save a few hundred dollars per server on HBA cards and SAN ports. But is it all good? Years back, I looked at using IP over fibre channel as a way to remove network interface cards from servers. The aim was to remove the NICs used for backup and put that traffic across the SAN using IPFC. I never did it. Not because I couldn’t; I’m sure technically it would have worked, but rather because the idea scared the willies out of “the management” for two reasons (a) we had no idea of the impact of two traffic types going over the same physical network and (b) the Network Team would have “sent the boys round” to sort us out.

Will this be any different with FCoE? Will anyone really be 100% happy mixing traffic? Will the politics allow the Networks teams to own SAN traffic entirely? Let’s face it, in large environments I currently advocate the separation of host, tape and replication traffic to separate fibre channel fabrics. I can’t imagine reversing my position and going back to single consolidated networks.

So then, is FCoE going to be better in smaller environments where the consolidation is more practical? Well, if that’s the case, then surely that makes FCoE just another niche player to FC, just like iSCSI.

It’s early days yet. There are a million-and-one questions which need to be answered, not least of which will be how FCoE will interoperate with standard FC, how drivers will interact with the existing storage protocol stack on a server and how performance/throughput will be managed. Some of these issues have been answered, however this blog entry is already far too long and rambling to include a discussion on these points this time and I will save them for another time.

AoE/FCoE/iSCSI

Robin harris discusses AoE from Coraid. I looked at this last year (reminder) as I saw it as a great way to get a FC/iSCSI solution at a low cost. However, before everyone rips out their FC SANs and runs to put an Ethernet solution in place, take one step back and consider the issues. Fibre Channel is successful because it works; because it is reliable. FC switches have features such as non-blocking architecture, QOS, preferred path and so on which help to remove or eliminate throughput or performance issues. Would ATA over Ethernet (or for that matter as it seems to be a topic of the moment) FC over Ethernet provide for that level of switch point to point bandwidth guarantee?

Consider also your monitoring tools. Both Brocade and Cisco offer features to do traffic redirecting (e.g. SPAN ports) to easily analyse SAN traffic without putting TAPs in place. Will AoE and FCoE offer that?

Consider security. Will AoE provide the same level of security as FC?

Without a doubt, you get what you pay for; however you should only pay for what you *need*. If you are running a mission critical application FC is still the best option - interoperability is more widely tested; diagnostic tools are mature; the technology is reliable. I do think there's a place for AoE, iSCSI and FCoE, but use it in the wrong place and what you save in cost, you may pay for later in downtime.