Lehman Brothers hits the rocks

The latest casualty of the credit crunch is Lehman Brothers who have filed for Chapter 11 Bankrupcy protection. See the report here from the BBC.

I worked for Lehman in the UK for a short period about 4 years ago. If their banking operations was run anything like IT, then there's no surprise they are out of business today.

I joined as a consultant after the entire storage team left for other jobs and within 6 months moved on myself after the outsourcing Sword of Damocles hovered over the team. The storage operation in place was fundamentally flawed, however based on the attention Lehman's will be getting, this may not be the best time for me to expand on those issues in writing. :-)

Large Hadron Collider, Data, Sweden and Nuclear Rockets

The testing of CERN's Large Hadron Collider has been widely reported over the last few days and weeks in the media. You can find more details here.

In simplistic terms, the project is looking to re-create the conditions experienced in the Big Bang at the start of the universe in an attempt to detect elusive atomic sub-particles, which should help to solidify and prove the Standard Model.

There has been a lot of speculation that the testing could lead to the creation of a black hole and implosion of the earth and the universe. In fact, my son came home from school yesterday and told me the rumour in the playground was that Sweden were intending to launch two nuclear missiles at each other that would lead to the destruction of the earth!! Obviously, I had to bring his astrophysics knowledge up to scratch. I guess I shouldn't be too harsh on him - he's only 10.

Anyway, the interesting storage angle is in how much data the project will produce on an annual basis. According to this rather helpful PDF, the project will produce data at a rate of 700MB/s or 15 petabytes (PB) per year (presumably the collider will not be run 24 hours a day). That's a lot of data, especially when you consider the project will take 2-3 years to produce enough data to detect the elusive higgs-boson particles!

The collected data will be analysed by the LHC Computing Grid (see here for details). Managing this data, which will be distributed down 4 tiers will be an incredible job, mainly for the complexities of maintaining concurrency of access to the data and all the analysis results.

Let's hope it works!!

FC Enhancements

A comment posted to my previous blog entry reminds me of a requirement I've had for some time from Fibre Channel. In the "Good Old Days" in my first working life as a mainframe systems programmer, I could very easily see a breakdown of response time against each storage device on an LPAR. Now, the passing years may have given me "rose tinted spectacles" (or more accurately now, contact lenses) of that time, but I seem to remember the reason that I could see seek time, disconnect and connect time was due to the design of the (then) MVS I/O subsystem. As each I/O (CCW) was processed, the hardware must have been adding a consistent timestamp to each part of the process; the I/O initiation, the connect, the disconnect and subsequent seek and then the reconnect and data transfer time to complete the I/O (if none of this makes sense, don't worry, it probably means you are under 40 years old and never wore sandals to work).

Nowadays, the I/O infrastructure is a different kettle of fish. Each part of the infrastructure (host, HBA, fabric, array) are provided by different vendors and have no consistent time reference, therefore tracking the time to execute a storage "exchange" is very difficult. There is (as far as I am aware) nowhere within a fibre channel packet to track this response time at each stage of the journey from host to storage.

If we want the next generation of storage networks to scale, then without a doubt we need to be able to track the journey of the I/O at each stage and use this information to provide better I/O profiling.

Now, just how do I become a member of the t11 committee.....

Drobo Update

I’ve had my Drobo for a few months now. For those of you not familiar with the technology, the Drobo is a storage device from a company called Data Robotics. Follow the link above to their website for full details.

I’d been looking for a decent home/home office storage device for some time. RAID support was a must and initially I thought I wanted NAS because my solution at the time was to keep a server running continuously. The server performs other tasks and I was using it for file serving too.

Previously I had taken the plunge with the Linksys NSLU2 which runs a modified version of Linux. Unfortunately at the time, the device only supported ext3 filesystems and as I loaded the device with more data, responses became erratic and I found the exported systems going read-only and losing content. Lucky for me the problem seemed to be the device rather than the actual data on disk and I was able to recover everything using a little software utility which allowed me to read ext3 devices on Windows. This experience scared me and it was time to look for something else.

The Drobo hit the spot for a number of reasons; firstly it was a dedicated device which took SATA II drives. It has a USB connection, allowing me to plug it directly into my existing server and third (and at the time most importantly) Data Robotics had just released a NAS head which could be used with the standard Drobo, or removed without affecting the format of data on the device itself.

So, as I said, I’ve had it for a few months and what is there to say about it? Well, not a lot. It works – and so far has worked flawlessly. But there are a few things of note.

Firstly, I have a BIOS incompatibility issue; when my server reboots, if the Drobo is plugged into a USB port, it hangs the server. I haven’t bothered to resolve this yet; weighing up the relative merits of just living with this problem or upgrading the BIOS on my server, then I fall squarely on the side of accepting the workaround of unplugging the drive at boot time and plugging it back in as the system comes up. If I was using a standalone PC, then I would obviously have fixed the problem.

Second, I was interested to see that despite my system having two 1TB drives and RAID protection, the X: drive I’d created reported back a 2TB file system. Was RAID on or not? Well, yes it was; the Drobo presents a 2TB file system regardless of the drives you have installed. It’s virtualisation in action! As you allocate all of the physical storage available, you get prompted to add or swap drives to match the physical demand. I like this feature as it’s a painless way to upgrade your storage over time and as terabyte drives drop in price (currently I’ve seen them at 1TB for £99) it helps smooth out the cost of upgrade because drive sizes can be mixed and matched.

Last, there’s the issue of firmware upgrade. Version 1.1.1 of firmware is available and it was a simple task to upgrade, however I can’t implement the code without rebooting the Drobo and that requires closing all the active files accessing the Drobo on the server. This is not a major problem though and wouldn’t be a problem on a standalone PC.

All in all, the Drobo looks good and does the job. Having 1TB of new capacity has encouraged me to spend time moving my data over in a controlled and structured fashion. The process will take months (a subject I will return to), but in the meantime I have bags of spare capacity and an easy upgrade path for both additional capacity and NAS connectivity.

Now, if anyone out there would like me to review their NAS product, then I’d be only too happy….

Problems Problems

This week I've been working on two interesting (ish) problems. Well, one more interesting than the other, one a case of the vendor needing to think about requirements more.

Firstly, Tuning Manager (my old software nemesis) strikes again. Within Tuning Manager it is possible to track performance for all LUNs in an array. The gotcha I found this week is that the list of monitored LUNs represents only those allocated to hosts and is a static list which must be refreshed each time an allocation is performed!

This is just lack of thought on behalf of the developers not to provide a "track everything" option so it isn't necessary to keep going into the product, selecting the agent, refreshing the LUN list and tagging them all over again. No wonder allocations can take so long and be fraught with mistakes when Storage Admins have to include in their process the requirement to manually update the tuning product. I'm still waiting for confirmation that there isn't a way to automatically report on all LUNs. If there isn't then a product enhancement will be required to meet what I want. In the meantime, I'll have to ensure things are updated manually. So if you configured Tuning Manager and the LUN list when you first installed an array, have a quick look to see if you're monitoring everything or not.

I'm sure some of you out there will point out, with good reason, why HTnM doesn't automatically scan all LUNs, but from my perspective, I'm never asked by senior management to monitor a performance issue *before* it has occurred, so I always prefer to have monitoring enabled for all devices and all subsystems if it doesn't have an adverse affect on performance.

Second was an issue with the way NTFS works. A number of filesystems on our SQL Server machines show high levels of fragmentation, despite there being plenty of freespace on the volumes in question. This fragmentation issue seems to occur even when a volume is cleared and files are reallocated from scratch.

A quick trawl around the web found me various assertions that NTFS deliberately leaves file clusters between files in order to provide an initial bit of expansion. I'm not sure this is true as I can't find a trusted source to indicate this is standard behaviour. In addition I wonder if it the way in which some products allocate files; for instance if a SQL backup starts to create a backup file it has no real idea how big the file will become. NTFS (I assume) will choose the largest block of freespace available and allocate the file there. If another process allocates a file almost immediately, then it will get allocated just after the first file (which may only be a few clusters in size at this stage). Then the first file gets extended and "leapfrogs" the second file, and so on, producing fragmentation in both files.

I'm not sure if this is what is happening, but if this is the way NTFS is working then it would explain the levels of fragmentation we see (some files have 200,000+ fragments in a 24GB file). In addition, I don't know for definite that the fragmentation is having a detremental impact on performance (these are SAN connected LUNs). Everything is still speculation. I guess I need to do more investigation...

Long term data retention

I spent some time earlier this week talking to COPAN. They produce high density storage systems, but not the sort of arrays you'd use for traditional data storage. Their product is pitched at the long term persistent storage market.

I'm sure you can read the website if you're interested however, I hadn't really thought through what this kind of technology could deliver. There are some fundamental issues the storage of "persistent" data needs to be sorted. For instance; how do you validate the data on your disk will still be there when you come to read it 12 months later? (disk Aerobics is the answer apparently; regular validation of disk content).

So, the target market for COPAN is long term data archive. They want you to keep your data on disk. Personally, I think if the price is right, then backup data on disk is a sensible proposition. Today's network connectivity and encryption technology means data doesn't need to be physically moved any longer. In fact, I'd suggest that removing the need to physically move data is the way forward. Disk-based data is inherently more reliable and accessible. COPAN (and others) have plenty of features that can make disk-based backup work.

Don't move the media. Just move the data. Sounds like a good strapline.

Storage protocols for VMware

I've been doing more VMware work recently. The deployment I'm working on is using SAN presented disk. The storage started as 50GB LUNs, quickly grew to 100GB and now we're deploying on 200GB LUNs, using VMFS and placing multiple VM guests on each meta volume.

Now, this presents a number of problems. Firstly, it was clear the LUN sizes weren't big enough in the first place. Second, migrating guests to larger LUNs had to be an offline process; present the new LUNs, shutdown the guest, clone the guest, restart the guest, blow the old guest away. A time intensive process, especially if it has to be repeated regularly.

Using FC presented LUNs/metas also presents another problem; if we choose to use remote replication (TrueCopy/SRDF) to provide DR failover then all the VM guests on a meta have to go on that failover too. This may not (almost certainly not!) be practical.

Add in the issue with lack of true active/active multipathing and restrictions on the number of LUNs presentable to an ESX server and FC LUNs don't seem that compelling.

The options are to consider iSCSI or store data on CIFS/NFS. I'm not keen on the CIFS/NFS option, iSCSI seems more attractive. It pushes the storage management away from the ESX Server and onto the VM guest; security is managed at the array level, rather than within ESX. Personally I think this is preferable let ESX (system) administrators do their job etc etc. One last benefit; I can present as many iSCSI LUNs as I like of whatever size. It means I can also stripe multiple LUNs; something I'm unlikely to do on VMFS presented devices.

Therefore I think iSCSI could be a great option. Then I thought of one curve ball; what if I could do thin provisioning on FC? Here's the benefit. Imagine creating 20 VM guests on a server, all running Win2K3. Standard deployment is 10GB for the root/boot disk but I'm only actually using about 5. The remainder is left to allow for maintenance/patching/temporary space (we don't want to have to rebuild servers) - applications and data go on separate volumes. I'll use a 200GB meta. Unfortunately it's 50% wasted. But bring in thin provisioning and I can allocate 10GB drives with impunity. I can allocate 20 or 30 or 40! FC is back on the menu. Incidentally, I'm more than aware that iSCSI devices can already be presented thin provisioned.

Lots of people tell me, why bother with thin provisioning. I think in VMware I've found a perfect usage.

Buy Buy Buy

More acquisition news; Brocade and McDATA are finally just Brocade after the purchase completes. There's a snazzy new logo

which I can't decide is like a pair of red angel wings, or some convoluted join of the M and B from the two companies. Personally, I preferred the old logo depicting a fabric, it worked for me. I even had a lovely "personal" email from Michael Klayko (as I'm sure lots of other people did).

Anyway, moving on, what's more interesting is IBM bought SoftTek. Now they have a great product, TDMF, a piece of software that really made me go "wow!" when I first saw it in action. It literally is a transparent data mover. With the mainframe version (I was a proper storage admin when I started out on the mainframe) you could sync up many volumes in real time and perform an instant swap on them, allowing migration of TB of storage in seconds. I used the product to migrate data from an old storage array to new without any user impact or input at all. I hope IBM uses its new jewel well.

iSCSI Security Part 2

To tie down my iSCSI test environment I've implemented IPsec between client and server. This allows me to encrypt either the traffic or headers of my IP stream. I chose ESP (Encapsulating Security Payload) as this gives full confidentiality to my data, rather than Authentication Header (AH) which provides integrity as to the source of the data.

Implementation on my Netapp Simulator and Windows client was easy; the ipsec command, a file setting on the filer and configuration of the IPsec settings through MMC on Windows. Once I'd dropped and relogged in the iSCSI targets, I did some testing.

Now before I go further, let me stress that this testing was only slightly scientific. Everything I'm using is virtual and on the same physical machine with one hard disk. The figures I'm quoting are indicative of the performance I received and not directly comparable with real-world systems, although you'll get the idea.

So I tested with HDTach and no IPsec enabled. Response time for a standard test was about 0.2ms and throughput ranged from 2-32MB/s over the course of the test (an average of around 17MB/s). With IPsec enabled, response time doubled to 0.4ms and throughput dropped significantly to 2-7MB/s with an average of 4.2MB/s. I repeated the tests a number of times with IPsec on and off and saw consistent results.

Just to be sure, I checked for other bottlenecks in my system. The virtualising PC was not bottlenecked, neither the filer or client. I can therefore put the performance change down to simply "more work to do".

Unsurprisingly, I couldn't check the IP packets with IPsec enabled. Although this is secure, it presents issues when diagnosing problems. I don't know whether tools exist to enable encrypted streams to be analysed, but Ethereal couldn't do it. My IPsec implementation was also rather simple - just a shared key for encrypting the traffic. If I was implementing this for real, I'd be discussing with the Network guys the best way to implement security.

Brocade/McDATA Merge Approved

The Brocade purchase of McDATA has been approved by both shareholders. The expected completion of the merger is 29 January.

I'm interested to see the merged product lines and how BrocDATA intends to support both product sets (especially the director class devices). There will be a lot of customers out there looking to see what bridging technology the merged company will produce and how the roadmap will look.

Whatever happens, Mc-cade needs to come out with something quickly or Cisco will be in and mercilessly stealing their market share.

A confession

Nigel commented on my slow provisioning post. I have to agree that Device Manager has its faults. I'm doing some work on Device and Tuning Manager in the coming weeks as I'm installing both on VMware guests. So, I should have some performance feedback.

The comment highlights how software still has a long way to go.

A few (well quite a few) years ago I needed to purchase a storage management product for implementing storage quotas on the mainframe. For those who remember DFSMS, it was just at the time SMS as a product was being implemented on MVS/ESA.

So, I asked my boss if I could purchase the software. He said I needed to a competitive analysis with other products and explain why my choice was the best. Unfortunately there were no other equivalent products out there, so I pondered my options and decided to invent one.

Based on a character at the time, I called the product "Loads-a-space" after a character called "Loadsamoney" - see the picture. Loadsamoney was a crass plasterer who always boasted how much money he made, it was the 80's boom time. Loadsamoney was a creation of Harry Enfield, so I called my ficticious software house Enfield Software.

I added Loads-a-space from Enfield Software to my proposal as the inferior product, got my approval and my purchase. It helped that my boss was Canadian and didn't know much about British Comedy.

OK, now I've got that off my chest, Nigel, I'd be interested to know how big your Device Manager deployment is and what you run it on, just to do some kind of comparison

iSCSI Continued (2)

After my previous post on iSCSI testing I promised some more detail. So my test environment is based on the Netapp Simulator version 7.2.1, which you can download if you're a Netapp customer - not sure if it's available to non-customers (it should be as it is a great marketing tool) but I guess if you want to find out you could ask Dave.

Netapp filers export LUNs as iSCSI devices. As far as I can tell, the LUN implementation on ONTAP is effectively a qtree (i.e. a share) based on the way I created it. Anyway, once created, I associated the LUN with an initiator group and the initiator group has the access associated with it (hope you're all following this). The screenshot here shows the output from the LUN show command and igroup show command which list the LUNs I created and their association. You can see the igroup can be used to provide access to a number of servers based on the iqn, iSCSI Qualified Name, which is used to reference an iSCSI target or initiator device. The iqn seems to be a "gentlemens agreement" format, based on the reverse DNS of the server on which the iSCSI device resides, plus the date and month that domain was registered. In this case I registered the test server I have the iSCSI initator on and I got "iqn.1991-05.com.microsoft" followed by the specific server identifier of "vmware2.vmware.brookend.com", the name of my server itself.

My first inclination (being a hacker of old) was to spoof this, so I configured another server (XP this time) with the iSCSI initiator and changed its iqn. Voila I can access the same disks, albeit from another IP address. Being iSCSI and block data, the filer simulator didn't care about multiple access (which is fine) and I spent some time trying to break the shared LUN writing and reading data from both sources.

Needless to say this example highlighted how simple security does (or doesn't) work. I know CHAP authentication is available and I suspect there are many more security options that I need to investigate, so that's my next thing. Getting standards and security right I think will be more important than making sure the network performs.

More details to come.

Manic Miner and Storage Resource Management

I tried a bit of nostalgia the other day. From a "freebie" CD-ROM I installed a games emulator for the ZX Spectrum, a personal computer that was hugely popular in the '80s. The game I installed was called Manic Miner, one of the original platform games. At the time (1983) it was a classic and (shamefully) I even hacked the copy I had to remove the protection (you had to load a 4 or 6 digit code from a sheet of blue paper, which couldn't be photocopied). When my children saw the game, they fell about laughing, not surprising when you compare it to their latest play, Star Wars Battlefront.

It made me think how things have changed in 20 years; from 32x24 graphics to 1280x1024 with advanced polygon shading etc. What has this to do with storage? Well, I ponder on what will happen to Storage Resource Management in the next 20 years.

I think what we'll see is artificial intelligence-based software managing our data. The software will proactively fix hardware faults, relocate data based on our usage/value policies, provide CDP and CDR, deliver optimum performance and make all storage administrators obsolete.

Er, well all except the last one; yes I do think the worries we have about SRM tools will be resolved, however I think with the growth in capacity, complexity and features of todays storage, that Storage Administrators will be needed for a long time to come.

Tower of Tera

Lots of talk today about the 1 terabyte drive from Hitachi. In fact the drive is more likely to be about 931GB based on the dubious practice of using decimal 1000's rather than binary (whilst we're on that subject, the concept of decimal versus binary does annoy me - what with that and overhead, on some of the AMS's I've installed, a 300GB drive comes out as 267GB).

So, yes, I want a 1TB drive - no idea what I want to put on it, or how I'll back it up - but I want one.