The Defrag Debate

I was asked again this week whether defragging of hard drives on Windows servers is really necessary. This is quite pertinent as the cost of deploying an enterprise-wide defrag tool can be significant and any opportunity to save money has to be a good one.

I discussed fragmentation last year (here) when looking into a problem which turned out to be the lack of free space consolidation. however I didn't discuss the potential performance impact.

So, reflecting on whether defragmentation is required or not, I'd say that in most cases the benefits are minimal. Here's why...

Firstly, hard drives have on-board cache; the larger and more "enterprise" the drive, then the larger the cache. We are also likely to see more and more hybrid drives on the market, which will have large amounts of fast memory fronting access to the old moveable components. Cache will mask the impact of fragmented files during writes as data will be written to cache and confirmed as written by the drive. The data can then be written to disk asynchronously afterwards. Obviously if cache becomes overrun, then the benefit will be negated.

Second, operating systems have built in file system performance techniques, including lazy writing and file prefetch. These features will try and minimise the latency issues of reading and writing to disk.

Third, if the data being accessed is a database, the database itself will also have lazy writer processes to asynchronously write data to disk.

Now all of the above applies directly to "traditional hard drives". In systems which have RAID controllers with onboard cache, then the issues will be less. Where storage is taken from a SAN with an enterprise or modular array, all reads and writes will occur through cache, giving the best options for performance and masking the underlying hard drive mechanics.

So, can fragmentation actually help? In fact, I have seen one instance when this occurred and it required slightly special circumstances. The file system was made up from a number of concatenated LUNs using a volume manager. As the server had multiple CPUs and the storage was SAN connected, then multiple I/Os could be issued to the volume. With more fragmentation, then the I/Os are spread across all of the LUNs and performance was increased.

NTFS Update

I did some more work on my NTFS issue on Friday. As previously mentioned, I was seeing NTFS filesystems with large levels of fragmentation even after drives were compressed.

The answer turns out to be quite simple; Windows doesn't consolidate the free space blocks which accumulate as files are created and deleted. So, as a test I started with a blank 10GB volume and created a large file on it. Sure enough the allocation occurs in a small (2 or 3) number of extents. I then deleted the large file and created 10,000 small (5K) files and deleted those too. I then re-created the large file, which immediately was allocated in 100's of small fragments and needed defragmentation immediately. The large file was created using the freespace blocks freed up from the small files.

What's not clear from the standard fragmentation tool provided with Windows is that the free space created by the deletion of files is added to a chain of free space blocks. These free space blocks are never consolidated even if they are contiguous (i.e. as in this instance where I deleted all the files on the disk). This means even if you *delete* everything on a volume, then the free space is still fragmented and files will be created with instant fragmentation. The other thing to note is that the standard Windows defragmenter doesn't attempt to consolidate those segments when a drive is defragmented, it simply ensures that files are re-allocated contiguously. It also doesn't report that fact either.

I'm currently downloading Diskeeper, which allegedly does consolidate free space. I'm going to trial this and see how it affects my fragmentation problem.

Incidentally, I used one of Sysinternals' free tools to look at the map of my test drive. Sysinternals were bought by Microsoft in the summer of 2006, however you can find their free tools here. I used Diskview to give me a map of the drive and understand what was happening as I created and deleted files. What I would like, however is a tool which displays the status of free space fragments. I haven't found one of those yet.

So, now I have an answer, I just have to determine whether I think fragmentation causes any kind of performance issue on SAN-presented disks!