NTFS Update

I did some more work on my NTFS issue on Friday. As previously mentioned, I was seeing NTFS filesystems with large levels of fragmentation even after drives were compressed.

The answer turns out to be quite simple; Windows doesn't consolidate the free space blocks which accumulate as files are created and deleted. So, as a test I started with a blank 10GB volume and created a large file on it. Sure enough the allocation occurs in a small (2 or 3) number of extents. I then deleted the large file and created 10,000 small (5K) files and deleted those too. I then re-created the large file, which immediately was allocated in 100's of small fragments and needed defragmentation immediately. The large file was created using the freespace blocks freed up from the small files.

What's not clear from the standard fragmentation tool provided with Windows is that the free space created by the deletion of files is added to a chain of free space blocks. These free space blocks are never consolidated even if they are contiguous (i.e. as in this instance where I deleted all the files on the disk). This means even if you *delete* everything on a volume, then the free space is still fragmented and files will be created with instant fragmentation. The other thing to note is that the standard Windows defragmenter doesn't attempt to consolidate those segments when a drive is defragmented, it simply ensures that files are re-allocated contiguously. It also doesn't report that fact either.

I'm currently downloading Diskeeper, which allegedly does consolidate free space. I'm going to trial this and see how it affects my fragmentation problem.

Incidentally, I used one of Sysinternals' free tools to look at the map of my test drive. Sysinternals were bought by Microsoft in the summer of 2006, however you can find their free tools here. I used Diskview to give me a map of the drive and understand what was happening as I created and deleted files. What I would like, however is a tool which displays the status of free space fragments. I haven't found one of those yet.

So, now I have an answer, I just have to determine whether I think fragmentation causes any kind of performance issue on SAN-presented disks!

iSCSI Security Part 1

I've done some more work on iSCSI. That meant rebuilding one of my iSCSI client environments as I totalled it by trying to add another drive to the VM guest. Anyway, panic over. I've got the client working again and I've been running traces to see how easy it is to locate relevant data in an unprotected iSCSI stream.

I mentioned before that I am using iSCSI on Win2K3 talking to a Netapp filer (simulator) with Ethereal on the host Win2K3 box to monitor I/O activity. I tried today to write to a test file with Ethereal running and tracing I/O activity. The tool is splendid in its interpretation of the data it sees, formatting the packets on the basis of the content. My save of a file containing some test data yielded some interesting results.

Firstly, I found Ethereal detected the logfile entries for NTFS being saved before my actual file. These are RCRD and RSTR records which NTFS uses to recover the filesystem should anything untoward happen before the data is actually committed to disk.

After this, I tracked the MFT entries being written. These are the actual file saves which contain my data. Non-encrypted I can see the content. Using Ethereal and a tool to map NTFS records, I could easily spy on data being stored on iSCSI volumes. Here are some screenshots:

The first shows the MFT entry that NFS stores for the file save. I used Winhex to examine the blocks on disk. Note the content of the file "The Quick Red.......", plus the file name "TestFile1.txt" in unicode, so using double characters. Contrast this to the Ethereal output which was scanning the network. This shows the same data captured by scanning the network. Ethereal is generous enough to even format the data to allow me to locate the iSCSI data. This is no good for any organisation which must offer data security. Part II will discuss what can be done.