Fugitive Thought

How to properly extend a file backed ntfs volume.
OR: Extending NTFS while avoiding the dreaded "Disk read error"

If you have ever resized an NTFS storage volume (physical disk/partition, file-backed, LVM, VHD, vmdk, what have you) for a real machine, a virtual machine (xen, qemu, vmware, Virtual Server, etc.), you know the familiar process: extend the volume, delete and recreate the partition to desired size, and resize the filesystem within to fit the changed partition boundaries.** You may be finished at this point, and if so, you are the cause of envy for many others, for if the resized volume contained an NTFS partition, you may be victim to a dreaded message: 'A disk read error occurred. Press Ctrl+Alt+Del to restart'.

This is very frustrating, but the solution, while less than obvious, is not overly difficult.

First, a word of explanation. The "disk read error" can occur (and most often does occur) when a disk containing an NTFS partition is resized. This is, of course, our particular dilemma. When disk capacity is altered, the disk geometry reported by the BIOS to the bootloader/OS changes accordingly. If that alteration causes the disk capacity to cross one of several thresholds (528Mb, 1Gb, 2Gb, 4Gb), the geometry change must be reflected in the NTFS partition boot sector. Unfortunately, not all resizing tools address such a change, and therefore, they do not properly avoid the error at boot-time. Fortunately for most users, expanding/shinking the disk capacity tends to fall within or without the abovestated thresholds, avoiding the "disk read error" issue.

Specifically, the Windows master boot record (MBR) and NTFS bootloader use Cylinder-Head-Sector (aka CHS) addressing on disks up to 8 Gb, which relies upon physical disk geometry to function. At the time the NTFS partition is created, the disk geometry is stored in the BIOS parameter block (BPB) of the NTFS boot sector. This geometry has two parameters: the number of sectors/track and the number of heads. The latter is more important, for the value of the heads parameter supplied by the BIOS varies along with disk capacity. (E.G., A disk between 1Gb and 2Gb in size has 64 heads; between 2Gb and 4Gb, 128 heads; over 4Gb (8Gb and beyond), 255 heads.)

It is at this juncture that we discover our problem -- the BPB heads value is *hardcoded* upon creation of the NTFS filesystem. Resizing a volume entails an inherent geometry change, after which the data hardcoded in the BPB may not reflect the state of the disk geometry reported by the BIOS. With such a mismatch, the NTFS booloaded is unable to properly initialize, fails, and displays our favorite "disk read error."

Thankfully, we can avoid this issue by altering the values present in the NTFS BPB; replacing the MBR and/or boot sector is not necessary.

The region of interest in the BPB exists at the address 0x7E00, which corresponds with sector 63 (the start of the filesystem itself after the bootloader). At a slight offset from 0x7E00, address 0x7E1A describes the head count, and this is where we must concentrate our efforts.

Read on for specific, step-by-step details.

**shrinking the volume is of course slightly different and partly in the reverse order. We will here deal with the extending of a volume.

Starting image = 4Gb

# ls -alh 2k3.img
-rw-r--r-- 1 root root 4G 2007-08-07 10:21 2k3.img

Observe the partition table before starting. First, map image to a block device so we can inspect with fdisk

# kpartx 2k3.img -av
add map loop2p1 : 0 8289729 linear /dev/loop2 63

Now print the parition table

# fdisk -lu /dev/loop2

Disk /dev/loop2: 4244 MB, 4244636672 bytes
128 heads, 63 sectors/track, 1028 cylinders, total 8290306 sectors
Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System
/dev/loop2p1 * 63 8289791 4144864+ 7 HPFS/NTFS

Delete the mapping so we can alter the disk itself

# kpartx 2k3.img -dv
del devmap : loop2p1
loop deleted : /dev/loop2

Append 8Gb of blank space (all zeros(nulls)) to the image
If using a sparse image, dd if=/dev/zero seek=10G count=0 or similar

# dd if=/dev/zero bs=1M count=8192 >> 2k3.img
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 266.031 seconds, 32.3 MB/s

Observe that image has grown by 8Gb

# ls -alh 2k3.img
-rw-r--r-- 1 root root 12G 2007-08-09 17:47 2k3.img

Map image to a block device and expose the partition within.
In this case, /dev/loop2 is the block device and /dev/mapper/loop2p1 is the partition

# kpartx 2k3.img -av
add map loop2p1 : 0 8289729 linear /dev/loop2 63

Edit the partition table with fdisk
We perform the following fdisk commands

p print partition table
D Delete partition
p 1 new partition starting at same spot
t 7 sets type of new partition to (hpfs/ntfs)
a 1 sets bootable flag on first partition
w writes parititon table to disk. ignore errors about rereading it

# fdisk -u /dev/loop2

The number of cylinders for this disk is set to 3108.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/loop2: 12.8 GB, 12834571264 bytes
128 heads, 63 sectors/track, 3108 cylinders, total 25067522 sectors
Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System
/dev/loop2p1 * 63 8289791 4144864+ 7 HPFS/NTFS

Command (m for help):

Command (m for help): d
Selected partition 1

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p

Partition number (1-4): 1
First sector (63-25067521, default 63): 63
Last sector or +size or +sizeM or +sizeK (63-25067521, default 25067521): 25067521

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 7
Changed system type of partition 1 to 7 (HPFS/NTFS)

Command (m for help): a
Partition number (1-4): 1

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Partitioning is now done. we need to alert the kernel to the new partition table, so delete the mapping and add it again.
This causes the system to reread the partition table of the block device (which we just edited).

# kpartx 2k3.img -dv
del devmap : loop2p1
loop deleted : /dev/loop2

# kpartx 2k3.img -av
add map loop2p1 : 0 25067459 linear /dev/loop2 63

Check filesystem for errors and if possible fix them.
If you see any errors, stop and try ntfsfix, etc.; talk to your good friend google.
(Warnings are generally OK)

# ntfsresize -P -i -f -v /dev/mapper/loop2p1
ntfsresize v1.12.1 (libntfs 8:1:0)
Using locale 'en_US.UTF-8'.
Device name : /dev/mapper/loop2p1
NTFS volume version: 3.1
Cluster size : 2048 bytes
Current volume size: 4244339200 bytes (4245 MB)
Current device size: 12834539008 bytes (12835 MB)
Checking filesystem consistency ...
Accounting clusters ...
Space in use : 1890 MB (44.5%)
Collecting resizing constraints ...
Estimating smallest shrunken size supported ...
File feature Last used at By inode
$MFT : 985 MB 0
Multi-Record : 2143 MB 27
$MFTMirr : 1072 MB 1
Compressed : 1351 MB 7270
Sparse : 980 MB 7781
Ordinary : 1373 MB 3218
You might resize at 1889527808 bytes or 1890 MB (freeing 2355 MB).
Please make a test run using both the -n and -s options before real resizing!

Now simulate a resize.
Again, if you see any errors, stop and try ntfsfix, google, etc.
(Warnings are generally OK)

# ntfsresize -P --force --force /dev/mapper/loop2p1 --no-action
ntfsresize v1.12.1 (libntfs 8:1:0)
Device name : /dev/mapper/loop2p1
NTFS volume version: 3.1
Cluster size : 2048 bytes
Current volume size: 4244339200 bytes (4245 MB)
Current device size: 12834539008 bytes (12835 MB)
New volume size : 12834535936 bytes (12835 MB)
Checking filesystem consistency ...
Accounting clusters ...
Space in use : 1890 MB (44.5%)
Collecting resizing constraints ...
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
The read-only test run ended successfully.

Now resize for real

# ntfsresize -P --force --force /dev/mapper/loop2p1
ntfsresize v1.12.1 (libntfs 8:1:0)
Device name : /dev/mapper/loop2p1
NTFS volume version: 3.1
Cluster size : 2048 bytes
Current volume size: 4244339200 bytes (4245 MB)
Current device size: 12834539008 bytes (12835 MB)
New volume size : 12834535936 bytes (12835 MB)
Checking filesystem consistency ...
Accounting clusters ...
Space in use : 1890 MB (44.5%)
Collecting resizing constraints ...
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
Syncing device ...
Successfully resized NTFS on device '/dev/mapper/loop2p1'.

Resizing is now done; delete the mapping.

# kpartx 2k3.img -dv
del devmap : loop2p1
loop deleted : /dev/loop2

It is now time to start the windows virtual machine.
It may be that your version of ntfsresize was sufficiently advanced to avoid issues.
If so, please enjoy using windows with your newly resized partition.
If, however, you are not as fortunate and receive the dreaded message below, continue onward.

A disk read error occurred
Press Ctrl+Alt+Del to restart

Shut down the virtual machine if you have not already.

We must now correct the head count value in the NTFS BPB region of the disk volume.
We may hexedit the file directly, as this a raw disk image.
Alternatively, if working upon an LVM volume or physical disk, we may work upon the device itself.
By the same token of course, we can also use kpartx to map the image to a block device, which can be edited.

As noted above, the addresses of the BPB of interest are as follows:
0x7E00 is sector 63, the start of the filesystem itself after the bootloader
0x7E1A describes the head count

Open the file or device using hexedit (or any other equivalent hex editor).

# hexedit 2k3.img

Press enter and type 7E1A to jump to that address.
See below the region in question as viewed in hexedit:

00007DF0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 EB 52 90 4E 54 46 53 20 20 20 20 00 02 04 00 00 00 00 00 00 00 F8 00 00
.................R.NTFS .............

00007E18 3F 00 80 00 3F 00 00 00 00 00 00 00 80 00 80 00 BC 7F 7E 01 00 00 00 00 5A 52 05 00 00 00 00 00 88 FB 07 00 00 00 00 00
?...?.............~.....ZR..............

0x7E1A is here equal to 0x80; type FF to change it and by pressing ctrl-X, y, and then enter.

The region after patching

00007DF0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 EB 52 90 4E 54 46 53 20 20 20 20 00 02 04 00 00 00 00 00 00 00 F8 00 00
.................R.NTFS .............
00007E18 3F 00 FF 00 3F 00 00 00 00 00 00 00 80 00 80 00 BC 7F 7E 01 00 00 00 00 5A 52 05 00 00 00 00 00 88 FB 07 00 00 00 00 00
?...?.............~.....ZR..............

Our time of frustration has concluded! Start up your virtual machine and let chkdsk verify that all is well in the filesystem.
(Don't forget to unmap the block device of the image if you went that route).

We're finished!

NOTE: if scripting this resizing process, we may do so easily without the need for a graphical, interactive hex editor.
Using xxd, we can readily and easily alter the VM image file or block device.
The following patches the image/device at 0x7E1A with the correct value of 255 heads (= 0xFF in hex):

# echo "00007E18: 3F00FF" | xxd -r - 2k3.img

NOTE: need dd, kpartx (from multipath-tools), ntfsresize (from ntfsprogs)
ntfsresize version ntfsresize v1.12.1 (libntfs 8:1:0)
COULD use lomount, suggest not
NOTE: the following assumes a raw disk format. convert to this format before proceeding.