Thanks for the info, for that was quite a mouthful, and thanks for your mighty contributions to FLOSS. When do you estimate (what kernel release) that ext4 will be considered ready for production systems?

Note that tst_extents was intended as a debugging tool, not as an officially supported interface. So it may very well change without warning. It’s basically an interface to the extent functions in libext2fs which I used when I was debugging it. I would not recommend trying to create a program based on interacting with tst_extents and trying to parse its output. You’re much better off writing C program that links with libext2fs directly, or using SWIG to link it into Python or Perl.

The extents information is stored in a tree structure, where the root node of the tree is located in the inode and stores at most 4 entries. Said entries can either be pointers to single blocks that can hold 340 entries assuming a 4k blocksize, or if the entry is a leaf entry, it will be a pointer to a file’s extents. In practice, ext4’s anti-fragmentation allocation algorithms are good enough that most of the time, 4 entries in the inode is more than enough to cover most normally sized files, or for big files or if things are fragmented, a single external block. Very rarely (as in at the moment I have a single such file on my filesystem) more than 4 blocks are needed, at which point a depth two extent tree might be needed. It’s usually not because the filesystem’s free space is fragmented, but because the file is sparse, such that the discontinuities are in the logical block numbers, not the physical block numbers.

Finally, note that the “all” command to tst_extents will walk the entire extent tree, which is far more convenient that just entering the ‘n’ command over and over.

I think I’m using tst_extents correctly:
1.) open /path/to/fs
2.) inode /path/to/vid.avi
3.) root… gives

(Left 0)
extent: lblk 0–179014, len 179015, pblk 66879488, flags: (none)

4.) n… gives

[deleted because wordpress snips it, important part is (Left 6)]

So, if I keep hitting ‘n’ for this particular file I find 7 extents and indeed there is a break in there which would explain filefrag reporting 2. I have a couple questions though:

1.) Is the root an extent too (which would make 8 total)? Or some kind of directory to the other ones? Why does the root report “Left 0″? If I use the procedure above (open->inode->root->n) and just parse “(Left X)”, is the number of extents used always X+1 or are there special cases?

2.) Is there a way to use tst_extents not interactively other than feeding commands through stdin?

Thanks,
Avery

I think I’m using tst_extents correctly:
1.) open /path/to/fs
2.) inode /path/to/vid.avi
3.) root… gives

(Left 0)
extent: lblk 0–179014, len 179015, pblk 66879488, flags: (none)

4.) n… gives

<<<inode->root->n) and just parse “(Left X)”, is the number of extents used always X+1 or are there special cases?

2.) Is there a way to use tst_extents not interactively other than feeding commands through stdin?

Thanks,
Avery

So if you were to look at vid.avi using a tool such as tst_extents, you would no doubt see a series of 128 meg extents, since that is the maximum number of blocks that can be encoded in the extents structure used is indeed 128 metabytes (assuming a 4k blocksize). But ext4 will try very hard to keep block allocations contiguous, even between ext4 extent encodings, such that if extent number #2 ended with block N, extent #3 will begin with N+1 if at all possible. Apparently, in your vid.avi file, this was true for all but one case, so filefrag reported two contiguous block ranges, which it reported as “2 extents”.

I hope that clarifies things!

I’m finally trying out ext4 and especially seeing how fragmentation is vs ext3. I’m running into some output that I don’t understand from filefrag:

# filefrag vid.avi
vid.avi: 2 extents found
# ls -l vid.avi
-rw-r–r– 1 avery avery 733243392 2008-08-12 02:01 vid.avi

Shouldn’t the minimum number of extents be 6 based on 128MB extents?

I found this bug which is similar and supposed to be fixed:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=458306

I thought I would post here before filing another one to make sure I’m not missing something obvious.

Unfortunately, the fast fsck times won’t show up if you convert the filesystem from ext3. Sorry, I should have made this clear.

New files will benefit from extents and delayed allocation, so you’ll see run-time performance by moving to ext4. However, the improvements in fsck time stem from changes in how the bitmap and inode tables are laid out (which require a fresh mkfs -t ext4; a conversion from ext3 won’t help that) and the use of extents instead of indirect blocks to take a huge amount of time out of pass 1 seeking to deal with the indirect blocks requires that all of your files larger than 48k (assuming 4k block files) are rewritten using extents. We do have code that will be available in the future to “migrate” files using indirect blocks to extents, but it still won’t be as good as a filesystem which is freshly made with ext4.

Finally, not all of the instructions on converting ext3 filesystems tell you to run tune2fs -O uninit_bg /dev/XXX; e2fsck /dev/XXX. This will also reduce the fsck time, but the first fsck run is really annoying since you have to individually answer “Y” to all of the questions about setting the group checksums. After that, e2fsck will be able to skip inode table blocks that are completely empty during pass 1. In the long run I need to improve e2fsprogs to make this step not quite so annoying, but even then, you’ll get the best performance (and the best anti-fragmentation resistance, et. al), by doing a backup, recreating the filesystem using mke2fs -t ext4, and then a restore.

I’ll also note just out of a sense of fair warning, that there’s an additional block and allocation layout change which I am planning which should make improvements in ext4’s fsck and fragmentation resistance. Essentially it’s a change in where ext4 decides to allocate blocks for directories, segregating them from blocks for regular inodes. As before, it’s fully backwards compatible, but to see the full results there will be the need to do a backup, re-mkfs, and restore. The changes I’m contemplating will be probably only a tiny incremental improvement (if it’s 5% better, I’ll be ecstatic), so it’s probably not a reason to justify a full rebuild, but I thought I would mention it. Since it won’t get done in time for the 2.6.29 merge window, it won’t see fruition in a stable release for six months (i.e., when 2.6.30 is released). So it’s probably not worth waiting for, since the improvements between the ext3 layout and the current ext4 layout and block allocation algorithms are a factor of 7 or so, plus or minus.

I’m having trouble having those promised fast fsck times. I’m running Debian’s 2.6.27 on Sid (early January). The filesystem has been converted from ext3 some weeks ago.