I have a secondary 180GB FAT32 partition where I install everything which
is not related to the Windows system, ie. mainly games and most apps.

  Reading an article about system stability and gaming performance, I got
reminded that it's a good idea to defrag disks from time to time, as a
very fragmented file system can significantly decrease performance,
especially with games which read gigabytes of data at a time from the disk.

  Last time I defragmented this partition was something like 3 years ago,
so I checked the partition. It was very heavily fragmented. From the a bit
more than 90 GB of data in the disk, about half was fragmented, and most
of this data was very heavily fragmented (large files fragmented into many
small pieces). Most prominently, the huge data files of the last few games
I had installed were very fragmented, so a significant loss in performance
was a very real possibility.

  So I ran Windows' own defrag on the partition. After almost 15 hours (!)
of work it had defragged about 30%, and the defrag program had become so
resource-heavy and unstable that I had to shut down the entire system. In
fact, the whole system had become so unstable that I was unable to shut
down the computer and had to resort to a hard reset, after finally succeeding
in closing the defrag program. Luckily this didn't cause any errors in the
partition.

  I googled for a lighter and more efficient defragmenter with good
reputation, and tried that. It was definitely more resource-light than
Windows' own defrag, and was able to defrag everything safely without
consuming increasing amounts of memory nor making the system unstable.
Even then it took it about 20 hours (!!!) to defrag everything. And this
was a program which advertised itself to be faster than Windows' own defrag.

  I can't understand why defragmentation takes such humongous amounts of
time. There were something like 40 or 50 GB worth of fragmented data,
but if you calculate how many times you can copy 50 GB inside a modern
SATA disk, I think that in 35 hours you could copy it over and over
thousands of times. (Assuming it would take 1 minute to copy 50 GB of
data, you could copy it over 2000 times in 35 hours.)

  Why does it take such a humongous amount of time? I can't understand.

  I also noticed that both defragmenters failed to use the free space
in the partition (over 80 GB) to their advantage, and instead moving
small clusters around one at a time. I'm pretty certain that if all the
free available space was used to copy data, the whole process could be
a lot faster.

-- 
                                                          - Warp

Post a reply to this message

On Mon, 01 Jun 2009 22:07:26 +0200, Warp <war### [at] tagpovrayorg> wrote:
>   I have a secondary 180GB FAT32 partition

That would be the biggest problem right there. FAT32 is really horrible on  
partitions that large.


> Last time I defragmented this partition was something like 3 years ago

That would be problem number two.


If you absolutely must use FAT32 (compatibility with other operating  
systems being the only plausible reason), the fastest way to defrag is to  
simply copy everything to another partition (or a spare drive), reformat,  
and then copy everything back.



-- 
FE

Post a reply to this message

On Mon, 01 Jun 2009 22:46:49 +0200, Fredrik Eriksson wrote:

> On Mon, 01 Jun 2009 22:07:26 +0200, Warp <war### [at] tagpovrayorg> wrote:
>>   I have a secondary 180GB FAT32 partition
> 
> That would be the biggest problem right there. FAT32 is really horrible
> on partitions that large.
> 
> 
>> Last time I defragmented this partition was something like 3 years ago
> 
> That would be problem number two.

Concur with both of those ideas.  Thing to keep in mind as well is that 
(at least IIRC) defragmenters tend to bypass caching mechanisms, so you 
lose some performance there as well (compared to doing a file copy, for 
example).

Jim

Post a reply to this message

Fredrik Eriksson <fe79}--at--{yahoo}--dot--{com> wrote:
> On Mon, 01 Jun 2009 22:07:26 +0200, Warp <war### [at] tagpovrayorg> wrote:
> >   I have a secondary 180GB FAT32 partition

> That would be the biggest problem right there. FAT32 is really horrible on  
> partitions that large.

  Why would FAT32 become slow with partitions that large?

  And anyways, no matter how slow it is, it still doesn't explain why
defragmenting 50 GB takes 35 hours.

> If you absolutely must use FAT32 (compatibility with other operating  
> systems being the only plausible reason), the fastest way to defrag is to  
> simply copy everything to another partition (or a spare drive), reformat,  
> and then copy everything back.

  Why can't defragmenters do exactly that when there's over half of free
space in the partition?

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Luckily this didn't cause any errors in the partition.

That's part of the reason. There's a sync before and after each block of 
file movement, which makes for a tremendous amount of seeking. (Not each 
allocated cluster, but each read/write pair. I don't know how big a buffer 
it uses.) It also re-reads the free space bitmap for each file, so if you're 
moving lots of small files to make room for a big one, that might take a 
majority of the time right there. But it's all designed that you wont lose 
data if someone else opens the file while it's defragging (read or write) or 
if you pull the plug while it's going.

Plus, Windows defrags online, which means it has to keep the buffers up to 
date in memory and things like that, not stomp on blocks just allocated to 
another file, etc.  I.e., you can be reading through a file while it's 
defragging and not get errors from reading the wrong buffers and such.

Was this Vista or XP? Vista intentionally yields the disk up to other 
processes while defragging. I also think Vista has a problem with a really 
highly fragmented file. For example, I had a file with 300,000+ fragments 
that Vista just choked trying to defrag. I fell back to sysinternals contig 
(which works nicely and lets you do just individual files or directories 
too). So it's not the kernel bit broken, but the UI/app on top of it.

>   Why does it take such a humongous amount of time? I can't understand.

It *is* a lot faster if you have lots of room on the disk, IME. The Vista 
defragger seems much slower than even the XP defragger.

>   I also noticed that both defragmenters failed to use the free space
> in the partition (over 80 GB) to their advantage, and instead moving
> small clusters around one at a time. I'm pretty certain that if all the
> free available space was used to copy data, the whole process could be
> a lot faster.

That bit is in the kernel. You just say "move clusters X thru Y of this file 
into the free space starting at Z" and it moves it. The likelihood that a 
different version of the defragger will be significantly faster at 
defragging a bunch of files is low, if the bottleneck is the I/O, because 
they all use the same FCNTL calls.

http://technet.microsoft.com/en-us/sysinternals/bb897427.aspx

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Fredrik Eriksson wrote:
> On Mon, 01 Jun 2009 22:07:26 +0200, Warp <war### [at] tagpovrayorg> wrote:
>>   I have a secondary 180GB FAT32 partition
> 
> That would be the biggest problem right there. FAT32 is really horrible 
> on partitions that large.

Oh, I didn't even see that part. Yes, agreed - the free space map on a FAT32 
partition that big is huge, and at the wrong end of the disk. :-) To defrag 
a piece of file, it has to read the the directory entry, then the FAT to get 
the location of the file, then read the FAT to find a free space, allocate 
the free space, sync the FAT, copy the sectors, sync the sectors, change the 
directory, sync the directory, then re-read the FAT and fix the previously 
allocated sectors to be free, and sync the FAT.  Lather rinse repeat.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Warp wrote:
>   Why would FAT32 become slow with partitions that large?

The FAT table is pre-allocated.

>   And anyways, no matter how slow it is, it still doesn't explain why
> defragmenting 50 GB takes 35 hours.

Lots of seeks due to trying to make it reliable even when people like you 
pull the plug in the middle. ;-)

>   Why can't defragmenters do exactly that when there's over half of free
> space in the partition?

They can if they want. You're going to be even slower, tho, because you're 
going to be seeking back and forth between the directory, the FAT, and the 
data, and now you're seeking even farther.

-- 
   Darren New, San Diego CA, USA (PST)
   There's no CD like OCD, there's no CD I knoooow!

Post a reply to this message

Warp wrote:

>   I can't understand why defragmentation takes such humongous amounts of
> time. There were something like 40 or 50 GB worth of fragmented data,
> but if you calculate how many times you can copy 50 GB inside a modern
> SATA disk, I think that in 35 hours you could copy it over and over
> thousands of times. (Assuming it would take 1 minute to copy 50 GB of
> data, you could copy it over 2000 times in 35 hours.)
> 
>   Why does it take such a humongous amount of time? I can't understand.

Defragging is limited primarily by how fast you can thrash the heads 
back and forth, not by the maximum sequential transfer speed for the drive.

I bet solid-state drives would defrag a lot faster. (But then, why would 
you bother defragging a solid-state drive in the first place?)

Post a reply to this message

>>   I have a secondary 180GB FAT32 partition
> 
> That would be the biggest problem right there. FAT32 is really horrible 
> on partitions that large.

I wasn't aware it *goes* that large.

I vaguely recall reading somewhere that to make FAT work on large 
partitions, you make the cluster size larger or something weird like 
that, which is probably not very efficient...

Post a reply to this message

Question: Assuming fully defragmented data, which is faster to read data
from a few big files: FAT32 or NTFS?

  I tried goolging for this info, but couldn't find any definitive answer.

-- 
                                                          - Warp

Post a reply to this message