 |
|
|
|
|
|
| |
| |
|
|
|
|
| |
| |
|
|
I have a secondary 180GB FAT32 partition where I install everything which
is not related to the Windows system, ie. mainly games and most apps.
Reading an article about system stability and gaming performance, I got
reminded that it's a good idea to defrag disks from time to time, as a
very fragmented file system can significantly decrease performance,
especially with games which read gigabytes of data at a time from the disk.
Last time I defragmented this partition was something like 3 years ago,
so I checked the partition. It was very heavily fragmented. From the a bit
more than 90 GB of data in the disk, about half was fragmented, and most
of this data was very heavily fragmented (large files fragmented into many
small pieces). Most prominently, the huge data files of the last few games
I had installed were very fragmented, so a significant loss in performance
was a very real possibility.
So I ran Windows' own defrag on the partition. After almost 15 hours (!)
of work it had defragged about 30%, and the defrag program had become so
resource-heavy and unstable that I had to shut down the entire system. In
fact, the whole system had become so unstable that I was unable to shut
down the computer and had to resort to a hard reset, after finally succeeding
in closing the defrag program. Luckily this didn't cause any errors in the
partition.
I googled for a lighter and more efficient defragmenter with good
reputation, and tried that. It was definitely more resource-light than
Windows' own defrag, and was able to defrag everything safely without
consuming increasing amounts of memory nor making the system unstable.
Even then it took it about 20 hours (!!!) to defrag everything. And this
was a program which advertised itself to be faster than Windows' own defrag.
I can't understand why defragmentation takes such humongous amounts of
time. There were something like 40 or 50 GB worth of fragmented data,
but if you calculate how many times you can copy 50 GB inside a modern
SATA disk, I think that in 35 hours you could copy it over and over
thousands of times. (Assuming it would take 1 minute to copy 50 GB of
data, you could copy it over 2000 times in 35 hours.)
Why does it take such a humongous amount of time? I can't understand.
I also noticed that both defragmenters failed to use the free space
in the partition (over 80 GB) to their advantage, and instead moving
small clusters around one at a time. I'm pretty certain that if all the
free available space was used to copy data, the whole process could be
a lot faster.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On Mon, 01 Jun 2009 22:07:26 +0200, Warp <war### [at] tag povrayorg> wrote:
> I have a secondary 180GB FAT32 partition
That would be the biggest problem right there. FAT32 is really horrible on
partitions that large.
> Last time I defragmented this partition was something like 3 years ago
That would be problem number two.
If you absolutely must use FAT32 (compatibility with other operating
systems being the only plausible reason), the fastest way to defrag is to
simply copy everything to another partition (or a spare drive), reformat,
and then copy everything back.
--
FE
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On Mon, 01 Jun 2009 22:46:49 +0200, Fredrik Eriksson wrote:
> On Mon, 01 Jun 2009 22:07:26 +0200, Warp <war### [at] tagpovrayorg> wrote:
>> I have a secondary 180GB FAT32 partition
>
> That would be the biggest problem right there. FAT32 is really horrible
> on partitions that large.
>
>
>> Last time I defragmented this partition was something like 3 years ago
>
> That would be problem number two.
Concur with both of those ideas. Thing to keep in mind as well is that
(at least IIRC) defragmenters tend to bypass caching mechanisms, so you
lose some performance there as well (compared to doing a file copy, for
example).
Jim
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Fredrik Eriksson <fe79}--at--{yahoo}--dot--{com> wrote:
> On Mon, 01 Jun 2009 22:07:26 +0200, Warp <war### [at] tagpovrayorg> wrote:
> > I have a secondary 180GB FAT32 partition
> That would be the biggest problem right there. FAT32 is really horrible on
> partitions that large.
Why would FAT32 become slow with partitions that large?
And anyways, no matter how slow it is, it still doesn't explain why
defragmenting 50 GB takes 35 hours.
> If you absolutely must use FAT32 (compatibility with other operating
> systems being the only plausible reason), the fastest way to defrag is to
> simply copy everything to another partition (or a spare drive), reformat,
> and then copy everything back.
Why can't defragmenters do exactly that when there's over half of free
space in the partition?
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> Luckily this didn't cause any errors in the partition.
That's part of the reason. There's a sync before and after each block of
file movement, which makes for a tremendous amount of seeking. (Not each
allocated cluster, but each read/write pair. I don't know how big a buffer
it uses.) It also re-reads the free space bitmap for each file, so if you're
moving lots of small files to make room for a big one, that might take a
majority of the time right there. But it's all designed that you wont lose
data if someone else opens the file while it's defragging (read or write) or
if you pull the plug while it's going.
Plus, Windows defrags online, which means it has to keep the buffers up to
date in memory and things like that, not stomp on blocks just allocated to
another file, etc. I.e., you can be reading through a file while it's
defragging and not get errors from reading the wrong buffers and such.
Was this Vista or XP? Vista intentionally yields the disk up to other
processes while defragging. I also think Vista has a problem with a really
highly fragmented file. For example, I had a file with 300,000+ fragments
that Vista just choked trying to defrag. I fell back to sysinternals contig
(which works nicely and lets you do just individual files or directories
too). So it's not the kernel bit broken, but the UI/app on top of it.
> Why does it take such a humongous amount of time? I can't understand.
It *is* a lot faster if you have lots of room on the disk, IME. The Vista
defragger seems much slower than even the XP defragger.
> I also noticed that both defragmenters failed to use the free space
> in the partition (over 80 GB) to their advantage, and instead moving
> small clusters around one at a time. I'm pretty certain that if all the
> free available space was used to copy data, the whole process could be
> a lot faster.
That bit is in the kernel. You just say "move clusters X thru Y of this file
into the free space starting at Z" and it moves it. The likelihood that a
different version of the defragger will be significantly faster at
defragging a bunch of files is low, if the bottleneck is the I/O, because
they all use the same FCNTL calls.
http://technet.microsoft.com/en-us/sysinternals/bb897427.aspx
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Fredrik Eriksson wrote:
> On Mon, 01 Jun 2009 22:07:26 +0200, Warp <war### [at] tagpovrayorg> wrote:
>> I have a secondary 180GB FAT32 partition
>
> That would be the biggest problem right there. FAT32 is really horrible
> on partitions that large.
Oh, I didn't even see that part. Yes, agreed - the free space map on a FAT32
partition that big is huge, and at the wrong end of the disk. :-) To defrag
a piece of file, it has to read the the directory entry, then the FAT to get
the location of the file, then read the FAT to find a free space, allocate
the free space, sync the FAT, copy the sectors, sync the sectors, change the
directory, sync the directory, then re-read the FAT and fix the previously
allocated sectors to be free, and sync the FAT. Lather rinse repeat.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> Why would FAT32 become slow with partitions that large?
The FAT table is pre-allocated.
> And anyways, no matter how slow it is, it still doesn't explain why
> defragmenting 50 GB takes 35 hours.
Lots of seeks due to trying to make it reliable even when people like you
pull the plug in the middle. ;-)
> Why can't defragmenters do exactly that when there's over half of free
> space in the partition?
They can if they want. You're going to be even slower, tho, because you're
going to be seeking back and forth between the directory, the FAT, and the
data, and now you're seeking even farther.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I can't understand why defragmentation takes such humongous amounts of
> time. There were something like 40 or 50 GB worth of fragmented data,
> but if you calculate how many times you can copy 50 GB inside a modern
> SATA disk, I think that in 35 hours you could copy it over and over
> thousands of times. (Assuming it would take 1 minute to copy 50 GB of
> data, you could copy it over 2000 times in 35 hours.)
>
> Why does it take such a humongous amount of time? I can't understand.
Defragging is limited primarily by how fast you can thrash the heads
back and forth, not by the maximum sequential transfer speed for the drive.
I bet solid-state drives would defrag a lot faster. (But then, why would
you bother defragging a solid-state drive in the first place?)
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> I have a secondary 180GB FAT32 partition
>
> That would be the biggest problem right there. FAT32 is really horrible
> on partitions that large.
I wasn't aware it *goes* that large.
I vaguely recall reading somewhere that to make FAT work on large
partitions, you make the cluster size larger or something weird like
that, which is probably not very efficient...
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Question: Assuming fully defragmented data, which is faster to read data
from a few big files: FAT32 or NTFS?
I tried goolging for this info, but couldn't find any definitive answer.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> Question: Assuming fully defragmented data, which is faster to read data
> from a few big files: FAT32 or NTFS?
If you're reading the entire file, I would guess it's roughly equal. If
you're seeking around a lot, I'd guess it's NTFS.
Why? FAT uses a linked list of clusters, with the first cluster pointed to
be the directory entry, as the allocated clusters of the file. (That's the
File Allocation Table: the linked lists.)
NTFS just stores the first cluster and number of clusters in the file table,
so calculating what cluster a given block of the file is on is probably
easier and requires fewer trips back to the start of the disk. I.e., you
don't have to read the disk to seek in the file.
You can even hear this if you have a floppy drive. Copy a 1 meg file to a
floppy, and you can hear the seek back to the FAT every handful of tracks
written.
This is just a guess, mind. I wouldn't think there's a whole lot of
difference on a machine with enough memory to hold the whole FAT in kernel
buffers.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Invisible wrote:
> Warp wrote:
>
>> I can't understand why defragmentation takes such humongous amounts of
>> time. There were something like 40 or 50 GB worth of fragmented data,
>> but if you calculate how many times you can copy 50 GB inside a modern
>> SATA disk, I think that in 35 hours you could copy it over and over
>> thousands of times. (Assuming it would take 1 minute to copy 50 GB of
>> data, you could copy it over 2000 times in 35 hours.)
>>
>> Why does it take such a humongous amount of time? I can't understand.
>
> Defragging is limited primarily by how fast you can thrash the heads
> back and forth, not by the maximum sequential transfer speed for the drive.
>
> I bet solid-state drives would defrag a lot faster. (But then, why would
> you bother defragging a solid-state drive in the first place?)
SSD also has limited "write" times, so, defragging the thing would end
up re-writing the data, and thus hastening the death of the media. Its
why most OSes that use them are "tweaked" so they never, if possible,
write the same file back to the same sectors each time, they way you
would on a HDD.
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
I don't understand the problems I'm getting defragmenting.
I'm trying to defragment my primary Windows partition, which is NTFS.
It's failing to defragment one big file because there's "no space" for it.
The file is about 1 GB in size, and there's over 9 GB of free space in the
partition.
The problem is that while defrag is defragmenting all the other files,
for some reason it's only compacting *some* of the files to be at the
beginning of the partition, but not others. The free space is scattered
with tiny small files which the defragmenter is not compacting, even though
it compacted other files just fine. The majority of the contents were nicely
packed to the beginning of the partition, except for these hundreds of tiny
files. And no, the tiny files are *not* marked as "unmovable". They are
marked with the same color as all the other movable files.
The other, lighter defragger is behaving the exact same way: It's just
refusing to move these small files, and consequently failing to defrag the
one huge file. No reason is given in either the visual representation of the
drive (all these tiny files are colored as regular, movable data) nor the
analysis reports for this.
As a consequence, I simply *can't* add any file bigger than a certain size
to this partition without it becoming fragmented. There isn't enough
contiguous free space because of all these scattered tiny files.
I can't understand why both programs are doing this.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> I'm trying to defragment my primary Windows partition, which is NTFS.
> It's failing to defragment one big file because there's "no space" for it.
> The file is about 1 GB in size, and there's over 9 GB of free space in the
> partition.
Buy a new hard drive with more free space? Seriously, lots of people fail
to factor in how much their time is worth when trying to fix stuff.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott <sco### [at] scottcom> wrote:
> > I'm trying to defragment my primary Windows partition, which is NTFS.
> > It's failing to defragment one big file because there's "no space" for it.
> > The file is about 1 GB in size, and there's over 9 GB of free space in the
> > partition.
> Buy a new hard drive with more free space? Seriously, lots of people fail
> to factor in how much their time is worth when trying to fix stuff.
So because Windows is unable to defragment a big file I should buy a
new hard drive? Exactly how is that going to help? It's not like a hard
drive is going to defragment the file.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> So because Windows is unable to defragment a big file I should buy a
> new hard drive?
If you really want the file defragmented, then yes, this might be the
cheapest way if you consider how much your time is worth.
> Exactly how is that going to help?
Well just the act of copying it over to another hard drive will probably
defragment it, and having more than 9GB of free space on a new hard drive
will likely make your life easier in terms of fragmented files in future.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott <sco### [at] scottcom> wrote:
> > Exactly how is that going to help?
> Well just the act of copying it over to another hard drive will probably
> defragment it
Copying what over? The entire Windows installation?
I'm not exactly sure XP survives such a transfer while still being fully
bootable from the new drive. Does it?
(I know from experience that a linux system can be fully transferred from
one disk to another with a simple "cp -ax", and it will then work identically
when the disks are swapped. However, I have got the impression that with
Windows it's not that easy.)
Anyways, it's just one fragmented file. It's not such a big deal. I'm just
wondering *why* the small files are not being compacted to the beginning of
the partition in order to create a larger contiguous empty space.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> Copying what over?
Just the 1GB file that is fragmented so badly. If the destination drive has
enough non-fragmented free space the file should be defragmented as it is
copied. If your original drive has enough non-fragmented free space you can
then just copy it back and it should be in a much better state. Note that
if this file is a windows system file you might need to do this outside of
windows for obvious reasons.
> I'm not exactly sure XP survives such a transfer while still being fully
> bootable from the new drive. Does it?
I don't see any big reasons why it wouldn't survive, you might have to
fiddle with a couple of things before it booted though (like MBR and the
boot.ini file). There are also lots of 3rd party programs that will do this
automatically for you, I'd be surprised if there wasn't a freeware one.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I don't understand the problems I'm getting defragmenting.
>
> I'm trying to defragment my primary Windows partition, which is NTFS.
> It's failing to defragment one big file because there's "no space" for it.
> The file is about 1 GB in size, and there's over 9 GB of free space in the
> partition.
>
> The problem is that while defrag is defragmenting all the other files,
> for some reason it's only compacting *some* of the files to be at the
> beginning of the partition, but not others. The free space is scattered
> with tiny small files which the defragmenter is not compacting, even though
> it compacted other files just fine. The majority of the contents were nicely
> packed to the beginning of the partition, except for these hundreds of tiny
> files. And no, the tiny files are *not* marked as "unmovable". They are
> marked with the same color as all the other movable files.
That's pretty random - and dissapointing.
A few things to consider:
1. I wonder if this is related to the MFT reservation area?
2. You might try using the "contig" tool free from System Internals. It
can be set to run during the boot sequence, so it has more or less
exclusive access to the disk.
3. Having a 1GB file split into (say) 3 fragments probably won't have
any measurable performance impact. It's when you have millions of 4KB
fragments that you have a problem.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott <sco### [at] scottcom> wrote:
> > Copying what over?
> Just the 1GB file that is fragmented so badly. If the destination drive has
> enough non-fragmented free space the file should be defragmented as it is
> copied. If your original drive has enough non-fragmented free space you can
> then just copy it back and it should be in a much better state. Note that
> if this file is a windows system file you might need to do this outside of
> windows for obvious reasons.
But that's the problem: There *isn't* enough contiguous space for the
large file to be stored unfragmented because there's a bunch of small files
scattered all over the free space, and the defragmenter is refusing to move
those files to make space, for an unknown reason.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> But that's the problem: There *isn't* enough contiguous space for the
> large file to be stored unfragmented because there's a bunch of small
> files
> scattered all over the free space, and the defragmenter is refusing to
> move
> those files to make space, for an unknown reason.
After you've moved off the 1GB file the defragmenter might behave
differently?
Anyway, even if there wasn't 1GB of contiguous free space, the file will
likely not get fragmented as much as it is currently, so it won't do any
harm.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott <sco### [at] scottcom> wrote:
> > But that's the problem: There *isn't* enough contiguous space for the
> > large file to be stored unfragmented because there's a bunch of small
> > files
> > scattered all over the free space, and the defragmenter is refusing to
> > move
> > those files to make space, for an unknown reason.
> After you've moved off the 1GB file the defragmenter might behave
> differently?
I did that the first time, and it didn't make any difference.
> Anyway, even if there wasn't 1GB of contiguous free space, the file will
> likely not get fragmented as much as it is currently, so it won't do any
> harm.
I just want to know why the defragmenter is refusing to move those files.
I would also like to make the page file one single contiguous block in the
disk. However, I'm unable to do that because of those small files.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> I just want to know why the defragmenter is refusing to move those files.
Did you try the command line version 'defrag'? On Vista at least, there
appears to be a -w option that "Performs full defragmentation. Attempts to
consolidate all file fragments, regardless of their size.". That might be
worth a shot, as it sounds from the wording that maybe the default is to
ignore small files or something?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I just want to know why the defragmenter is refusing to move those files.
That makes two of us.
> I would also like to make the page file one single contiguous block in the
> disk. However, I'm unable to do that because of those small files.
System Internals has a "PageDefrag" tool for this exact purpose.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Invisible wrote:
> System Internals has a "PageDefrag" tool for this exact purpose.
Norton's defragger, at least, used to have a 'defragment free space'
checkbox. No idea why that wasn't a default setting, or why almost no
defraggers nowadays do that (haven't checked Norton anything in ages
ever since around when they started shoehorning themselves into the
whole system vs. being a more-or-less standalone app/suite of apps).
--
Tim Cook
http://empyrean.freesitespace.net
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I can't understand why both programs are doing this.
Possibilities: The small files are actually directories that someone has
open? The defragger has seen that it already looked at those files and
decided it won't speed things up to move them (i.e., stupid algorithm)?
Most of the old defraggers had an option to consolidate free space, but
the later versions seem to have eliminated that. Hey, here's a business
opportunity for ya. ;-)
I assume you're smart enough to try it with everything you can run stopped.
Try this:
http://www.dirms.com/home/docs/dirms1.asp
The CLI version is free. It takes forever to run, especially with the
defaults. But it claims to compact the files better.
This is why *compacting* garbage collectors are good! ;-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I'm not exactly sure XP survives such a transfer while still being fully
> bootable from the new drive. Does it?
You can do it if you put the files in the same place, maintaining the
fragmentation. :-?
> (I know from experience that a linux system can be fully transferred from
> one disk to another with a simple "cp -ax",
That's nice about Linux. All you need to do is fix up the grub stuff,
rewriting the boot sectors or whatever grub does when you reinit it.
> Anyways, it's just one fragmented file. It's not such a big deal. I'm just
> wondering *why* the small files are not being compacted to the beginning of
> the partition in order to create a larger contiguous empty space.
Because the calculation is "how much time will I spend moving it compared to
the time I spend defragging it." NTFS keeps track of how often you use each
file, too, so the system can look and say "he only accesses this once a
month; there's no benefit to moving it closer to its directory."
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> I don't see any big reasons why it wouldn't survive,
You can't copy the registry files (especially SAM) easily while the system
is running, for one. Restoring it to the new drive is also tricky for the
same reason. Basically, you'd have to write your own full-blown program to
do this. You can't just use the usual backup/restore stuff.
> There are also lots of 3rd party programs that will do
> this automatically for you, I'd be surprised if there wasn't a freeware
> one.
The only ones I found that actually work basically take images of the drive,
not individual files, meaning fragmentation persists.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> Did you try the command line version 'defrag'? On Vista at least, there
> appears to be a -w option that "Performs full defragmentation. Attempts
> to consolidate all file fragments, regardless of their size.".
No. Vista defrag won't defrag chunks if they're already >64M or some such
size. I.e., you can have a file with three 66M chunks and defrag won't try
to defrag it without the -w flag. Again, the cost/benefit ratio.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New <dne### [at] sanrrcom> wrote:
> Try this:
> http://www.dirms.com/home/docs/dirms1.asp
No dice. Also that simply refuses to move those tiny files, and is unable
to defragment that one big file. I even tried to boot to safe mode and run
it from there. No banana.
My C drive looks like this, without that big file and without the Windows
page file:
http://warp.povusers.org/snaps/c-drive.png
I can't understand what those small files are. They are not marked as
unmovable nor anything special. They just are there, and no defragmenter
is moving them.
(The master file table certainly takes a humongous amount of space.
Is that really normal?)
Btw, this is how it looks after I restore that one big file:
http://warp.povusers.org/snaps/c-drive2.png
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> No dice. Also that simply refuses to move those tiny files, and is unable
> to defragment that one big file. I even tried to boot to safe mode and run
> it from there. No banana.
Weird. I never really let it grind all the way through, but it did
defragment some stuff for me.
> http://warp.povusers.org/snaps/c-drive.png
I think I have an older (or freer) version of that program. I don't have all
those options. :-)
> I can't understand what those small files are. They are not marked as
> unmovable nor anything special. They just are there, and no defragmenter
> is moving them.
My guess, offhand, would be either directories or small files the system
happens to have open for some reason.
Oh, wait... Do you have system restore turned on? I bet you can't move files
that are the backing store of virtual volumes. Try going into system on the
control panel and turning off system restore, and see if those files go away
after the system has had time to sync the drive.
Also, maybe consider scheduling a chkdsk on boot. Perhaps the blocks wound
up being marked used but aren't really used any more?
> (The master file table certainly takes a humongous amount of space.
> Is that really normal?)
It never gets any smaller. If you run chkdsk, it'll tell you how much of the
MFT space is actually in use. There's also "reserved but unallocated" space
at the end of the MFT (normally 1/8th your disk). You can put files there,
but the system will use that space last, because once your MFT is
fragmented, you can't defrag it.
> Btw, this is how it looks after I restore that one big file:
> http://warp.povusers.org/snaps/c-drive2.png
That's only 4 fragments or so. Doesn't seem too bad. But I can understand
the frustration of not being able to figure out what those files are.
I bet it wouldn't be *too* hard to write a program to tell you file names
and cluster numbers on stdout, so you could figure out which files they are.
(Actually, DriveXML will do that as part of backing up the disk, writing it
into the xml-formatted backup file, but then you need to actually back up
the disk to get that file using that program. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On Thu, 04 Jun 2009 21:33:21 +0200, Warp <war### [at] tagpovrayorg> wrote:
>
> My C drive looks like this, without that big file and without the
> Windows page file:
>
> http://warp.povusers.org/snaps/c-drive.png
>
> I can't understand what those small files are. They are not marked as
> unmovable nor anything special. They just are there, and no defragmenter
> is moving them.
Try analysing the drive with MyDefrag (http://www.mydefrag.com). It will
show you what file occupies any given part of the disk.
> (The master file table certainly takes a humongous amount of space.
> Is that really normal?)
The size of the MFT will of course depend on the amount of files on the
drive, but can easily be 100 MB or more.
Also, keep in mind that Windows will by default reserve up to an eighth of
the drive for the MFT to grow into. Not all defraggers distinguish between
the two parts.
--
FE
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Fredrik Eriksson wrote:
> Try analysing the drive with MyDefrag (http://www.mydefrag.com). It will
> show you what file occupies any given part of the disk.
Cool. He needs some SEO in there. I never found that, even when I was
looking for better defraggers. :-)
> The size of the MFT will of course depend on the amount of files on the
> drive, but can easily be 100 MB or more.
I think the allocated space is 1KB for each file, for the maximum number of
files you've ever allocated. I.e., 1KB per file, but it never shrinks.
> Also, keep in mind that Windows will by default reserve up to an eighth
> of the drive for the MFT to grow into.
Actually, it's at least 1/8th, and you can bumpt it up as high as 50% when
you first format the drive if you want. In case you know you'll have lots of
tiny files.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Here's some immovable files:
System restore files
Control panel->system->system restore
turn it off for now
USN journals
fsutil usn queryjournal c:
write down max size, allocation delta.
fsutil usn deletejournal /d c:
Frees the space. Takes a minute or so, actually.
Sync/checkdisk/reboot/whatever
Defrag (and pagedfrg if you're on xp)
Then turn back on system restore and
fsutil usn createjournal m=maxvalue a=alocdelta c:
See if that gets rid of your bits. Plus disk cleanup, of course.
Also, perhaps you at one time recycled a whooole lot of files, such that
your subdirectory under the recycle bin is especially large?
cd c:\$RECYCLE.BIN
dir /a
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> Why does it take such a humongous amount of time? I can't understand.
Speaking of which, on a slightly different topic...
When I boot and auto-login, it takes about 3 minutes before everything is
all settled down and happy.
If I log out, then log back in again, I'm back within maybe 5 seconds,
because everything's already in RAM and such.
I bet the thing would boot way faster if it just loaded the entire
executable for everything it needs to load at the time the program is
launched, rather than letting demand paging fight it out. I have enough RAM
to comfortably hold everything I run, so why not? I guess that'll get put
in eventually. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I don't understand the problems I'm getting defragmenting.
>
> I'm trying to defragment my primary Windows partition, which is NTFS.
> It's failing to defragment one big file because there's "no space" for it.
> The file is about 1 GB in size, and there's over 9 GB of free space in the
> partition.
>
> The problem is that while defrag is defragmenting all the other files,
> for some reason it's only compacting *some* of the files to be at the
> beginning of the partition, but not others. The free space is scattered
> with tiny small files which the defragmenter is not compacting, even though
> it compacted other files just fine. The majority of the contents were nicely
> packed to the beginning of the partition, except for these hundreds of tiny
> files. And no, the tiny files are *not* marked as "unmovable". They are
> marked with the same color as all the other movable files.
>
> The other, lighter defragger is behaving the exact same way: It's just
> refusing to move these small files, and consequently failing to defrag the
> one huge file. No reason is given in either the visual representation of the
> drive (all these tiny files are colored as regular, movable data) nor the
> analysis reports for this.
>
> As a consequence, I simply *can't* add any file bigger than a certain size
> to this partition without it becoming fragmented. There isn't enough
> contiguous free space because of all these scattered tiny files.
>
> I can't understand why both programs are doing this.
>
The Norton one, which you "used to be able to get as a separate
utility", fracking Norton..., would rearrange things so that a) commonly
used files where all in the same space, things in the same directories
where in the same space, actual directory data was located near the
start of the disk, so it was accessed faster, and compacted "all" empty
space, so that your existing files where at the start of the drive, not
scattered all over the damn place. Then they came out with Win95, and
some files got "locked" so you couldn't move them at all, never mind how
fracking fragmented they got/made things, but it "still" used smart
rearranging on everything else. Now.. I have no idea what it actually
does, or it even technically available, especially as something other
than part of 3-4 other things you don't need, and slow the system, do to
them running in the background all the time.
I personally find it incomprehensible that nothing else anyone makes
seems to be able to do anything as "simple" as compressing used space to
the start of the drive, so your "unused" space is contiguous, never mind
being "smart enough" to defrag that big 1G file, even without "enough
room", by arranging other fragments to give you the space, instead of
just leaving them empty. Its like handing a moron a paint brush and
coming back to find themselves sitting on the top of the kitchen counter
whining, "I painted myself into a corner!", only.. there are sections of
floor, all in between them and the door, none big enough to jump to or
step on, but never the less "unpainted", which they planned to "get back
to later". :p And, that the MS defrag takes the idiot amount of time it
does to paint itself into that same corner... Obviously mentally
challenged, "as well as" stupid. lol
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> Anyway, even if there wasn't 1GB of contiguous free space, the file will
>> likely not get fragmented as much as it is currently, so it won't do any
>> harm.
>
> I just want to know why the defragmenter is refusing to move those files.
> I would also like to make the page file one single contiguous block in the
> disk. However, I'm unable to do that because of those small files.
>
Simplest answer is, it behaved that way in 3.11, when it was "even more
critical", and MS never bothered to steal the idea of fixing it from the
few companies whose products **did** do it. ;)
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Patrick Elliott wrote:
> Simplest answer is, it behaved that way in 3.11, when it was "even more
> critical",
Huh? Did MS make a defragmenter for Win3.11? I thought it was just the dos
defrag? All the third-party defrags I remember using did a fine job of
defragmenting absolutely everything.
> and MS never bothered to steal the idea of fixing it from the
> few companies whose products **did** do it. ;)
Cite? Certainly the XP defragmenter moves files around to open up free
spaces. I suspect it's just files that have absolute addresses in them,
like the UNC and system restore points.
Hey Warp, if you get it defragged, let us know what it was? :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Patrick Elliott wrote:
> I personally find it incomprehensible that nothing else anyone makes
Everyone uses the same API. If there's a reason the files can't be moved, no
program is going to be able to move them.
> seems to be able to do anything as "simple" as compressing used space
Clearly it isn't that simple.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> Did you try the command line version 'defrag'? On Vista at least, there
>> appears to be a -w option that "Performs full defragmentation. Attempts
>> to consolidate all file fragments, regardless of their size.".
>
> No. Vista defrag won't defrag chunks if they're already >64M or some such
> size. I.e., you can have a file with three 66M chunks and defrag won't try
> to defrag it without the -w flag. Again, the cost/benefit ratio.
Ah ok, I checked on my XP box and there is a -f flag, to force defrag even
with low freespace - maybe this would help Warp?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> I don't see any big reasons why it wouldn't survive,
>
> You can't copy the registry files (especially SAM) easily while the system
> is running, for one. Restoring it to the new drive is also tricky for the
> same reason.
Why can't you just copy all the files while the OS isn't running? eg from
another PC/OS/LiveCD/boot disc?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Fredrik Eriksson <fe79}--at--{yahoo}--dot--{com> wrote:
> Try analysing the drive with MyDefrag (http://www.mydefrag.com).
That program actually did the trick. It moved those tiny files to make
space. Now my C drive layout looks like this, *with* that huge file and
*with* the Windows page file:
http://warp.povusers.org/snaps/c-drive3.png
(AFAIK those gaps are intentionally left by MyDefrag. The idea is that
programs creating small temp files will create them in those gaps, and
since the gaps are close to the beginning of the disk, they will be faster
to access.)
So what were those tiny files which other defraggers refused to move?
I don't know. MyDefrag gave weird names to them. I don't remember exactly
now, but they were something like C:\$SOMETHING:$SOMETHINGE_ELSE
I don't remember the exact names, but they looked meaningful. Maybe they
were somehow related to the NTFS file system.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New <dne### [at] sanrrcom> wrote:
> Everyone uses the same API. If there's a reason the files can't be moved, no
> program is going to be able to move them.
MyDefrag was able to move those strange clusters, even though it uses the
regular defragmenter system API to do so...
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> Ah ok, I checked on my XP box and there is a -f flag, to force defrag
> even with low freespace - maybe this would help Warp?
I don't think that'll help either. If he doesn't have a 1G hole on the disk,
and he can't move the files that are in the way, no number of flags is going
to compact the files. The -f flag doesn't affect that - defrag runs or it
doesn't.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> Why can't you just copy all the files while the OS isn't running? eg
> from another PC/OS/LiveCD/boot disc?
You could if you have the permissions, of course. You just can't copy the
drive you booted off of because of the registry. Most specifically, you
can't copy *to* the drive you booted off of.
There are a few immovable files (like the one holding the boot code, the
MFT, etc). There are a couple types of unmovable files (like the system
restore checkpoints, which have coded in their data the cluster numbers they
represent). There are permission systems designed to keep you from writing
to the file that holds the security information. You can't use the normal
access mechanisms (i.e., "copy") to copy the registry files around because
of security concerns.
It's just much more complicated than Linux. It's not impossible, or nobody
would be able to back up their computer. But the only software I've found
that actually will make a complete backup and then *restore* it uses imaging
techniques, not per-file copying.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I don't remember the exact names, but they looked meaningful. Maybe they
> were somehow related to the NTFS file system.
Anything with a $ at the front is a file in NTFS used to run the file
system. Everything on NTFS is actually in a file, so you have things like
$RECYCLE.BIN (not really a magic file, tho)
$MFT (basically, inodes)
$BOOT (boot file the boot sector points to - the "superblock")
$BITMAP (the free space bitmap)
$USN (unique serial number file - no ext3 equivalent I know of)
$JOURNAL (transaction journal)
You also have files for the keys used for encryption and security
descriptors. (In later NTFS versions, each different ACL is hashed into its
place in the security descriptor file and a pointer to it is stored in the
file. Apparently a space saving over storing the full ACL with each file.)
Stuff with a : in the name means you have a different fork of the file. Kind
of like how System7 had data forks and resource forks on Mac, except each
fork is a byte stream and you can have as many as you want. Forks that start
with a $ are used by MS, either the file system or the shell.
Without knowing the actual name, it would be hard to guess what they were.
Glad you got it fixed, tho.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> I bet the thing would boot way faster if it just loaded the entire
> executable for everything it needs to load at the time the program is
> launched, rather than letting demand paging fight it out. I have enough
> RAM
> to comfortably hold everything I run, so why not? I guess that'll get put
> in eventually. :-)
On Linux I have things set up so that my whole desktop environment is
preloaded into RAM right after the login manager is started.
There is a tool that records file reads. You configure it to run, then
reboot, and login, maybe open some apps you usually use. Next time you
reboot, another tool will read the recorded log and use the readahead
syscall on them. The file list is even sorted by their physical location on
the disk.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Nicolas Alvarez wrote:
> There is a tool that records file reads. You configure it to run, then
> reboot, and login, maybe open some apps you usually use. Next time you
> reboot, another tool will read the recorded log and use the readahead
> syscall on them. The file list is even sorted by their physical location on
> the disk.
Yeah, XP has that built in, and Vista to an even greater extent. But they
only work on (AFAIU) the demand-paged blocks, not entire files. Does the
Linux tool read an entire file, or just the parts that were paged in at
start-up? I suspect 5 seconds would be plenty to read all the login files if
the heads didn't thrash like mad.
It just seems like it could be done better. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> Nicolas Alvarez wrote:
>> There is a tool that records file reads. You configure it to run, then
>> reboot, and login, maybe open some apps you usually use. Next time you
>> reboot, another tool will read the recorded log and use the readahead
>> syscall on them. The file list is even sorted by their physical location
>> on the disk.
>
> Yeah, XP has that built in, and Vista to an even greater extent. But they
> only work on (AFAIU) the demand-paged blocks, not entire files. Does the
> Linux tool read an entire file, or just the parts that were paged in at
> start-up? I suspect 5 seconds would be plenty to read all the login files
> if the heads didn't thrash like mad.
>
> It just seems like it could be done better. :-)
The command-line tools work in terms of whole files. The syscall takes a
file descriptor and a byte range.
readahead-watch(8): Watches the mount points given, or the root filesystem,
for files opened during the boot sequence and closed without writing to
them. Orders the list by on-disk position and then writes it to the output
file.
readahead-list(8): Loads lists from the given file and performs readahead(2)
on each entry.
readahead(2): populates the page cache with data from a file so that
subsequent reads from that file will not block on disk I/O.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Nicolas Alvarez wrote:
> The command-line tools work in terms of whole files. The syscall takes a
> file descriptor and a byte range.
>
> readahead-watch(8): Watches the mount points given, or the root filesystem,
> for files opened during the boot sequence and closed without writing to
> them. Orders the list by on-disk position and then writes it to the output
> file.
>
> readahead-list(8): Loads lists from the given file and performs readahead(2)
> on each entry.
>
> readahead(2): populates the page cache with data from a file so that
> subsequent reads from that file will not block on disk I/O.
Yeah! That's exactly what I need on Vista. :-) Especially if it would read
the whole file in one blow, or at least each contiguous piece. The vista
mechanisms just don't really do the trick, I think.
Actually, with the USN, you wouldn't even need readahead-watch. Just parse
the USN journal since the last boot and build the list from that. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
|
|
