 |
|
|
|
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Invisible wrote:
> Warp wrote:
>
>> I can't understand why defragmentation takes such humongous amounts of
>> time. There were something like 40 or 50 GB worth of fragmented data,
>> but if you calculate how many times you can copy 50 GB inside a modern
>> SATA disk, I think that in 35 hours you could copy it over and over
>> thousands of times. (Assuming it would take 1 minute to copy 50 GB of
>> data, you could copy it over 2000 times in 35 hours.)
>>
>> Why does it take such a humongous amount of time? I can't understand.
>
> Defragging is limited primarily by how fast you can thrash the heads
> back and forth, not by the maximum sequential transfer speed for the drive.
>
> I bet solid-state drives would defrag a lot faster. (But then, why would
> you bother defragging a solid-state drive in the first place?)
SSD also has limited "write" times, so, defragging the thing would end
up re-writing the data, and thus hastening the death of the media. Its
why most OSes that use them are "tweaked" so they never, if possible,
write the same file back to the same sectors each time, they way you
would on a HDD.
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
I don't understand the problems I'm getting defragmenting.
I'm trying to defragment my primary Windows partition, which is NTFS.
It's failing to defragment one big file because there's "no space" for it.
The file is about 1 GB in size, and there's over 9 GB of free space in the
partition.
The problem is that while defrag is defragmenting all the other files,
for some reason it's only compacting *some* of the files to be at the
beginning of the partition, but not others. The free space is scattered
with tiny small files which the defragmenter is not compacting, even though
it compacted other files just fine. The majority of the contents were nicely
packed to the beginning of the partition, except for these hundreds of tiny
files. And no, the tiny files are *not* marked as "unmovable". They are
marked with the same color as all the other movable files.
The other, lighter defragger is behaving the exact same way: It's just
refusing to move these small files, and consequently failing to defrag the
one huge file. No reason is given in either the visual representation of the
drive (all these tiny files are colored as regular, movable data) nor the
analysis reports for this.
As a consequence, I simply *can't* add any file bigger than a certain size
to this partition without it becoming fragmented. There isn't enough
contiguous free space because of all these scattered tiny files.
I can't understand why both programs are doing this.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> I'm trying to defragment my primary Windows partition, which is NTFS.
> It's failing to defragment one big file because there's "no space" for it.
> The file is about 1 GB in size, and there's over 9 GB of free space in the
> partition.
Buy a new hard drive with more free space? Seriously, lots of people fail
to factor in how much their time is worth when trying to fix stuff.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott <sco### [at] scott com> wrote:
> > I'm trying to defragment my primary Windows partition, which is NTFS.
> > It's failing to defragment one big file because there's "no space" for it.
> > The file is about 1 GB in size, and there's over 9 GB of free space in the
> > partition.
> Buy a new hard drive with more free space? Seriously, lots of people fail
> to factor in how much their time is worth when trying to fix stuff.
So because Windows is unable to defragment a big file I should buy a
new hard drive? Exactly how is that going to help? It's not like a hard
drive is going to defragment the file.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> So because Windows is unable to defragment a big file I should buy a
> new hard drive?
If you really want the file defragmented, then yes, this might be the
cheapest way if you consider how much your time is worth.
> Exactly how is that going to help?
Well just the act of copying it over to another hard drive will probably
defragment it, and having more than 9GB of free space on a new hard drive
will likely make your life easier in terms of fragmented files in future.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott <sco### [at] scottcom> wrote:
> > Exactly how is that going to help?
> Well just the act of copying it over to another hard drive will probably
> defragment it
Copying what over? The entire Windows installation?
I'm not exactly sure XP survives such a transfer while still being fully
bootable from the new drive. Does it?
(I know from experience that a linux system can be fully transferred from
one disk to another with a simple "cp -ax", and it will then work identically
when the disks are swapped. However, I have got the impression that with
Windows it's not that easy.)
Anyways, it's just one fragmented file. It's not such a big deal. I'm just
wondering *why* the small files are not being compacted to the beginning of
the partition in order to create a larger contiguous empty space.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> Copying what over?
Just the 1GB file that is fragmented so badly. If the destination drive has
enough non-fragmented free space the file should be defragmented as it is
copied. If your original drive has enough non-fragmented free space you can
then just copy it back and it should be in a much better state. Note that
if this file is a windows system file you might need to do this outside of
windows for obvious reasons.
> I'm not exactly sure XP survives such a transfer while still being fully
> bootable from the new drive. Does it?
I don't see any big reasons why it wouldn't survive, you might have to
fiddle with a couple of things before it booted though (like MBR and the
boot.ini file). There are also lots of 3rd party programs that will do this
automatically for you, I'd be surprised if there wasn't a freeware one.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I don't understand the problems I'm getting defragmenting.
>
> I'm trying to defragment my primary Windows partition, which is NTFS.
> It's failing to defragment one big file because there's "no space" for it.
> The file is about 1 GB in size, and there's over 9 GB of free space in the
> partition.
>
> The problem is that while defrag is defragmenting all the other files,
> for some reason it's only compacting *some* of the files to be at the
> beginning of the partition, but not others. The free space is scattered
> with tiny small files which the defragmenter is not compacting, even though
> it compacted other files just fine. The majority of the contents were nicely
> packed to the beginning of the partition, except for these hundreds of tiny
> files. And no, the tiny files are *not* marked as "unmovable". They are
> marked with the same color as all the other movable files.
That's pretty random - and dissapointing.
A few things to consider:
1. I wonder if this is related to the MFT reservation area?
2. You might try using the "contig" tool free from System Internals. It
can be set to run during the boot sequence, so it has more or less
exclusive access to the disk.
3. Having a 1GB file split into (say) 3 fragments probably won't have
any measurable performance impact. It's when you have millions of 4KB
fragments that you have a problem.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott <sco### [at] scottcom> wrote:
> > Copying what over?
> Just the 1GB file that is fragmented so badly. If the destination drive has
> enough non-fragmented free space the file should be defragmented as it is
> copied. If your original drive has enough non-fragmented free space you can
> then just copy it back and it should be in a much better state. Note that
> if this file is a windows system file you might need to do this outside of
> windows for obvious reasons.
But that's the problem: There *isn't* enough contiguous space for the
large file to be stored unfragmented because there's a bunch of small files
scattered all over the free space, and the defragmenter is refusing to move
those files to make space, for an unknown reason.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> But that's the problem: There *isn't* enough contiguous space for the
> large file to be stored unfragmented because there's a bunch of small
> files
> scattered all over the free space, and the defragmenter is refusing to
> move
> those files to make space, for an unknown reason.
After you've moved off the 1GB file the defragmenter might behave
differently?
Anyway, even if there wasn't 1GB of contiguous free space, the file will
likely not get fragmented as much as it is currently, so it won't do any
harm.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott <sco### [at] scottcom> wrote:
> > But that's the problem: There *isn't* enough contiguous space for the
> > large file to be stored unfragmented because there's a bunch of small
> > files
> > scattered all over the free space, and the defragmenter is refusing to
> > move
> > those files to make space, for an unknown reason.
> After you've moved off the 1GB file the defragmenter might behave
> differently?
I did that the first time, and it didn't make any difference.
> Anyway, even if there wasn't 1GB of contiguous free space, the file will
> likely not get fragmented as much as it is currently, so it won't do any
> harm.
I just want to know why the defragmenter is refusing to move those files.
I would also like to make the page file one single contiguous block in the
disk. However, I'm unable to do that because of those small files.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
> I just want to know why the defragmenter is refusing to move those files.
Did you try the command line version 'defrag'? On Vista at least, there
appears to be a -w option that "Performs full defragmentation. Attempts to
consolidate all file fragments, regardless of their size.". That might be
worth a shot, as it sounds from the wording that maybe the default is to
ignore small files or something?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I just want to know why the defragmenter is refusing to move those files.
That makes two of us.
> I would also like to make the page file one single contiguous block in the
> disk. However, I'm unable to do that because of those small files.
System Internals has a "PageDefrag" tool for this exact purpose.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Invisible wrote:
> System Internals has a "PageDefrag" tool for this exact purpose.
Norton's defragger, at least, used to have a 'defragment free space'
checkbox. No idea why that wasn't a default setting, or why almost no
defraggers nowadays do that (haven't checked Norton anything in ages
ever since around when they started shoehorning themselves into the
whole system vs. being a more-or-less standalone app/suite of apps).
--
Tim Cook
http://empyrean.freesitespace.net
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I can't understand why both programs are doing this.
Possibilities: The small files are actually directories that someone has
open? The defragger has seen that it already looked at those files and
decided it won't speed things up to move them (i.e., stupid algorithm)?
Most of the old defraggers had an option to consolidate free space, but
the later versions seem to have eliminated that. Hey, here's a business
opportunity for ya. ;-)
I assume you're smart enough to try it with everything you can run stopped.
Try this:
http://www.dirms.com/home/docs/dirms1.asp
The CLI version is free. It takes forever to run, especially with the
defaults. But it claims to compact the files better.
This is why *compacting* garbage collectors are good! ;-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I'm not exactly sure XP survives such a transfer while still being fully
> bootable from the new drive. Does it?
You can do it if you put the files in the same place, maintaining the
fragmentation. :-?
> (I know from experience that a linux system can be fully transferred from
> one disk to another with a simple "cp -ax",
That's nice about Linux. All you need to do is fix up the grub stuff,
rewriting the boot sectors or whatever grub does when you reinit it.
> Anyways, it's just one fragmented file. It's not such a big deal. I'm just
> wondering *why* the small files are not being compacted to the beginning of
> the partition in order to create a larger contiguous empty space.
Because the calculation is "how much time will I spend moving it compared to
the time I spend defragging it." NTFS keeps track of how often you use each
file, too, so the system can look and say "he only accesses this once a
month; there's no benefit to moving it closer to its directory."
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> I don't see any big reasons why it wouldn't survive,
You can't copy the registry files (especially SAM) easily while the system
is running, for one. Restoring it to the new drive is also tricky for the
same reason. Basically, you'd have to write your own full-blown program to
do this. You can't just use the usual backup/restore stuff.
> There are also lots of 3rd party programs that will do
> this automatically for you, I'd be surprised if there wasn't a freeware
> one.
The only ones I found that actually work basically take images of the drive,
not individual files, meaning fragmentation persists.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> Did you try the command line version 'defrag'? On Vista at least, there
> appears to be a -w option that "Performs full defragmentation. Attempts
> to consolidate all file fragments, regardless of their size.".
No. Vista defrag won't defrag chunks if they're already >64M or some such
size. I.e., you can have a file with three 66M chunks and defrag won't try
to defrag it without the -w flag. Again, the cost/benefit ratio.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New <dne### [at] sanrrcom> wrote:
> Try this:
> http://www.dirms.com/home/docs/dirms1.asp
No dice. Also that simply refuses to move those tiny files, and is unable
to defragment that one big file. I even tried to boot to safe mode and run
it from there. No banana.
My C drive looks like this, without that big file and without the Windows
page file:
http://warp.povusers.org/snaps/c-drive.png
I can't understand what those small files are. They are not marked as
unmovable nor anything special. They just are there, and no defragmenter
is moving them.
(The master file table certainly takes a humongous amount of space.
Is that really normal?)
Btw, this is how it looks after I restore that one big file:
http://warp.povusers.org/snaps/c-drive2.png
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> No dice. Also that simply refuses to move those tiny files, and is unable
> to defragment that one big file. I even tried to boot to safe mode and run
> it from there. No banana.
Weird. I never really let it grind all the way through, but it did
defragment some stuff for me.
> http://warp.povusers.org/snaps/c-drive.png
I think I have an older (or freer) version of that program. I don't have all
those options. :-)
> I can't understand what those small files are. They are not marked as
> unmovable nor anything special. They just are there, and no defragmenter
> is moving them.
My guess, offhand, would be either directories or small files the system
happens to have open for some reason.
Oh, wait... Do you have system restore turned on? I bet you can't move files
that are the backing store of virtual volumes. Try going into system on the
control panel and turning off system restore, and see if those files go away
after the system has had time to sync the drive.
Also, maybe consider scheduling a chkdsk on boot. Perhaps the blocks wound
up being marked used but aren't really used any more?
> (The master file table certainly takes a humongous amount of space.
> Is that really normal?)
It never gets any smaller. If you run chkdsk, it'll tell you how much of the
MFT space is actually in use. There's also "reserved but unallocated" space
at the end of the MFT (normally 1/8th your disk). You can put files there,
but the system will use that space last, because once your MFT is
fragmented, you can't defrag it.
> Btw, this is how it looks after I restore that one big file:
> http://warp.povusers.org/snaps/c-drive2.png
That's only 4 fragments or so. Doesn't seem too bad. But I can understand
the frustration of not being able to figure out what those files are.
I bet it wouldn't be *too* hard to write a program to tell you file names
and cluster numbers on stdout, so you could figure out which files they are.
(Actually, DriveXML will do that as part of backing up the disk, writing it
into the xml-formatted backup file, but then you need to actually back up
the disk to get that file using that program. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
On Thu, 04 Jun 2009 21:33:21 +0200, Warp <war### [at] tagpovrayorg> wrote:
>
> My C drive looks like this, without that big file and without the
> Windows page file:
>
> http://warp.povusers.org/snaps/c-drive.png
>
> I can't understand what those small files are. They are not marked as
> unmovable nor anything special. They just are there, and no defragmenter
> is moving them.
Try analysing the drive with MyDefrag (http://www.mydefrag.com). It will
show you what file occupies any given part of the disk.
> (The master file table certainly takes a humongous amount of space.
> Is that really normal?)
The size of the MFT will of course depend on the amount of files on the
drive, but can easily be 100 MB or more.
Also, keep in mind that Windows will by default reserve up to an eighth of
the drive for the MFT to grow into. Not all defraggers distinguish between
the two parts.
--
FE
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Fredrik Eriksson wrote:
> Try analysing the drive with MyDefrag (http://www.mydefrag.com). It will
> show you what file occupies any given part of the disk.
Cool. He needs some SEO in there. I never found that, even when I was
looking for better defraggers. :-)
> The size of the MFT will of course depend on the amount of files on the
> drive, but can easily be 100 MB or more.
I think the allocated space is 1KB for each file, for the maximum number of
files you've ever allocated. I.e., 1KB per file, but it never shrinks.
> Also, keep in mind that Windows will by default reserve up to an eighth
> of the drive for the MFT to grow into.
Actually, it's at least 1/8th, and you can bumpt it up as high as 50% when
you first format the drive if you want. In case you know you'll have lots of
tiny files.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Here's some immovable files:
System restore files
Control panel->system->system restore
turn it off for now
USN journals
fsutil usn queryjournal c:
write down max size, allocation delta.
fsutil usn deletejournal /d c:
Frees the space. Takes a minute or so, actually.
Sync/checkdisk/reboot/whatever
Defrag (and pagedfrg if you're on xp)
Then turn back on system restore and
fsutil usn createjournal m=maxvalue a=alocdelta c:
See if that gets rid of your bits. Plus disk cleanup, of course.
Also, perhaps you at one time recycled a whooole lot of files, such that
your subdirectory under the recycle bin is especially large?
cd c:\$RECYCLE.BIN
dir /a
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> Why does it take such a humongous amount of time? I can't understand.
Speaking of which, on a slightly different topic...
When I boot and auto-login, it takes about 3 minutes before everything is
all settled down and happy.
If I log out, then log back in again, I'm back within maybe 5 seconds,
because everything's already in RAM and such.
I bet the thing would boot way faster if it just loaded the entire
executable for everything it needs to load at the time the program is
launched, rather than letting demand paging fight it out. I have enough RAM
to comfortably hold everything I run, so why not? I guess that'll get put
in eventually. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I don't understand the problems I'm getting defragmenting.
>
> I'm trying to defragment my primary Windows partition, which is NTFS.
> It's failing to defragment one big file because there's "no space" for it.
> The file is about 1 GB in size, and there's over 9 GB of free space in the
> partition.
>
> The problem is that while defrag is defragmenting all the other files,
> for some reason it's only compacting *some* of the files to be at the
> beginning of the partition, but not others. The free space is scattered
> with tiny small files which the defragmenter is not compacting, even though
> it compacted other files just fine. The majority of the contents were nicely
> packed to the beginning of the partition, except for these hundreds of tiny
> files. And no, the tiny files are *not* marked as "unmovable". They are
> marked with the same color as all the other movable files.
>
> The other, lighter defragger is behaving the exact same way: It's just
> refusing to move these small files, and consequently failing to defrag the
> one huge file. No reason is given in either the visual representation of the
> drive (all these tiny files are colored as regular, movable data) nor the
> analysis reports for this.
>
> As a consequence, I simply *can't* add any file bigger than a certain size
> to this partition without it becoming fragmented. There isn't enough
> contiguous free space because of all these scattered tiny files.
>
> I can't understand why both programs are doing this.
>
The Norton one, which you "used to be able to get as a separate
utility", fracking Norton..., would rearrange things so that a) commonly
used files where all in the same space, things in the same directories
where in the same space, actual directory data was located near the
start of the disk, so it was accessed faster, and compacted "all" empty
space, so that your existing files where at the start of the drive, not
scattered all over the damn place. Then they came out with Win95, and
some files got "locked" so you couldn't move them at all, never mind how
fracking fragmented they got/made things, but it "still" used smart
rearranging on everything else. Now.. I have no idea what it actually
does, or it even technically available, especially as something other
than part of 3-4 other things you don't need, and slow the system, do to
them running in the background all the time.
I personally find it incomprehensible that nothing else anyone makes
seems to be able to do anything as "simple" as compressing used space to
the start of the drive, so your "unused" space is contiguous, never mind
being "smart enough" to defrag that big 1G file, even without "enough
room", by arranging other fragments to give you the space, instead of
just leaving them empty. Its like handing a moron a paint brush and
coming back to find themselves sitting on the top of the kitchen counter
whining, "I painted myself into a corner!", only.. there are sections of
floor, all in between them and the door, none big enough to jump to or
step on, but never the less "unpainted", which they planned to "get back
to later". :p And, that the MS defrag takes the idiot amount of time it
does to paint itself into that same corner... Obviously mentally
challenged, "as well as" stupid. lol
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> Anyway, even if there wasn't 1GB of contiguous free space, the file will
>> likely not get fragmented as much as it is currently, so it won't do any
>> harm.
>
> I just want to know why the defragmenter is refusing to move those files.
> I would also like to make the page file one single contiguous block in the
> disk. However, I'm unable to do that because of those small files.
>
Simplest answer is, it behaved that way in 3.11, when it was "even more
critical", and MS never bothered to steal the idea of fixing it from the
few companies whose products **did** do it. ;)
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Patrick Elliott wrote:
> Simplest answer is, it behaved that way in 3.11, when it was "even more
> critical",
Huh? Did MS make a defragmenter for Win3.11? I thought it was just the dos
defrag? All the third-party defrags I remember using did a fine job of
defragmenting absolutely everything.
> and MS never bothered to steal the idea of fixing it from the
> few companies whose products **did** do it. ;)
Cite? Certainly the XP defragmenter moves files around to open up free
spaces. I suspect it's just files that have absolute addresses in them,
like the UNC and system restore points.
Hey Warp, if you get it defragged, let us know what it was? :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Patrick Elliott wrote:
> I personally find it incomprehensible that nothing else anyone makes
Everyone uses the same API. If there's a reason the files can't be moved, no
program is going to be able to move them.
> seems to be able to do anything as "simple" as compressing used space
Clearly it isn't that simple.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> Did you try the command line version 'defrag'? On Vista at least, there
>> appears to be a -w option that "Performs full defragmentation. Attempts
>> to consolidate all file fragments, regardless of their size.".
>
> No. Vista defrag won't defrag chunks if they're already >64M or some such
> size. I.e., you can have a file with three 66M chunks and defrag won't try
> to defrag it without the -w flag. Again, the cost/benefit ratio.
Ah ok, I checked on my XP box and there is a -f flag, to force defrag even
with low freespace - maybe this would help Warp?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
>> I don't see any big reasons why it wouldn't survive,
>
> You can't copy the registry files (especially SAM) easily while the system
> is running, for one. Restoring it to the new drive is also tricky for the
> same reason.
Why can't you just copy all the files while the OS isn't running? eg from
another PC/OS/LiveCD/boot disc?
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Fredrik Eriksson <fe79}--at--{yahoo}--dot--{com> wrote:
> Try analysing the drive with MyDefrag (http://www.mydefrag.com).
That program actually did the trick. It moved those tiny files to make
space. Now my C drive layout looks like this, *with* that huge file and
*with* the Windows page file:
http://warp.povusers.org/snaps/c-drive3.png
(AFAIK those gaps are intentionally left by MyDefrag. The idea is that
programs creating small temp files will create them in those gaps, and
since the gaps are close to the beginning of the disk, they will be faster
to access.)
So what were those tiny files which other defraggers refused to move?
I don't know. MyDefrag gave weird names to them. I don't remember exactly
now, but they were something like C:\$SOMETHING:$SOMETHINGE_ELSE
I don't remember the exact names, but they looked meaningful. Maybe they
were somehow related to the NTFS file system.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New <dne### [at] sanrrcom> wrote:
> Everyone uses the same API. If there's a reason the files can't be moved, no
> program is going to be able to move them.
MyDefrag was able to move those strange clusters, even though it uses the
regular defragmenter system API to do so...
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> Ah ok, I checked on my XP box and there is a -f flag, to force defrag
> even with low freespace - maybe this would help Warp?
I don't think that'll help either. If he doesn't have a 1G hole on the disk,
and he can't move the files that are in the way, no number of flags is going
to compact the files. The -f flag doesn't affect that - defrag runs or it
doesn't.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
scott wrote:
> Why can't you just copy all the files while the OS isn't running? eg
> from another PC/OS/LiveCD/boot disc?
You could if you have the permissions, of course. You just can't copy the
drive you booted off of because of the registry. Most specifically, you
can't copy *to* the drive you booted off of.
There are a few immovable files (like the one holding the boot code, the
MFT, etc). There are a couple types of unmovable files (like the system
restore checkpoints, which have coded in their data the cluster numbers they
represent). There are permission systems designed to keep you from writing
to the file that holds the security information. You can't use the normal
access mechanisms (i.e., "copy") to copy the registry files around because
of security concerns.
It's just much more complicated than Linux. It's not impossible, or nobody
would be able to back up their computer. But the only software I've found
that actually will make a complete backup and then *restore* it uses imaging
techniques, not per-file copying.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> I don't remember the exact names, but they looked meaningful. Maybe they
> were somehow related to the NTFS file system.
Anything with a $ at the front is a file in NTFS used to run the file
system. Everything on NTFS is actually in a file, so you have things like
$RECYCLE.BIN (not really a magic file, tho)
$MFT (basically, inodes)
$BOOT (boot file the boot sector points to - the "superblock")
$BITMAP (the free space bitmap)
$USN (unique serial number file - no ext3 equivalent I know of)
$JOURNAL (transaction journal)
You also have files for the keys used for encryption and security
descriptors. (In later NTFS versions, each different ACL is hashed into its
place in the security descriptor file and a pointer to it is stored in the
file. Apparently a space saving over storing the full ACL with each file.)
Stuff with a : in the name means you have a different fork of the file. Kind
of like how System7 had data forks and resource forks on Mac, except each
fork is a byte stream and you can have as many as you want. Forks that start
with a $ are used by MS, either the file system or the shell.
Without knowing the actual name, it would be hard to guess what they were.
Glad you got it fixed, tho.
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> I bet the thing would boot way faster if it just loaded the entire
> executable for everything it needs to load at the time the program is
> launched, rather than letting demand paging fight it out. I have enough
> RAM
> to comfortably hold everything I run, so why not? I guess that'll get put
> in eventually. :-)
On Linux I have things set up so that my whole desktop environment is
preloaded into RAM right after the login manager is started.
There is a tool that records file reads. You configure it to run, then
reboot, and login, maybe open some apps you usually use. Next time you
reboot, another tool will read the recorded log and use the readahead
syscall on them. The file list is even sorted by their physical location on
the disk.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Nicolas Alvarez wrote:
> There is a tool that records file reads. You configure it to run, then
> reboot, and login, maybe open some apps you usually use. Next time you
> reboot, another tool will read the recorded log and use the readahead
> syscall on them. The file list is even sorted by their physical location on
> the disk.
Yeah, XP has that built in, and Vista to an even greater extent. But they
only work on (AFAIU) the demand-paged blocks, not entire files. Does the
Linux tool read an entire file, or just the parts that were paged in at
start-up? I suspect 5 seconds would be plenty to read all the login files if
the heads didn't thrash like mad.
It just seems like it could be done better. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> Nicolas Alvarez wrote:
>> There is a tool that records file reads. You configure it to run, then
>> reboot, and login, maybe open some apps you usually use. Next time you
>> reboot, another tool will read the recorded log and use the readahead
>> syscall on them. The file list is even sorted by their physical location
>> on the disk.
>
> Yeah, XP has that built in, and Vista to an even greater extent. But they
> only work on (AFAIU) the demand-paged blocks, not entire files. Does the
> Linux tool read an entire file, or just the parts that were paged in at
> start-up? I suspect 5 seconds would be plenty to read all the login files
> if the heads didn't thrash like mad.
>
> It just seems like it could be done better. :-)
The command-line tools work in terms of whole files. The syscall takes a
file descriptor and a byte range.
readahead-watch(8): Watches the mount points given, or the root filesystem,
for files opened during the boot sequence and closed without writing to
them. Orders the list by on-disk position and then writes it to the output
file.
readahead-list(8): Loads lists from the given file and performs readahead(2)
on each entry.
readahead(2): populates the page cache with data from a file so that
subsequent reads from that file will not block on disk I/O.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Nicolas Alvarez wrote:
> The command-line tools work in terms of whole files. The syscall takes a
> file descriptor and a byte range.
>
> readahead-watch(8): Watches the mount points given, or the root filesystem,
> for files opened during the boot sequence and closed without writing to
> them. Orders the list by on-disk position and then writes it to the output
> file.
>
> readahead-list(8): Loads lists from the given file and performs readahead(2)
> on each entry.
>
> readahead(2): populates the page cache with data from a file so that
> subsequent reads from that file will not block on disk I/O.
Yeah! That's exactly what I need on Vista. :-) Especially if it would read
the whole file in one blow, or at least each contiguous piece. The vista
mechanisms just don't really do the trick, I think.
Actually, with the USN, you wouldn't even need readahead-watch. Just parse
the USN journal since the last boot and build the list from that. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> Yeah! That's exactly what I need on Vista. :-)
Or not even that, really. If exec() would just read the whole file into RAM
in one blow, it would probably be 10x as fast. I should be able to start up
half a gig of executables in well under 10 seconds, and I'm pretty sure I'm
not loading that much stuff up when I log in. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> Patrick Elliott wrote:
>> Simplest answer is, it behaved that way in 3.11, when it was "even
>> more critical",
>
> Huh? Did MS make a defragmenter for Win3.11? I thought it was just the
> dos defrag? All the third-party defrags I remember using did a fine job
> of defragmenting absolutely everything.
>
There is a relevant difference between DOS Defrag and a Windows one,
back when Win3.11 "was" basically just DOS with a GUI? lol
>> and MS never bothered to steal the idea of fixing it from the few
>> companies whose products **did** do it. ;)
>
> Cite? Certainly the XP defragmenter moves files around to open up free
> spaces. I suspect it's just files that have absolute addresses in them,
> like the UNC and system restore points.
>
Hmm. Nope. I had the *identical* issue with 95 and 98. You may be partly
correct, to some extent, in that "often" the virtual memory file gets
fragmented to hell too, and the defrag can't move those (unlike Norton
used to in 3.11->98). So, if "that" file gets hacked up into small bits,
the defragger will "never" fix them. Running it "over and over and over"
under 95/98 could get it to "partly" fix some of the gaps, but at some
point it would just flat out refuse to move anything else around, so you
had empty spots between them, no matter how many times it ran. MS Defrag
has "never" changed or improved this behavior in "any" release.
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Patrick Elliott wrote:
> There is a relevant difference between DOS Defrag and a Windows one,
> back when Win3.11 "was" basically just DOS with a GUI? lol
Yes, lol. Because you ran the defrag under DOS, nothing else ran at the same
time, lol.
>>> and MS never bothered to steal the idea of fixing it from the few
>>> companies whose products **did** do it. ;)
>>
>> Cite? Certainly the XP defragmenter moves files around to open up
>> free spaces.
>>
> Hmm. Nope.
OK. I guess you don't know what "cite" means. :-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> Darren New <dne### [at] sanrrcom> wrote:
>> Everyone uses the same API. If there's a reason the files can't be moved, no
>> program is going to be able to move them.
>
> MyDefrag was able to move those strange clusters, even though it uses the
> regular defragmenter system API to do so...
>
Ah, but see... They used it in a "non-standard" way. In all likelihood,
with Win7 they will release a new .DEFRAG API, which ***accidentally***
breaks this non-standard method, while offering some crazy ass
alternative, like using another drive to "automatically" tranfer files,
temporarily, so they are defragged when copied back (will only work if
you have multiple drives/partitions), kind of like how they provides
some new ways in .NET to "fake" access to the old IUNKNOWN functions in
the ActiveX API, which where never properly documented, and couldn't be
used easily, thereby letting you hot-load new .NET objects, and allow
your application to use them without "hard coding" the interfaces into
your application before hand, but which also, even more completely,
prevents you using them in the "design mode" that is only available to
you when "building" an interface in Visual Studio.
See, there are plenty of documents that, and I love this, tell you how
to "make" controls that "respond to" design mode, so you can better
handle them in Visual Studio, but... gods forbid someone wants to make
an application of their own that took advantage of that feature, so you
could "design" an interface, and say.. save it to a layout document, to
be reloaded later, when actually using the new interface. Your only
"allowed" to do that via the Visual Studio. And now, you can't even
"try" to do it without that, because the new .NET buries the interfaces
and APIs needed to even attempt it under 50 new layers of BS (while
giving you half of what you wanted in the first place).
So, basically, figure that if this is a feature people will want, its
something MS will provide "half" of the function for in the new API,
while simultaneously making it completely impossible to use the new API
to get "all" of the functionality any more. ;) lol
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
> Patrick Elliott wrote:
>> There is a relevant difference between DOS Defrag and a Windows one,
>> back when Win3.11 "was" basically just DOS with a GUI? lol
>
> Yes, lol. Because you ran the defrag under DOS, nothing else ran at the
> same time, lol.
>
>>>> and MS never bothered to steal the idea of fixing it from the few
>>>> companies whose products **did** do it. ;)
>>>
>>> Cite? Certainly the XP defragmenter moves files around to open up
>>> free spaces.
>> Hmm. Nope.
>
> OK. I guess you don't know what "cite" means. :-)
>
You want me to what? Find a web page written by someone even more
annoyed by the behavior than I am to give a jaded statement to the same
effect? lol Seriously though, I don't dispute that it "does" move things
around. What it doesn't do is "move them if it doesn't see any reason
to, and consolidating free space isn't always, apparently, a 'reason'".
And it flat out refuses to move things that it "could", if the bothered
to put in some way to copy a block of an "in use" file, and.. well do
the stuff that defraggers do to every other file anyway. I mean, can you
seriously tell me that they couldn't come up with "some" way to be able
to, I don't know, write a block of memory to some other part of the
virtual file, then invalidate the part being moved, then revert again
when done, or "something"?
And, the idea that DOS based defraggers where the only ones in 3.11
isn't "entirely" accurate. The Norton one came up during 3.11, and it
"did" defrag "everything", at that time, including the virtual memory
file, and just about anything and everything else "except" for a small
handful of active files in the OS. Say, 90% of all the stuff that no
defragger today will even touch, for the most part.
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Patrick Elliott wrote:
> You want me to what? Find a web page written by someone even more
> annoyed by the behavior than I am to give a jaded statement to the same
> effect?
I rest my case.
> Seriously though, I don't dispute that it "does" move things
> around. What it doesn't do is "move them if it doesn't see any reason
> to, and consolidating free space isn't always, apparently, a 'reason'".
The XP defragger usually consolodated much free space for me. Not 100%, but
certainly some.
> And it flat out refuses to move things that it "could", if the bothered
> to put in some way to copy a block of an "in use" file,
I'm glad you know better than Microsoft how easy it is to do that sort of thing.
> and.. well do
> the stuff that defraggers do to every other file anyway. I mean, can you
> seriously tell me that they couldn't come up with "some" way to be able
> to, I don't know, write a block of memory to some other part of the
> virtual file, then invalidate the part being moved, then revert again
> when done, or "something"?
I don't even know what you're talking about there.
> And, the idea that DOS based defraggers where the only ones in 3.11
> isn't "entirely" accurate. The Norton one came up during 3.11, and it
> "did" defrag "everything", at that time, including the virtual memory
> file, and just about anything and everything else "except" for a small
> handful of active files in the OS. Say, 90% of all the stuff that no
> defragger today will even touch, for the most part.
FAT is a lot easier to defrag than NTFS, yes. But feel free to continue
ranting while cackling gleefully. ;-)
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp <war### [at] tagpovrayorg> wrote:
> The other, lighter defragger is behaving the exact same way: It's just
> refusing to move these small files, and consequently failing to defrag the
> one huge file. No reason is given in either the visual representation of the
> drive (all these tiny files are colored as regular, movable data) nor the
> analysis reports for this.
> I can't understand why both programs are doing this.
Sounds like there's some serious reason to not touch them. Maybe they're
currently open whenever you run defrag?
Tried booting in safe mode and then defragging? That should reduce the number of
processes running, and thus the number of open files.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp <war### [at] tagpovrayorg> wrote:
> Why does it take such a humongous amount of time? I can't understand.
As someone already said, head movements is the limiting factor here.
Once the defrag software has made up its mind where to ultimately place the
files, the remaining time it will spend...
- picking up clusters from their original position (remember, the files start
heavily fragmented, so *many* major head movements to be expected here already)
- copying those clusters to their target position (or, more frequently, an
interim position) far away (another major head movement; fortunately, we're
talking about contiguous clusters now)
- updating the File Allocation Tables (there's typically two of them) at the
start of the disk (one more major head mvoement), to reflect the changes
- doing roughly the same to move clusters from interim positions to their target
position (fortunately, they should be contiguous already by now).
The speed of a defrag software will heavily depend on how good it is at
minimizing head movement; for instance, if for some reason clusters of a file
happen to be ordered wrong way round (e.g. C...A...B...D), a naive defrag
software may try to pick them up in logical order, while an intelligent one
could recognize the situation and pick them up in physical order instead.
Similarly, a naive software may move clusters out of the way to the very end of
the disk - while an intelligent one may temporarily store some clusters even
where other clusters are intended to end up, and just make sure the spot is
free again soon enough.
Having lots of physical main memory installed should help, because a bigger
number of clusters can be processed at once, and at least head seeks for
writing the clusters back and updating the FAT will be reduced.
Having a disk with >50% free space helps a lot, too - otherwise clusters may
have to be moved back and forth multiple times before reaching their final
destination.
For some really hard cases, it may be worth considering to mirror the whole
disk, delete all the (movable) files from the original one (remember to empty
the recycle bin), then copy those files back from the mirror. If we're talking
about an OS partition, doing this on a different machine may work (in case it
doesn't, you can still do an exact restore from the mirror).
Placing applications, games and user data on a partition separate from the OS is
a very helpful thing, too.
> I also noticed that both defragmenters failed to use the free space
> in the partition (over 80 GB) to their advantage, and instead moving
> small clusters around one at a time. I'm pretty certain that if all the
> free available space was used to copy data, the whole process could be
> a lot faster.
Maybe the defrag software was actually able to move most clusters directly to
their intended destination, without having to temporarily store them somewhere
else. Or it chose temporary storage location the smart way, as close to the
source or intended destination as possible, to minimize head movements.
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Tim Cook wrote:
> Norton's defragger, at least, used to have a 'defragment free space'
> checkbox. No idea why that wasn't a default setting, or why almost no
Because if you compact all files, and then grow a file at all, you have
two options: use the free space at the end, or fragment the file.
If you don't want to rewrite the whole file, fragmenting it is (in that
situation) the easier solution.
--
Chambers
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Darren New wrote:
>> And it flat out refuses to move things that it "could", if the
>> bothered to put in some way to copy a block of an "in use" file,
>
> I'm glad you know better than Microsoft how easy it is to do that sort
> of thing.
>
Not actually knowing how their mess works, I have no real idea what road
blocks they may have created to make it impossible, but at the same
time, we are talking about Microsoft here... The people that only
"recently" fixed certain bugs in Excel and other things, which have been
around since pretty much when the product "first" came out too. Tell me,
is the "some years Windows doesn't recognize one day out of the year,
because its code for most DBs/Spreadsheets/anything using dates of
calenders, is hard coded to think there will never be 4 Saturdays in a
month, fixed yet? And how long has that been around? lol
See, when some random 3rd party application, like MyDefrag can manage,
using the same API, what Microsoft has failed to, one has to wonder if
MS even *cares* that their version is, and has been, basically broken
from day one.
--
void main () {
If Schrödingers_cat is alive or version > 98 {
if version = "Vista" {
call slow_by_half();
call DRM_everything();
}
call functional_code();
}
else
call crash_windows();
}
<A HREF='http://www.daz3d.com/index.php?refid=16130551'>Get 3D Models,
3D Content, and 3D Software at DAZ3D!</A>
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Patrick Elliott wrote:
> See, when some random 3rd party application, like MyDefrag can manage,
> using the same API, what Microsoft has failed to,
You don't know what other variables were involved that kept the blocks from
being moved. Warp already said they were management files and not normal
user files, and the other defraggers couldn't move them either. They could
have been from a system restore point that expired and got cleaned up
automatically, for example.
> one has to wonder if
> MS even *cares* that their version is, and has been, basically broken
> from day one.
Funky how "does not do what Patrick thinks it should" translates to "broken".
--
Darren New, San Diego CA, USA (PST)
There's no CD like OCD, there's no CD I knoooow!
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
|
|
