Bought a new 750G hard drive today, so I thought I'd test out the 
formatting speed question. All tests on USB2 connection to same computer.

Format as ext3 - 10:10 (That's 10 minutes ten seconds)
Fsck as ext3   - 11:30 (No files)

Format as NTFS (quick) - 00:14
chkdsk (no files)      - 00:06

I'm pretty sure ext2/3 have control information proportional to the size 
of the drive, where NTFS doesn't.

-- 
Darren New / San Diego, CA, USA (PST)
  Helpful housekeeping hints:
   Check your feather pillows for holes
    before putting them in the washing machine.

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> Format as ext3 - 10:10 (That's 10 minutes ten seconds)
> Fsck as ext3   - 11:30 (No files)

> Format as NTFS (quick) - 00:14
> chkdsk (no files)      - 00:06

  It seems that you are trying to make some kind of point or statement.
However, that would be quite moot, unless you have to format disks on
a daily basis, which is quite rare.

  (Btw, are you absolutely sure fsck is not performing a surface scan?)

-- 
                                                          - Warp

Post a reply to this message

>> Format as ext3 - 10:10 (That's 10 minutes ten seconds)
>> Fsck as ext3   - 11:30 (No files)
> 
>> Format as NTFS (quick) - 00:14
>> chkdsk (no files)      - 00:06
> 
>   It seems that you are trying to make some kind of point or statement.
> However, that would be quite moot, unless you have to format disks on
> a daily basis, which is quite rare.

More trying to figure out how much of the disk surface gets erased 
during a format by measuring how long it takes. (14 seconds is obviously 
nowhere near enough time to erase a significant percentage of the drive, 
for example.)

Hey Darren, try a low-level format now...

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Warp wrote:
> Darren New <dne### [at] sanrrcom> wrote:
>> Format as ext3 - 10:10 (That's 10 minutes ten seconds)
>> Fsck as ext3   - 11:30 (No files)
> 
>> Format as NTFS (quick) - 00:14
>> chkdsk (no files)      - 00:06
> 
>   It seems that you are trying to make some kind of point or statement.

A couple weeks ago, Andrew pointed out that his ext2 format took a few 
seconds and his NTFS format took 20 minutes. I measured it on a 40G 
partition and found they each took less than 15 seconds if you didn't do 
the scan, but I remembered it taking much longer at work.

So when I got a work-sized drive at home that wasn't full of stuff I 
wanted to keep, I tried it again.

> However, that would be quite moot, unless you have to format disks on
> a daily basis, which is quite rare.

Yep, pretty much. I'm not trying to bash anything. I didn't even try the 
other Linux file systems.

>   (Btw, are you absolutely sure fsck is not performing a surface scan?)

Yes. A surface scan would take about 15 hours on this drive. (I'm 
filling it with random data right now, as I don't really trust to 
backups a drive I've never actually written to.) It's going about one 
gig per 50 seconds.

I'll also note that at work, an fsck on a file system with a lot of 
files (like, a couple million files) takes maybe 20 to 40 minutes on an 
IDE drive, as opposed to just a couple on NTFS. Clearly different 
trade-offs were made in the design of the file systems.

I'd like to find a file system I can reliably use from OSX, Windows, and 
Linux, myself. OSX can almost handle NTFS (as in, it seems to be able to 
read/write NTFS as long as you don't hit the parts that aren't 
supportable with the UNIX file system semantics), but as far as I can 
tell, NTFS still isn't safely writable from Linux? That's what google is 
telling me, at least.

-- 
Darren New / San Diego, CA, USA (PST)
  Helpful housekeeping hints:
   Check your feather pillows for holes
    before putting them in the washing machine.

Post a reply to this message

Orchid XP v8 wrote:
> Hey Darren, try a low-level format now...

I don't think you can do that over USB. :-)

-- 
Darren New / San Diego, CA, USA (PST)
  Helpful housekeeping hints:
   Check your feather pillows for holes
    before putting them in the washing machine.

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> I didn't even try the other Linux file systems.

  I use ReiserFS for all my linux partitions. Besides being journaled,
it's more compact than many other FSs because it packs file tails into
single sectors. (In other words, if the data in the last sector of two
files would fit in a single sector, ReiserFS puts then into the same
sector.)

  One disadvantage of ReiserFS is that to run reiserfsck the partition
must be read-only. This means that if you need to run it on your main
partition you'll have to boot to single-user mode and remount the
partition to read-only before you can run reiserfsck.

  Another minor (in my case very minor) quirk is that deleting very
large files can take some time (in the order of several seconds),
unlike with some other FSs where file deletion happens instantaneously.
(OTOH, I don't know if this is a property of ReiserFS itself or the
journaling mechanism, and if other journaled file systems suffer from
the same phenomenon.)

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:

>   I use ReiserFS for all my linux partitions.

I've not used it, but I've heard good things about it.

> Besides being journaled,
> it's more compact than many other FSs

I doubt I'd notice the difference. Heck, I don't even notice the 
difference between ext2 and ext3...

What I would be curios to know is this. The traditional Unix file 
permissions system allows you to assign a set of permissions to the 
"owner", another set to the "group", and a third set to "everybody 
else". Is there any kind of extension that allows you to assign 
arbitrary permissions to individual users and/or groups beyond what the 
traditional system allows? (I would anticipate the Big Problem with 
attempting to do this would be that it breaks everything that expects 
the old behaviour!)

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 <voi### [at] devnull> wrote:
> > Besides being journaled,
> > it's more compact than many other FSs

> I doubt I'd notice the difference. Heck, I don't even notice the 
> difference between ext2 and ext3...

  You might notice a difference in disk usage if you have enormous
amounts of small files. (In a non-compacting FS every file takes a
multiple of 4 kB, rounded up. With ReiserFS they can take less. For
example if you have 1000 files, each one 128 bytes in length, instead
of taking 4 MB of disk space they only take 128 kB.)

> What I would be curios to know is this. The traditional Unix file 
> permissions system allows you to assign a set of permissions to the 
> "owner", another set to the "group", and a third set to "everybody 
> else". Is there any kind of extension that allows you to assign 
> arbitrary permissions to individual users and/or groups beyond what the 
> traditional system allows? (I would anticipate the Big Problem with 
> attempting to do this would be that it breaks everything that expects 
> the old behaviour!)

  There might be some file systems which support additional permissions,
but I don't remember right now.

-- 
                                                          - Warp

Post a reply to this message

Am Sun, 06 Jul 2008 19:03:47 +0100 schrieb Orchid XP v8:

> Warp wrote:
> 
>>   I use ReiserFS for all my linux partitions.
> 
> I've not used it, but I've heard good things about it.
> 
>> Besides being journaled,
>> it's more compact than many other FSs
> 
> I doubt I'd notice the difference. Heck, I don't even notice the
> difference between ext2 and ext3...
> 
> What I would be curios to know is this. The traditional Unix file
> permissions system allows you to assign a set of permissions to the
> "owner", another set to the "group", and a third set to "everybody
> else". Is there any kind of extension that allows you to assign
> arbitrary permissions to individual users and/or groups beyond what the
> traditional system allows? (I would anticipate the Big Problem with
> attempting to do this would be that it breaks everything that expects
> the old behaviour!)

It is called "Access Control Lists" and part of POSIX. Manpage is called 
acl(5).

Post a reply to this message

Warp wrote:
> Darren New <dne### [at] sanrrcom> wrote:
>> I didn't even try the other Linux file systems.
> 
>   I use ReiserFS for all my linux partitions. Besides being journaled,
> it's more compact than many other FSs because it packs file tails into
> single sectors. (In other words, if the data in the last sector of two
> files would fit in a single sector, ReiserFS puts then into the same
> sector.)

That's handy. (The difference between ext2 and ext3 is that ext3 is 
journaled. You can even switch back and forth between the two of them by 
essentially deleting or creating the journal file.)

>   One disadvantage of ReiserFS is that to run reiserfsck the partition
> must be read-only. This means that if you need to run it on your main
> partition you'll have to boot to single-user mode and remount the
> partition to read-only before you can run reiserfsck.

I'm pretty sure that's true of ext3, also. Indeed, I'm not sure I know 
any file system you can check for consistency while it's in use. (Not if 
you want to actually correct flaws, at least.)

Does it work to set /forcefsck and reboot, or do you actually have to be 
sitting there to do it?

>   Another minor (in my case very minor) quirk is that deleting very
> large files can take some time (in the order of several seconds),
> unlike with some other FSs where file deletion happens instantaneously.

Yeah. I've had files large enough that it takes literally minutes to 
delete on ext3, so that's no surprise either. Ext3 actually stores a 
list of blocks the file occupies, even if they're contiguous, so 
deleting a 500-gig file is going to take time just looking up all the 
blocks it's on.

I just used ext3 because it was the default and I had to get stuff 
working before I knew as much as I do now. It turns out it's one of the 
more poorly-performing file systems available.

-- 
Darren New / San Diego, CA, USA (PST)
  Helpful housekeeping hints:
   Check your feather pillows for holes
    before putting them in the washing machine.

Post a reply to this message

Warp wrote:
>   You might notice a difference in disk usage if you have enormous
> amounts of small files. (In a non-compacting FS every file takes a
> multiple of 4 kB, rounded up. With ReiserFS they can take less. For
> example if you have 1000 files, each one 128 bytes in length, instead
> of taking 4 MB of disk space they only take 128 kB.)

Interestingly enough, in NTFS, if all the data fits in the same sector 
as the inode[1], it goes into the same sector.

>   There might be some file systems which support additional permissions,
> but I don't remember right now.

There are, and yes, they aren't especially well supported, especially by 
things like rsync, svn, backup programs, etc.


[1] For the NTFS definition of inode.

-- 
Darren New / San Diego, CA, USA (PST)
  Helpful housekeeping hints:
   Check your feather pillows for holes
    before putting them in the washing machine.

Post a reply to this message

Just some more timings, for S&G:

362000 files, 77G used of 215G, ext3 IDE - 7 minutes 20 seconds.

428000 files, 6.5G used of 82G - ext3 IDE - 2:40

100,000 files, 24G used, NTFS IDE - 2:20

90,000 files, 160G used, NTFS IDE - 1:30.

Really all over the board. It looks like the biggest factor for ext3 is 
the capacity of the drive, and I *suspect* the biggest complexity in 
NTFS is the complexity of the directory structure. (The 3rd one has a 
much more complex directory structure right now than the fourth one).

-- 
Darren New / San Diego, CA, USA (PST)
  Helpful housekeeping hints:
   Check your feather pillows for holes
    before putting them in the washing machine.

Post a reply to this message

Darren New wrote:
> 
> I'd like to find a file system I can reliably use from OSX, Windows, and 
> Linux, myself. OSX can almost handle NTFS (as in, it seems to be able to 
> read/write NTFS as long as you don't hit the parts that aren't 
> supportable with the UNIX file system semantics), but as far as I can 
> tell, NTFS still isn't safely writable from Linux? That's what google is 
> telling me, at least.
> 

Check out NTFS-3G:
http://www.ntfs-3g.org/

-- 
Eero "Aero" Ahonen
    http://www.zbxt.net
       aer### [at] removethiszbxtnetinvalid

Post a reply to this message

>>   You might notice a difference in disk usage if you have enormous
>> amounts of small files. (In a non-compacting FS every file takes a
>> multiple of 4 kB, rounded up. With ReiserFS they can take less. For
>> example if you have 1000 files, each one 128 bytes in length, instead
>> of taking 4 MB of disk space they only take 128 kB.)
> 
> Interestingly enough, in NTFS, if all the data fits in the same sector 
> as the inode[1], it goes into the same sector.

...which the result that if you have many hundreds of tiny files in a 
single directory... the directory structure becomes rather large and 
slow to traverse. ;-) (Still better than wasting half your harddrive 
though!)

>>   There might be some file systems which support additional permissions,
>> but I don't remember right now.
> 
> There are, and yes, they aren't especially well supported, especially by 
> things like rsync, svn, backup programs, etc.

I thought that would be problematic. ;-) But it at least exists though?

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
>> Interestingly enough, in NTFS, if all the data fits in the same sector 
>> as the inode[1], it goes into the same sector.
> 
> ....which the result that if you have many hundreds of tiny files in a 
> single directory... the directory structure becomes rather large and 
> slow to traverse. ;-) (Still better than wasting half your harddrive 
> though!)

Are you joking? Or are you serious? Because from what I understand, the 
directory structures have nothing in them but names and pointers into 
the master file table (which stores the file records that are the 
equivalent of inodes). In other words, the directory hasn't anything to 
do with the files, and doesn't get bigger just because you're storing 
small files.

I could be wrong about that, tho, if you actually have information that 
says otherwise.

> I thought that would be problematic. ;-) But it at least exists though?

On some file systems, yes.

-- 
Darren New / San Diego, CA, USA (PST)
  Helpful housekeeping hints:
   Check your feather pillows for holes
    before putting them in the washing machine.

Post a reply to this message

Eero Ahonen wrote:
> Check out NTFS-3G:
> http://www.ntfs-3g.org/

Very cool! Thank you!

-- 
Darren New / San Diego, CA, USA (PST)
  Helpful housekeeping hints:
   Check your feather pillows for holes
    before putting them in the washing machine.

Post a reply to this message

>>> Interestingly enough, in NTFS, if all the data fits in the same 
>>> sector as the inode[1], it goes into the same sector.
>>
>> ....which the result that if you have many hundreds of tiny files in a 
>> single directory... the directory structure becomes rather large and 
>> slow to traverse. ;-) (Still better than wasting half your harddrive 
>> though!)
> 
> Are you joking? Or are you serious? Because from what I understand, the 
> directory structures have nothing in them but names and pointers into 
> the master file table (which stores the file records that are the 
> equivalent of inodes). In other words, the directory hasn't anything to 
> do with the files, and doesn't get bigger just because you're storing 
> small files.
> 
> I could be wrong about that, tho, if you actually have information that 
> says otherwise.

The information I heard is that "small" files are stored in-line inside 
the directory structure to prevent additional seeks (and wasted 
half-blocks). The source I got that from wasn't highly technical, so 
this might be a simplification.

-- 
http://blog.orphi.me.uk/
http://www.zazzle.com/MathematicalOrchid*

Post a reply to this message

Orchid XP v8 wrote:
> The information I heard is that "small" files are stored in-line inside 
> the directory structure to prevent additional seeks (and wasted 
> half-blocks). The source I got that from wasn't highly technical, so 
> this might be a simplification.

The "directory structure" includes more than the directories.

There's a "master file table", which holds all the metadata for each 
file, such as its names, what directories it's in, how big it is, the 
pointer to the ACL, what clusters it's stored on, when it was created, 
where it's backed up, what streams it has and where, the encryption 
information, and so on. The MFT is basically an array of file objects, 
and each file object is an array of file properties. One of those file 
properties *can* be the data of the file.

Then there are directories, which store the names and pointers into the 
master file table as B-trees.

If you're familiar with old UNIX style file systems, the "master file 
table" holds the equivalent of the i-nodes, and the directories hold the 
equivalent of the directories.

Having lots of files in a directory leaves it the same size regardless 
of how much data is in the files, because the directories just refer to 
offsets in the master file table.

(Now you understand the three passes of chkdsk. :-)

The reason the formatting is fast is the master file table is just a 
file, as is the free space map and ACL lists (and boot sector, for that 
matter), and they're all dynamically allocated as needed just like any 
other file, so it's a pretty constant amount of data written during a 
format.

Ext2 pre-allocates all the i-nodes, and hence must actually clear them 
(i.e., mark them as not in use). And (presumedly) for speed, it spreads 
these i-nodes out across the disk, so there are thousands of seeks to 
format a 750G disk.

-- 
Darren New / San Diego, CA, USA (PST)
  Helpful housekeeping hints:
   Check your feather pillows for holes
    before putting them in the washing machine.

Post a reply to this message