On Sun, 08 Mar 2009 15:08:22 +0100, [GDS|Entropy]  
<gds### [at] hotmailcom> wrote:
> "Eero Ahonen" <aer### [at] removethiszbxtnetinvalid> wrote in message
> news:49b3d036$1@news.povray.org...
>> [GDS|Entropy] wrote:
>>> I *really* hate those.
>>> Can't pov just use a freakin swap file?? Grrr....
>>
>> Yes, it can. Tested with 12G consuming render and 1G of physical RAM :).
>>
>> -Aero
>>
>
> I don't understand why I'm getting those errors then... It will suck up  
> 2gb of ram and then it dies, sometimes catastrophically. I have like 80gb
> free on disk too.

Because you are using a 32-bit OS. With a few tweaks you could push the  
limit to 3 GB, but that is as much as you can get for a single 32-bit  
application.


-- 
FE

Post a reply to this message

[GDS|Entropy] <gds### [at] hotmailcom> wrote:
> I don't understand why I'm getting those errors then... It will suck up 2gb 
> of ram and then it dies, sometimes catastrophically. I have like 80gb free 
> on disk too.

  You have a 32-bit computer and you are running the 32-bit povray
executable, yes? 32-bit programs cannot allocate more than 2 GB
(or in Windows, if you turn on some setting, 3GB) of memory. It's
a question of adressable space.

  It would be possible to address more data by using 64-bit addresses
(which is how eg. files of over 4GB are handled), but in a 32-bit
computer managing 64-bit addresses will impose a speed penalty, as
every operation requiring memory access or even just copying these
addresses around involves extra steps. This would also require some
heavy refactoring of the entire POV-Ray source code.

-- 
                                                          - Warp

Post a reply to this message

"Warp" <war### [at] tagpovrayorg> wrote in message 
news:49b3f1f4@news.povray.org...
> [GDS|Entropy] <gds### [at] hotmailcom> wrote:
>> I don't understand why I'm getting those errors then... It will suck up 
>> 2gb
>> of ram and then it dies, sometimes catastrophically. I have like 80gb 
>> free
>> on disk too.
>
>  You have a 32-bit computer and you are running the 32-bit povray
> executable, yes? 32-bit programs cannot allocate more than 2 GB
> (or in Windows, if you turn on some setting, 3GB) of memory. It's
> a question of adressable space.

I know that in 32bit OS progs are limited to 2gb max or so each, but why 
can't pov use a swap file like photoshop?

>  It would be possible to address more data by using 64-bit addresses
> (which is how eg. files of over 4GB are handled), but in a 32-bit
> computer managing 64-bit addresses will impose a speed penalty, as
> every operation requiring memory access or even just copying these
> addresses around involves extra steps. This would also require some
> heavy refactoring of the entire POV-Ray source code.

Yeah...

Damn....I'm going to have to break down and install XP Pro 64 I think...

I'd like to upgrade to quadcore first though, since its only like $200 now 
for the proc..

You don't happen to know what that "some setting" is that would let me use 
3gb for progs until then do you? That would really be useful...

I freakin swear, my next computer is going to have at least 1333mhz mobo 
bus, 16Gb+ DDR3 Ram, RAID5 10krpm SATA (SAS to much $$), 2x768mb nvidia 
geforce sli, eSATA ports, USB3 ports, xp pro 64...hells...I might even go 
for one of those nvidia Tesla systems...(wish I could snatch up one of the 
Cray CX1s...$30k tho..)

And if windows 7 actually does make you pay monthly for features else they 
get turned off, then screw it...I'm just going to drop windows altogether 
and run Mono and Ubuntu.

ian

Post a reply to this message

[GDS|Entropy] wrote:
> You don't happen to know what that "some setting" is that would let me use 
> 3gb for progs until then do you? That would really be useful...

Look at the /3GB switch setting in boot.ini, as described here:

http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx

Post a reply to this message

[GDS|Entropy] wrote:
> I know that in 32bit OS progs are limited to 2gb max or so each, but why 
> can't pov use a swap file like photoshop?

The confusion is between the amount of semiconductor RAM available and the 
amount of space in the address space.

Maybe an analogy would help: At the bank, they give you 4 squares on the 
deposit slip to write the amount of your deposit. The most you can deposit 
on one slip is $9999. Now, maybe their cash drawer only has room to fit 
$1000 at a time, so the use the "swap space" in the vault to move different 
drawers in and out of the vault to handle peoples deposits. You can have 
lots of people depositing $100 and $200 at a time, and you can have one 
person deposit $6000 (but that person will wait while tellers run back and 
forth to the vault), but one person just can't put more than $9999 in their 
account regardless of how big the bank vault is.

You're running up against there only being 32 squares on the deposit slip, 
regardless of how much room is left in the vault. You'll find you can run 
(slowly) another copy of POV at the same time that the first is running out 
of memory.

POV could use a swap file and manually track it, but that's a *lot* of code 
for relatively little gain. (Indeed, pretty much all the text editors and 
word processors back in 9-bit days did the same sort of thing.)

It would have been more gain had 64-bit computers not come along and 
eliminated the need to track that manually. Now, it's probably cheaper to 
install Win64 than it is to pay someone to code up a swap file in POV. :-) 
The power of mass market solutions trumps per-application solutions, in this 
case.

> You don't happen to know what that "some setting" is that would let me use 
> 3gb for progs until then do you? That would really be useful...

GIYF. http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx

"The virtual address space of processes and applications is still limited to 
2 GB unless the /3GB switch is used in the Boot.ini file."

> xp pro 64

I have noticed that on high-end machines, Vista seems to do a better job of 
handling (for example) disk and windowing stuff at the same time. I can copy 
from disk1 to disk2 and disk3 to disk4 and still not bring the system to its 
knees, which wasn't true of my xp systems.

> And if windows 7 actually does make you pay monthly for features else they 
> get turned off, then screw it

I think a huge number of people would agree. Have you heard anything along 
those lines announced? I'd be surprised if anything other than "enterprise" 
type stuff (like Exchange servers, say) needed an ongoing license fee.

-- 
   Darren New, San Diego CA, USA (PST)
   My fortune cookie said, "You will soon be
   unable to read this, even at arm's length."

Post a reply to this message

"Warp" <war### [at] tagpovrayorg> wrote in message
news:49b3f1f4@news.povray.org...
> [GDS|Entropy] <gds### [at] hotmailcom> wrote:

> > I don't understand why I'm getting those errors then... It will suck up
2gb
> > of ram and then it dies, sometimes catastrophically. I have like 80gb
free
> > on disk too.

>   It would be possible to address more data by using 64-bit addresses
> (which is how eg. files of over 4GB are handled), but in a 32-bit
> computer managing 64-bit addresses will impose a speed penalty, as

When working with swap files, there is no speed penalty (nothing appreciable
anyway), whether it's 2GB, 2TB or 2EB.

Post a reply to this message

"Darren New" <dne### [at] sanrrcom> wrote in message
news:49b41abe$1@news.povray.org...

> It would have been more gain had 64-bit computers not come along and
> eliminated the need to track that manually. Now, it's probably cheaper to
> install Win64 than it is to pay someone to code up a swap file in POV. :-)
> The power of mass market solutions trumps per-application solutions, in
this
> case.

Swapping to disk is immaterial of data bus width, it all depends on how much
physical memory you have installed. Physical memory is still relatively
expensive and not dense enough, there are physical and electronic
limitations, so even 64 bit systems shipping today may be limited to as
little as 6GB.

On top of that, for high efficiency, automatic memory management will not
beat application managed memory. It's the application that knows (or should
know, since moderns paradigms of today, carried out to the extreme, seem to
strive to make an application as ignorant as possible of the overall
picture) what data needs to be in memory and what data can be swapped out.
OS manages memory more or less blindly, ignorant of the application's
architecture or needs.

Not directly related, but one of my biggest beef is with applications that
don't know or care how big of data they are dealing with. Some archivers,
for instance, instead of uncompressing/compressing an archive to the
destination folder directly, uncompress it to a windows temporary folder,
and then copy it to the destination. It's assinine as it is, but for a 1MB
file, it's bearable. For a 1GB file, it's absolutely ridiculous. Similarly,
I've, on more than one occasion, cancelled a 2hr download by mistake and
went scrambling to the temporary internet files folder, because due to
thoughtless GUI design, the download window, with the *focus on cancel
button*, comes to the foreground once the download is complete and Windows
starts copying (again, assinine) the file from the temp folder to its
destination folder. That behaviour, as dumb as it is, may be acceptable for
a download that takes 2 seconds, and the shortsighted programmer surely did
not think of anything else, but not for a download that takes 2 hrs during
which time the user starts typing and working on other things on the
computer.

Similarly, a text/hex editor that uses the same method (of loading into
memory) for opening a 1KB file as a 1GB file is, to put it mildly, has got
to be written by monkeys. No application that deals with variable size data,
IMO, should be ignorant about the amount of data it may be dealing with, and
chose sensible strategies for different amounts of data.

Post a reply to this message

somebody wrote:
> "Darren New" <dne### [at] sanrrcom> wrote in message
> news:49b41abe$1@news.povray.org...
> 
>> It would have been more gain had 64-bit computers not come along and
>> eliminated the need to track that manually. Now, it's probably cheaper to
>> install Win64 than it is to pay someone to code up a swap file in POV. :-)
>> The power of mass market solutions trumps per-application solutions, in
> this
>> case.
> 
> Swapping to disk is immaterial of data bus width, it all depends on how much
> physical memory you have installed.

No. It also depends on how much address space you have available to work 
with, if you're talking about OS-supported demand-paged memory.

If you're talking about what Photoshop does, basically building its own 
demand-paged addressing system on top of the application, then no, that 
makes less difference. Photoshop has that feature because it was originally 
written in the days of 8-bit and 16-bit machines, where it was normal for 
the data to exceed the address space even in usual uses. Most programs of 
the time had the same kinds of features. It's very difficult to add 
after-the-fact, especially to a program written in a language where it's 
easy to make assumptions about how memory is organized.

> so even 64 bit systems shipping today may be limited to as
> little as 6GB.

6GB of physical RAM. That's not your problem. Your problem is that you have 
only 2GB of address space. You'd run into the same limitation regardless of 
how much physical RAM or disk you have.

> On top of that, for high efficiency, automatic memory management will not
> beat application managed memory.

That's debatable. It's certainly sometimes true. Not always, tho.

> Similarly, a text/hex editor that uses the same method (of loading into
> memory) for opening a 1KB file as a 1GB file is, to put it mildly, has got
> to be written by monkeys.

Talk to RSM. I tried to use his emacs a long time ago, and the executable 
wouldn't even link into the address space of an 8086. :-) Other editors of 
the time (just as powerful) knew memory was limited and did things to handle 
it.

-- 
   Darren New, San Diego CA, USA (PST)
   My fortune cookie said, "You will soon be
   unable to read this, even at arm's length."

Post a reply to this message

"Darren New" <dne### [at] sanrrcom> wrote in message
news:49b450c9@news.povray.org...
> somebody wrote:

> > Swapping to disk is immaterial of data bus width, it all depends on how
much
> > physical memory you have installed.

> No. It also depends on how much address space you have available to work
> with, if you're talking about OS-supported demand-paged memory.

I meant application managed swapping.

> If you're talking about what Photoshop does, basically building its own
> demand-paged addressing system on top of the application, then no, that
> makes less difference. Photoshop has that feature because it was
originally
> written in the days of 8-bit and 16-bit machines, where it was normal for
> the data to exceed the address space even in usual uses.

Yes, "usual use" is the operative phrase. It's very hard to anticipate what
a "usual use" is, even down 6 months down the road. Adobe got lucky being
forced to make correct decisions at that time. You are also correct that's
it's very difficult to alter the architecture of an existing program, as you
note below. Thus, the safe method is to *not* assume your data will always
fit into physical memory, even if it will fit into the address space for the
foreseeable future. If the possibility is there that the data may not fit
into physical memory, you have to ask if you or the OS will be in a better
position to decide when, if, what and how to page. OS, to me, should be the
last resort. I see entirely too many programs today being written with most
inefficient use of memory since "OS will manage it". With 64 bit address
space, I am pretty sure that we will see applications that are memory hogs
and inefficient at levels we have never seen before.

> Most programs of
> the time had the same kinds of features. It's very difficult to add
> after-the-fact, especially to a program written in a language where it's
> easy to make assumptions about how memory is organized.

Post a reply to this message

somebody wrote:
> Thus, the safe method is to *not* assume your data will always
> fit into physical memory, even if it will fit into the address space for the
> foreseeable future.

I disagree. You balance the functionality you lose against the speed and/or 
efficiency and/or programming convenience you gain.

 > If the possibility is there that the data may not fit
> into physical memory, you have to ask if you or the OS will be in a better
> position to decide when, if, what and how to page. 

Yep. And you have to balance it against the fact that the OS *will* make 
that decision regardless of whether you want it to or not, and regardless of 
whether you're coding your own paging or not. You also have to balance it 
against the fact that the OS knows who else is using the memory, the 
performance characteristics of the various devices in the machine (including 
page tables, main memory, USB memory, and the disks it may have available to 
swap to), paging behavior of previous executions of the program, and so on. 
I'm not sure you'd gain much by reimplementing all that logic in each 
application and *still* not having the global knowledge the OS has.

> OS, to me, should be the last resort. 

I'm glad all your programs implement their own custom swap files. Me, I'd 
rather have actual improvements in the outputted results, especially given 
that we both already have machines that can handle plenty of memory.

If 64-bit chips and OSes weren't already available, I might agree with you 
that it is important. But I'd rather see POV making full use of all the 
cores than getting slowed down to support someone running a 32-bit OS on a 
64-bit machine.

> I see entirely too many programs today being written with most
> inefficient use of memory since "OS will manage it". With 64 bit address
> space, I am pretty sure that we will see applications that are memory hogs
> and inefficient at levels we have never seen before.

Then don't buy them. :-)

-- 
   Darren New, San Diego CA, USA (PST)
   My fortune cookie said, "You will soon be
   unable to read this, even at arm's length."

Post a reply to this message