 |
|
|
|
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Tom Austin <taustin> wrote:
> Ger wrote:
>> When trying to run a render that uses over 3Gb memory I get this error
>>
>
> ...
>
>
>> Keeping it below 3Gb (by reducing the number of lights) it works fine.
>>
>> Just wondering if I should keep that 3Gb limit in mind or if there is
>> something else going wrong.
>>
>> Running:
>> Athlon 64x2 3800+
>> 4 Gb mem internal (6Gb swap) (so available mem should not be a problem)
>> Suse 10 with Linux athlon64 2.6.13-15-bigsmp kernel
>> Self compiled Povray 3.6.1
>
> It looks like you know what you are doing, but here are some things I
> have run across regarding memory - based on 32 bit systems.
>
> A single application (thread) is limited to using 2GB or 3GB depending
> on your OS and settings within the OS.
>
> I have found that typically in
> Windows the limit is 2GB and Linux is 3GB (tho these can be affected by
> OS settings). I have run into these ceilings myself running various
> applications. I have found no way of allowing for more memory usage in
> a 32bit system. I set up 16GB swap once only to have the application
> crash at 3GB.
This is most likely due to limits inside the application, not the OS
>
> Also, the OS might be capable of using much more than 2GB (or 3GB), but
> it must be in the form of separate applications (or threads). It might
> be entirely possible to run (3) 2GB applications -
I'm currently running 2 renders of 2.5 Gb each, alongside the usual bunch of
office thingies, mail, news, browsers, photo editors, etc etc.
Swap space usage is upto 3.5 Gb now.
> tho I have not tried
> or tested this myself. Others here with more knowledge can set me
> straight.
>
> I remember that these is a configuration within Linux that allows it to
> use different memory mapping to allow for large memory usage. If I
> remember correctly, it was a kernel option and required the kernel to be
> recompiled.
Correct, when you go above 3 Gb you will need to change the kernel to the
"big" version allowing for more internal memory, and this is where my
problems started. It turned out that something fishy had happened when I
switched kernels.
>
> If your system is 64 bit, it could be that something is still limiting
> it to using 32 bit memory addressing.
Not under Linux, I don't know about 64bit Windows
>
> Now, my experience is only with a 32 bit system. I understand that 64
> bit systems have much different limitations, but I have no experience
> with them. I hope that this sheds a bit of light for you.
>
With the arrival of 64 bit systems the phrase "The sky's the limit" has been
superseded with "The sky's just the beginning" :)
>
>
>
> LAter... Tom
--
Ger
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Ger wrote:
> Tom Austin <taustin> wrote:
>
>> It looks like you know what you are doing, but here are some things I
>> have run across regarding memory - based on 32 bit systems.
>>
And, yes, you do know what you are doing :-)
>> A single application (thread) is limited to using 2GB or 3GB depending
>> on your OS and settings within the OS.
>>
>> I have found that typically in
>> Windows the limit is 2GB and Linux is 3GB (tho these can be affected by
>> OS settings). I have run into these ceilings myself running various
>> applications. I have found no way of allowing for more memory usage in
>> a 32bit system. I set up 16GB swap once only to have the application
>> crash at 3GB.
>
> This is most likely due to limits inside the application, not the OS
>
Not on a 32 bit system where accessing that much memory is a problem.
Lots of research and trial and error have shown me this.
>> Also, the OS might be capable of using much more than 2GB (or 3GB), but
>> it must be in the form of separate applications (or threads). It might
>> be entirely possible to run (3) 2GB applications -
>
> I'm currently running 2 renders of 2.5 Gb each, alongside the usual bunch of
> office thingies, mail, news, browsers, photo editors, etc etc.
> Swap space usage is upto 3.5 Gb now.
>
>
> Correct, when you go above 3 Gb you will need to change the kernel to the
> "big" version allowing for more internal memory, and this is where my
> problems started. It turned out that something fishy had happened when I
> switched kernels.
>
That's what it started to look like to me when I was replying to your
message earlier. That running 64 bit should not have any problems, but
something is still holding you back. And a misconfiguration in the
kernel could do that :-)
>> If your system is 64 bit, it could be that something is still limiting
>> it to using 32 bit memory addressing.
>
> Not under Linux, I don't know about 64bit Windows
>
A misconfiguration could hold you back - what I was getting at.
>> Now, my experience is only with a 32 bit system. I understand that 64
>> bit systems have much different limitations, but I have no experience
>> with them. I hope that this sheds a bit of light for you.
>>
>
> With the arrival of 64 bit systems the phrase "The sky's the limit" has been
> superseded with "The sky's just the beginning" :)
>
Every once in a while we throw something at a computer that is just too
much for it. I know of two incidents where I've needed more than 2GB of
memory. One was due to a bug in AutoCAD - which I now have a workaround
for.. the other is aligning 50+ 3D data scans (2 million data points
each) which we do in sections.
Maybe our next computer will be a 64 bit system so that we can see how
it affects our work flow.
LAter... Tom
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Ger wrote:
> This is most likely due to limits inside the application, not the OS
Errr, no. It's because a 32-bit address can only address 4G. What would
you have malloc() return for the fifth gig of memory? The "limit inside
the application" would be one of not manually implementing
bigger-than-32-bit pointers and its own swap space.
Most people who went from 16-bit 32-bit processors stopped worrying
about implementing more per-process storage than available address
space. Of course, it was all the rage being able to edit a file bigger
than 64K back in 8080 days. :-)
--
Darren New / San Diego, CA, USA (PST)
"I think these anchovies are spoiled.
They're not flat."
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Tom Austin <taustin> wrote:
> It looks like you know what you are doing, but here are some things I
> have run across regarding memory - based on 32 bit systems.
> A single application (thread) is limited to using 2GB or 3GB depending
> on your OS and settings within the OS.
Don't base your answer on 32-bit systems since he is using a 64-bit one.
A friend of mine has an amd64 and he tried how much memory he can
allocate in linux. In linux you can actually allocate more memory than
is physically available (even when counting swap space). It's only when
you try to *use* more memory than what is physically available when it is
not possible.
I don't remember exactly how much he could allocate, but it was less
than 2^64. However, it was way over 2^32. It might have been something
like 2^48 or whatever. I don't know why there's such a limit, but I assume
that his motherboard limits the address space of the system to that
(it's not unusual for motherboards of new processors to not to support
everything the processor could).
I don't know why he cannot use more than 3GB in his system, though.
--
- Warp
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> Tom Austin <taustin> wrote:
>> It looks like you know what you are doing, but here are some things I
>> have run across regarding memory - based on 32 bit systems.
>
>> A single application (thread) is limited to using 2GB or 3GB depending
>> on your OS and settings within the OS.
>
> Don't base your answer on 32-bit systems since he is using a 64-bit one.
>
> A friend of mine has an amd64 and he tried how much memory he can
> allocate in linux. In linux you can actually allocate more memory than
> is physically available (even when counting swap space). It's only when
> you try to *use* more memory than what is physically available when it is
> not possible.
> I don't remember exactly how much he could allocate, but it was less
> than 2^64. However, it was way over 2^32. It might have been something
> like 2^48 or whatever. I don't know why there's such a limit, but I assume
> that his motherboard limits the address space of the system to that
> (it's not unusual for motherboards of new processors to not to support
> everything the processor could).
>
> I don't know why he cannot use more than 3GB in his system, though.
>
I couldn't use the memory because something went wrong when I switched
kernels. I upgraded my machine from 2Gb to 4 Gb and had to switch to the
"big" kernel. It supports >3Gb memory. In my case the "bigsmp" kernel since
I have an AMD X2 (dual core).
I have since reinstalled my system and for the sake of testing ran renders
that needed > 10Gb of memory and that went just fine.
--
Ger
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Warp wrote:
> Tom Austin <taustin> wrote:
>> It looks like you know what you are doing, but here are some things I
>> have run across regarding memory - based on 32 bit systems.
>
>> A single application (thread) is limited to using 2GB or 3GB depending
>> on your OS and settings within the OS.
>
> Don't base your answer on 32-bit systems since he is using a 64-bit one.
>
Your statement is very judgmental and a week after the fact.
You may not have read completely what I wrote:
"If your system is 64 bit, it could be that something is still limiting
it to using 32 bit memory addressing. "
I offered what information I knew. He might have already known it all,
but then he might not have. I didn't presume that my information about
32 bit systems would solve his problems. I only thought that if I
offered what I knew that it might trip his mind on what was actually the
problem.
And his problem was a 32 bit one.
> I don't know why he cannot use more than 3GB in his system, though.
>
read the thread - he didn't have 'big' memory compiled into the kernel.
LAter... Tom
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Tom Austin <taustin> wrote:
> Warp wrote:
>> Tom Austin <taustin> wrote:
>>> It looks like you know what you are doing, but here are some things I
>>> have run across regarding memory - based on 32 bit systems.
>>
>>> A single application (thread) is limited to using 2GB or 3GB depending
>>> on your OS and settings within the OS.
>>
>> Don't base your answer on 32-bit systems since he is using a 64-bit
>> one.
>>
>
> Your statement is very judgmental and a week after the fact.
>
>
> You may not have read completely what I wrote:
>
> "If your system is 64 bit, it could be that something is still limiting
> it to using 32 bit memory addressing. "
>
> I offered what information I knew. He might have already known it all,
> but then he might not have. I didn't presume that my information about
> 32 bit systems would solve his problems. I only thought that if I
> offered what I knew that it might trip his mind on what was actually the
> problem.
>
> And his problem was a 32 bit one.
Neither 32bit nor 64bit. It was a screwed up kernel.
What put me on the wrong track at first was the fact that povray crashed
at(or very near) the 3Gb limit (which is 32 bit)
>
>> I don't know why he cannot use more than 3GB in his system, though.
>>
>
> read the thread - he didn't have 'big' memory compiled into the kernel.
>
I did have the "big" kernel but it was erroneous and that's what threw me
off
> LAter... Tom
--
Ger
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
| |
|
|
Ger wrote:
> Tom Austin <taustin> wrote:
>
>>
>> I offered what information I knew. He might have already known it all,
>> but then he might not have. I didn't presume that my information about
>> 32 bit systems would solve his problems. I only thought that if I
>> offered what I knew that it might trip his mind on what was actually the
>> problem.
>>
>> And his problem was a 32 bit one.
>
> Neither 32bit nor 64bit. It was a screwed up kernel.
>
Ok, a kernel that was saying it was big, but acted small :-)
Whatever the problem was, it is now fixed :-)
>
> I did have the "big" kernel but it was erroneous and that's what threw me
> off
>
Don't you love it when the problem is something deep that's hard to
find. At least it wasn't something that took you years to figure out by
accident :-)
Thanks for your reply in setting me straight.
I was mainly defending my desire to help out a fellow POVer.
LAter... Tom
Post a reply to this message
|
|
| |
| |
|
|
|
|
| |
|
|
