In most unix systems, and in my case most relevantly in linux, there's
a very useful command-line utility (or, in many cases, a shell built-in
command) called "time". This utility can be used to very accurately
measure how much system-time, user-time and overall time a program takes
during its execution. This is extremely useful for example when comparing
the speed of several versions of a program.

  Now, what I would also like to have is a similar utility or built-in
command which tells be the peak memory usage of a program. I want to know,
for example, if an optimization I made in a program really saved a relevant
amount of memory or not. Sounds rather simple, doesn't it?

  Well, to my amazement there doesn't seem to be any such utility in
existence for unix or linux in particular.

  There are tons of ways of seeing how much a program is *currently*
taking. For example, you can use utilities like "top", "pmap", "vmstat"
and "ps" for this purpose. They all can tell how much memory a program
is taking.

  However, those are no good. If a program runs just for a few seconds,
constantly allocating more and more memory, those utilities are completely
useless. Even if you get to run them during the execution of the program,
you only get a completely random result: They show you how much memory
the program is using *at that moment*. They don't show you the overall
peak memory usage of the program. The program may well allocate humongous
amounts of memory more after you took a look at it with any of those
utilities. They don't tell you anything relevant. (Those utilities are
only useful for programs which are running for a long time, and you want
to take a look on how much they are consuming memory currently.)

  What I don't understand is why there can't be any such utility.
Given that utilities like "top" & co. can measure the memory usage
of a process, it would sound quite plausible that it would be rather
easy to make a utility similar to "time" which measures the peak memory
usage of a program.

  But no. There doesn't seem to be any such program. I'm completely
puzzled about why this is so.

  (And before anyone suggests using the gnu version of "time", don't
bother: It doesn't work. It always gives 0 as the amount of memory used,
which is documented to happen when it doesn't support measuring that.)

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   But no. There doesn't seem to be any such program. I'm completely
> puzzled about why this is so.

For one thing, there's no unified performance measurement/reporting 
mechanisms in Linux. And in the original versions of Linux, you really 
could only fit a tiny number of programs in memory at once. (The reason 
fork() works the way it does is it was implemented originally (as in 
Version 7) as "swap out this process, but don't clear it out of memory 
afterwards.)

So there's no good way of seeing how much your peak memory is, or your 
disk I/O rate, or how much network bandwidth you're taking up, because 
you'd have to build each of these reporting infrastructures from 
scratch. There's no "performance monitor" kind of thing where you could 
implement *just* the memory watcher and already have the libraries to 
track peak, average, etc.

-- 
   Darren New / San Diego, CA, USA (PST)
     "That's pretty. Where's that?"
          "It's the Age of Channelwood."
     "We should go there on vacation some time."

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> So there's no good way of seeing how much your peak memory is, or your 
> disk I/O rate, or how much network bandwidth you're taking up, because 
> you'd have to build each of these reporting infrastructures from 
> scratch. There's no "performance monitor" kind of thing where you could 
> implement *just* the memory watcher and already have the libraries to 
> track peak, average, etc.

  Yet there are plenty of resource monitors which can show cpu usage,
bandwidths, memory usage, disk usage, and lots of info on a per-process
basis ('top' being the basic example).

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
>   Yet there are plenty of resource monitors which can show cpu usage,
> bandwidths, memory usage, disk usage, and lots of info on a per-process
> basis ('top' being the basic example).

But that's peeking into the current system tables. Seeing what it is 
right now is easy. Reliably recording it moment by moment needs more, 
methinks.

If saying "hey, I just called sbrk() for another page of memory" or "I 
just did six I/Os to satisfy a select query" was as easy as sending 
something to syslog, I expect a lot more libraries would implement those 
sorts of measurements. As it is, there's no good place for the C runtime 
(for example) to record the max memory used, and no easy way to look at 
it when it's done. So adding this capability needs the implementor not 
only to record how much memory was used, but to write a program to 
display the histogram, to decide on a file format to use, to build the 
configuration subsystem to say what file it should go into, etc. And if 
it's something in the kernel, you have to probably build a device driver 
to turn it on and off, etc.  Probably seems like a lot of work.

Of course, I'm just guessing at a reason.

Say, did you try things like Purify? Shouldn't that be able to track it, 
since it replaces malloc and free?

-- 
   Darren New / San Diego, CA, USA (PST)
     "That's pretty. Where's that?"
          "It's the Age of Channelwood."
     "We should go there on vacation some time."

Post a reply to this message

Darren New a écrit :
> Say, did you try things like Purify? Shouldn't that be able to track it, 
> since it replaces malloc and free?
> 

What's really strange is that valgrind, which tracks everything having 
to do with memory, does not provide the peak usage in the output. It's 
on the graph written by massif, one of the tools, but there is no 
numerical value.

Post a reply to this message

Vincent Le Chevalier <gal### [at] libertyallsurfspamfr> wrote:
> What's really strange is that valgrind, which tracks everything having 
> to do with memory, does not provide the peak usage in the output. It's 
> on the graph written by massif, one of the tools, but there is no 
> numerical value.

  valgrind only counts how much memory is *explicitly* allocated by
the program (eg. with malloc calls). It doesn't show the *real* amount
of memory used by the object. The real amount may be larger for several
reasons, such as each 'malloc()' or 'new' call causing extra 16 bytes to
be allocated, the memory page size in linux being 4 kilobytes, etc etc.
Also memory fragmentation may well cause the program to allocate
considerably more memory than it's really using, and valgrind won't
show you that because it only counts explit mallocs and frees. (The real
amount of memory used can only be examined with system calls because
it's the OS which is book-keeping the real amount of memory allocated for
each process.)

  Examining brk calls with strace gets much closer to the truth, but
it may not be a completely sure way of telling the real amount of memory
the process is using (as some memory may be allocate with mmap2).

-- 
                                                          - Warp

Post a reply to this message

Darren New <dne### [at] sanrrcom> wrote:
> If saying "hey, I just called sbrk() for another page of memory" or "I 
> just did six I/Os to satisfy a select query" was as easy as sending 
> something to syslog, I expect a lot more libraries would implement those 
> sorts of measurements.

  I'm not exactly sure what you mean by that. Examining the brk() system
calls made by a process is quite easy using 'strace' (or in some unix systems
'truss').

  Examining brk() calls can actually be used to get a pretty accurate figure
of the memory usage of the program, but it's not always that unambiguous.
Basically you would need to know what is the base address used by the
program so that you can substract the largest value of brt() from that.
Usually the first brk() request made by the process is a good bet, but
the result may still be a bit biased. Another way is to put some debug
print at the beginning of main() and start counting brk() calls from
there, which gives the amount of memory allocated by the program after
its main() has started. (This can be done easily in the program itself
by making an sbrk(0) call at the beginning of main() and another at the
end and calculating their difference, which will give the amount of memory
in bytes that the program has allocated.)

  OTOH, only examining brk() calls is not the whole story and doesn't
always give the real amount of memory allocated by the program. This is
because libc sometimes makes cleverl tricks using mmap2() to allocate
memory, bypassing brk(). Thus any memory allocated that way will be
missed if we examine only brk().

  But anyways, for my purposes it would be enough to get whatever tools
like 'top' are using to measure the memory usage of the program.

> Say, did you try things like Purify? Shouldn't that be able to track it, 
> since it replaces malloc and free?

  Simply examining mallocs and frees performed by the program does not
give an accurate figure. The actual amount of memory used by the program
may be significantly larger. For example memory fragmentation can cause
this quite prominently.

-- 
                                                          - Warp

Post a reply to this message

Warp wrote:
> Darren New <dne### [at] sanrrcom> wrote:
>> If saying "hey, I just called sbrk() for another page of memory" or "I 
>> just did six I/Os to satisfy a select query" was as easy as sending 
>> something to syslog, I expect a lot more libraries would implement those 
>> sorts of measurements.
> 
>   I'm not exactly sure what you mean by that. Examining the brk() system
> calls made by a process is quite easy using 'strace' (or in some unix systems
> 'truss').

I mean that if programmers didn't need to build the entire reporting 
infrastructure, but could instead just "add points to the graph", I 
expect programmers would be more likely to include the code to do so.

The comparison with syslog was meant to say: If every programmer had to 
recreate the entire syslog deamon and reporting infrastructure, you'd 
have far fewer programs capable of doing distributed monitoring. As it 
is, all the program needs to do is drop a UDP packet, and the whole 
"where does it go, who sees it, and do I send a page" thing is already 
written for it.

If there was a system call that said
   add_to_counter("MemoryPagesUsed", 1)
   add_to_counter("MemoryPagesUsed", -1)
already written, you'd probably see a version of the C runtime library 
with a switch that would invoke those calls within malloc and free.

Plus, you'd probably see a bunch more servers reporting the same way: 
pages served per second from Apache, queries answered by MySQL, etc. 
Rather than (for example) having to parse Apache web logs to figure that 
out for Apache, and someone else's logs to figure it out for *that* 
program, and etc.

>   But anyways, for my purposes it would be enough to get whatever tools
> like 'top' are using to measure the memory usage of the program.

Yah. I was kind of disappointed when I was running "top" and seeing (for 
example) several programs blocked, nobody running, no disk I/O, no 
network I/O, etc.  vmstat turned out to tell the real story, but not on 
a per-processes basis.

>   Simply examining mallocs and frees performed by the program does not
> give an accurate figure. The actual amount of memory used by the program
> may be significantly larger. For example memory fragmentation can cause
> this quite prominently.

Fair enough. :-)  I wasn't sure just what you were after.

-- 
   Darren New / San Diego, CA, USA (PST)
     "That's pretty. Where's that?"
          "It's the Age of Channelwood."
     "We should go there on vacation some time."

Post a reply to this message

Sounds like a job for the kernel.  Doesn't the kernel support logging?

-- 
...Ben Chambers
www.pacificwebguy.com

Post a reply to this message

Warp wrote:
> 
>   Now, what I would also like to have is a similar utility or built-in
> command which tells be the peak memory usage of a program. I want to know,
> for example, if an optimization I made in a program really saved a relevant
> amount of memory or not. Sounds rather simple, doesn't it?
> 
>   Well, to my amazement there doesn't seem to be any such utility in
> existence for unix or linux in particular.
> 
>   There are tons of ways of seeing how much a program is *currently*
> taking. For example, you can use utilities like "top", "pmap", "vmstat"
> and "ps" for this purpose. They all can tell how much memory a program
> is taking.
> 

I believe that in the proc directory you can dig down to get to the 
actual process information.
I would check to see if there is a 'file' for the specific process that 
reports the current memory usage.
I know that it is these files that ps uses for its reporting.

Just a thought on a direction that could give you what you are after.


Tom

Post a reply to this message